HitKeep Single-Binary Web Analytics Architecture

HitKeep is designed to give you control over every layer of the analytics stack without requiring you to operate multiple services. The main analytics path, dashboard API, embedded queue, and DuckDB storage live in one Go binary. Optional integrations such as SMTP, S3 backup/archive storage, favicon lookup, MCP docs tools, and AI provider enrichment can make outbound calls when you enable or configure them. For citable binary size, memory use, storage boundaries, privacy behavior, exports, and non-goals, see Facts and Limits.

High-Level Data Flow

Browser and server-side pageview or event data flows through four stages on the leader node:

1. Ingestion: The HTTP server receives a pageview or event from the tracking script.
2. Buffering: The hit is published to an in-memory NSQ queue, decoupling the HTTP response from the write path.
3. Processing: Internal consumers read from the queue concurrently, group messages by resolved tenant store, and build short-lived micro-batches.
4. Storage: Each store-local batch is written to embedded DuckDB through the appender API, then the queued messages are acknowledged.

The tracking script (hk.js) is also served from your instance. Opt-in Web Vitals use the same-origin hk-vitals.js split bundle. No CDN or third-party analytics domain is required.

System Boundaries

The easiest way to understand HitKeep today is to separate it into four concerns:

1. Public ingest and API surface
2. Shared control plane
3. Tenant-local analytics data plane
4. Background lifecycle workers

1. Storage: DuckDB

HitKeep uses DuckDB, an in-process OLAP database engine, as its primary store.

Why DuckDB instead of PostgreSQL or ClickHouse?

• Runs inside the Go process — no separate server, no socket, no authentication
• Columnar storage optimizes analytical queries (aggregations, time-bucketing) over row-based access patterns
• A single file (hitkeep.db) per database — but multiteam installs contain multiple DuckDB files under one data-path, so backups should capture the full directory tree
• ~120 MB per million hits compressed — efficient for VPS-class storage
• Queryable directly with the DuckDB CLI or any Parquet-compatible tool — your data is portable by nature

# Read your analytics database directly — no HitKeep running required
duckdb /var/lib/hitkeep/data/tenants/{default-tenant-id}/hitkeep.db \
  "SELECT date_trunc('day', timestamp), count(*) FROM hits GROUP BY 1 ORDER BY 1 DESC LIMIT 7;"

Control Plane and Data Plane

HitKeep separates storage into two planes:

• Control plane (hitkeep.db): users, sessions, authentication, site metadata, share links, team membership, and user preferences.
• Data plane (per-team DuckDB files): hits, events, goals, funnels, and pre-aggregated rollups.

Starting with 2.13, every tenant—including the default tenant—has a data-plane database at {data_path}/tenants/{team_id}/hitkeep.db. The default tenant file is the root of one shared DuckDB data-plane instance; other active tenant files are attached lazily under isolated catalogs. Application analytics always resolve a tenant catalog before querying. hitkeep.db retains empty compatibility analytics tables during the 2.13 bridge release, but tenant files are authoritative.

/var/lib/hitkeep/data/
├── hitkeep.db                          # Control plane only
└── tenants/
    ├── {default_team_id}/hitkeep.db    # Shared data-plane root
    └── {team_id}/hitkeep.db            # Attached team catalog

Every data pathway — ingestion, API reads, background workers, exports — resolves the correct data plane database before touching analytics data. See Teams and Data Isolation for details.

Storage Boundary Diagram

The important rule is simple: identity and ownership metadata live in the control plane; analytics rows live in the resolved data plane.

2. Ingestion Buffering: Embedded NSQ

Writing to a columnar database synchronously per HTTP request creates lock contention when traffic spikes. HitKeep avoids that by embedding NSQ, a distributed messaging platform, in-process, and by flushing queued analytics rows to DuckDB in short appender batches instead of issuing one SQL INSERT per message.

• Decoupling: The ingest HTTP handler validates and enqueues the hit in memory, completing the request in microseconds.
• Burst absorption: Traffic spikes (a site goes viral, a product launches) queue up without database pressure. The writer consumes at a steady, optimal pace.
• Store-local batching: The consumer resolves the destination data plane first, then groups queued hits and events per DuckDB store before flushing them.
• Columnar-friendly writes: DuckDB’s appender API bypasses per-row SQL parsing and matches the database’s preferred bulk-ingest path.
• Zero configuration: NSQ runs inside the process on configurable loopback ports. No Kafka cluster, no broker to manage.

NSQ’s TCP and HTTP interfaces bind to 127.0.0.1 by default and are not exposed externally.

In practice, the write path now looks like this:

1. HTTP handler validates and publishes the hit or event to NSQ.
2. One of the in-process consumers picks up the message.
3. The consumer resolves the correct tenant analytics store for the site_id.
4. Messages targeting the same store are coalesced into a small batch for a short time window.
5. The batch is flushed through DuckDB’s appender API.
6. Only after the flush succeeds are the corresponding NSQ messages acknowledged.

Read and Write Paths

The read path and write path both go through the same tenant resolution layer.

Read operations resolve ownership first, then open the correct analytics data plane:

Browser and server-side ingest writes acknowledge the HTTP request before the DuckDB batch flush:

That applies to:

• dashboard reads
• API exports
• share link reads
• API clients
• ecommerce analytics
• opportunity detectors and saved recommendations
• background workers
• site transfer copy/delete operations

3. Clustering: Leader / Follower

For high availability, HitKeep supports a Leader/Follower topology using HashiCorp Memberlist (gossip protocol) for node discovery.

Roles

Role	Behavior
Leader	Opens the control-plane and tenant DuckDB files, runs embedded NSQ, consumers, workers, MCP, and stateful API handlers
Follower	Serves the HTTP process without stateful stores; proxies browser /ingest and /ingest/event requests to the leader

There is exactly one Leader at any time. If the Leader goes down, the cluster elects a new one — provided the PVC or data volume can be re-attached (Kubernetes StatefulSets handle this automatically).

Request Flow

Load Balancer → Follower → [proxy /ingest or /ingest/event] → Leader → NSQ → DuckDB
Load Balancer → Leader   → [dashboard/API/MCP/stateful handlers] → Tenant Store Manager → DuckDB

For single-server deployments (Docker Compose, systemd), the node acts as Leader implicitly.

Optional MCP Route

The optional MCP route is mounted on the main HitKeep HTTP server. It is disabled by default and is registered only on the leader after leader services are ready.

MCP requests use the Streamable HTTP transport and authenticate with existing API client bearer tokens. They do not accept dashboard cookies. The route is stateless at the protocol layer: 2026-07-28 clients use sessionless server/discover and self-contained requests, while legacy clients retain initialize negotiation. Standardized MCP method/name/version headers are validated by the SDK. Analytics reads pass through the same API client permission checks and tenant store resolution used by REST handlers.

The v1 MCP surface is read-only and aggregate-only. It can list visible sites, return overview, event, ecommerce, Web Vitals, AI visibility, saved Opportunities, and imported Search Console analytics, and expose local help resources. Where a report supports visitor context, MCP can return city/provider/ASN aggregate breakdowns without exposing raw IP addresses, user agents, session IDs, or visitor rows. List and discovery responses expose private five-minute cache hints; static help/metric resources expose private one-hour hints; docs resources use the configured docs-cache TTL, and analytics tool results are not cacheable. Docs tools may fetch official HitKeep documentation as markdown from the configured docs origin, but analytics data is not sent to the docs site. Docs markdown is cached in a bounded in-memory LRU cache.

Optional AI Model Route

Optional AI provider calls are disabled by default. When enabled, HitKeep calls the configured provider, model, or OpenAI-compatible gateway route only for features that explicitly use AI.

The first feature is Opportunity Recommendations. Deterministic detectors read tenant-local analytics first and decide the opportunity type, evidence, impact, confidence, score, and status. HitKeep accepts only validated translation keys, params, cited evidence IDs, and safe structured output from AI enrichment. Raw prompts, raw provider payloads, and provider secrets are not persisted.

4. Security Layer

The HTTP server enforces multiple security controls before any data is processed:

• Rate limiting: Per-IP token bucket limiters on /ingest, /api/*, and /api/login. Configurable rate and burst.
• Cross-origin request protection: Uses Go’s standard Fetch Metadata and Origin checks to reject unsafe cross-origin browser requests while allowing safe navigation and non-browser API clients.
• JWT authentication: HTTP-only cookies signed with a configurable secret. Short expiry by default.
• WebAuthn: FIDO2/Passkey challenge-response for passwordless login.
• TOTP: RFC 6238 time-based one-time passwords for second-factor auth.
• Social sign-in: Google, GitHub, and Microsoft authorization code flows use one-time state, PKCE S256, immutable provider identities, verified-email or HitKeep confirmation rules, and the normal local MFA handoff.
• Team OIDC SSO: Team-scoped providers use discovery, issuer and audience validation, nonce, PKCE S256, verified email, and live membership or invitation checks.

See Configure Social Sign-In and Configure OIDC Single Sign-On for the two provider models and their different account boundaries.

Authentication Modes

HitKeep currently supports four request-authentication modes across the API surface:

• Session cookies are for interactive users in the dashboard.
• Personal API clients are user-owned bearer tokens.
• Team API clients are tenant-owned bearer tokens that survive individual user departure.
• Share tokens are read-only public-style analytics access scoped to one shared site/dashboard view.

The optional MCP server reuses API client bearer tokens only. This makes MCP access revocable through the same API client lifecycle as other server-to-server integrations.

5. Frontend Architecture

The dashboard is a Single Page Application served from the same binary:

• Framework: Angular v22 with Signals for reactive state
• UI library: OptimusUI with Tailwind CSS
• API contract: All dashboard functionality uses the same JSON REST API as external clients
• Tracking snippet: hk.js is minified with esbuild and served from your instance. It uses sendBeacon() with a keepalive fetch fallback, in-memory retry and dedupe state, and only stores the existing opaque session tuple in sessionStorage. Web Vitals stay in a separate opt-in hk-vitals.js bundle. See Tracker Architecture.

The dashboard itself is now multi-context:

• active site
• active team
• user/session state
• permission state
• analytics filters and date range

Those contexts stay on the client, but authoritative ownership and access checks always happen server-side in the shared control plane before analytics data is read.

6. Data Lifecycle and Recovery

HitKeep now has three distinct storage lifecycle concepts and they should not be confused:

1. Live databases for current reads and writes
2. Retention archives for old analytics rows exported to Parquet
3. Backup snapshots / purge flows for disaster recovery and irreversible cleanup

Important distinctions:

• -archive-path is for Parquet retention archives and related artifacts.
• -data-path is where live tenant DuckDB files live.
• checkpoints serialize WAL flushing per database; the periodic interval is configurable, while migrations checkpoint both before schema work and immediately after commit. Backups and clean shutdowns also require checkpoints.
• automatic recovery is classified and fail-closed. It retains a permission-restricted database/WAL bundle before removing non-unique secondary indexes implicated by the recognized mutation failure or, with explicit opt-in, bypassing a recognized unreplayable application WAL. Startup migrations run before the store is published and persist a durable checksum of the closed base database while the migration WAL remains authoritative; only a matching checksum permits automatic migration-WAL completion without the broader opt-in. Primary-key and unique indexes are not removed.
• recovery bundles are local incident artifacts, not backup snapshots, and are not pruned by backup retention.
• archiving a team is a reversible control-plane state until you run the purge path.
• purging an archived team is irreversible and removes its live tenant database directory.

7. Ecommerce in the Architecture

Ecommerce is not a separate subsystem. It is an opinionated query layer over the same tenant-local events tables.

That means:

• ecommerce data inherits team isolation automatically
• ecommerce filters reuse the same site/session attribution model
• ecommerce backups, restores, transfers, and retention follow the same tenant data-plane rules as other analytics data

What the Binary Contains

hitkeep (single executable)
├── Go HTTP server
├── Embedded NSQ broker + consumer
├── DuckDB engine + SQL migrations
├── Angular dashboard (compiled, embedded)
├── hk.js tracking snippet (embedded)
├── Optional MCP route (leader only)
├── Optional AI model route (disabled by default)
└── Background workers (retention, rollups, reports, imports, backups)

Every layer is auditable. The full source is on GitHub under the MIT license.

Related

• Configuration Reference
• Teams and Data Isolation
• Installation Guide
• Trusted Proxies
• Opportunity Recommendations

HitKeep Cloud runs this exact binary stack in managed EU (Frankfurt) or US (Virginia) infrastructure, with the same source-visible product foundation as self-hosted deployments. Start with HitKeep Cloud →