HitKeep Single-Binary Web Analytics Architecture

HitKeep is designed to give you control over every layer of the analytics stack without requiring you to operate multiple services. The main analytics path, dashboard API, embedded queue, and DuckDB storage live in one Go binary. Optional integrations such as SMTP, S3 backup/archive storage, favicon lookup, MCP docs tools, and AI provider enrichment can make outbound calls when you enable or configure them. For citable binary size, memory use, storage boundaries, privacy behavior, exports, and non-goals, see Facts and Limits.

High-Level Data Flow

Browser and server-side pageview or event data flows through four stages on the leader node:

1. Ingestion: The HTTP server receives a pageview or event from the tracking script.
2. Buffering: The hit is published to an in-memory NSQ queue, decoupling the HTTP response from the write path.
3. Processing: Internal consumers read from the queue concurrently, group messages by resolved tenant store, and build short-lived micro-batches.
4. Storage: Each store-local batch is written to embedded DuckDB through the appender API, then the queued messages are acknowledged.

flowchart TD
    Browser["Browser tracker\nor server-side ingest"] --> HTTP["Leader Go HTTP server"]
    HTTP --> Queue["Embedded NSQ\nin-memory buffering"]
    Queue --> Consumer["Ingest consumers\nconcurrent workers"]
    Consumer --> Resolver["Tenant Store Manager\nresolve site_id to data plane"]
    Resolver --> Batch["Store-local\nmicro-batches"]
    Batch --> DuckDB[("DuckDB tenant data plane")]

    AIFetch["AI crawler log forwarder"] --> HTTP
    HTTP --> DirectWrite["Permission check\ndirect aggregate insert"]
    DirectWrite --> Resolver

The tracking script (hk.js) is also served from your instance. Opt-in Web Vitals use the same-origin hk-vitals.js split bundle. No CDN or third-party analytics domain is required.

System Boundaries

The easiest way to understand HitKeep today is to separate it into four concerns:

1. Public ingest and API surface
2. Shared control plane
3. Tenant-local analytics data plane
4. Background lifecycle workers

flowchart TD
    Entry["Entry points\ntrackers, dashboard,\nAPI clients, MCP,\nAI fetch"] --> HTTP["Go HTTP server\nsingle binary"]
    HTTP --> Auth
    HTTP --> Queue["Embedded NSQ"]
    Queue --> IngestConsumer["Ingest consumers\nstore-local batches"]
    IngestConsumer --> Resolver

    Auth["Auth, validation,\nrate limits, permissions"] --> Control["Shared control plane\nhitkeep.db"]
    Auth --> Resolver
    Workers["Leader-only workers\nrollups, retention,\nreports, backups"] --> Resolver
    Resolver["Tenant Store Manager"] --> DefaultDB["Default tenant data plane\nshared hitkeep.db"]
    Resolver --> TenantDB["Team tenant data planes\ndata-path/tenants/team_id/hitkeep.db"]

    AIGateway["Configured AI provider\nor gateway"] -.-> HTTP

1. Storage: DuckDB

HitKeep uses DuckDB, an in-process OLAP database engine, as its primary store.

Why DuckDB instead of PostgreSQL or ClickHouse?

• Runs inside the Go process — no separate server, no socket, no authentication
• Columnar storage optimizes analytical queries (aggregations, time-bucketing) over row-based access patterns
• A single file (hitkeep.db) per database — but multiteam installs contain multiple DuckDB files under one data-path, so backups should capture the full directory tree
• ~120 MB per million hits compressed — efficient for VPS-class storage
• Queryable directly with the DuckDB CLI or any Parquet-compatible tool — your data is portable by nature

# Read your analytics database directly — no HitKeep running required
duckdb /var/lib/hitkeep/data/hitkeep.db \
  "SELECT date_trunc('day', timestamp), count(*) FROM hits GROUP BY 1 ORDER BY 1 DESC LIMIT 7;"

Control Plane and Data Plane

HitKeep separates storage into two planes:

• Control plane (hitkeep.db): users, sessions, authentication, site metadata, share links, team membership, and user preferences.
• Data plane (per-team DuckDB files): hits, events, goals, funnels, and pre-aggregated rollups.

For single-team instances, both planes coexist in one hitkeep.db file. When additional teams are created, each team gets its own data plane database at {data_path}/tenants/{team_id}/hitkeep.db. The only reference crossing the boundary is the site_id — no queries JOIN across planes. During the bridge release, shared goals and funnels remain only as compatibility copies for rollback safety; tenant-local copies are authoritative for non-default teams.

/var/lib/hitkeep/data/
├── hitkeep.db                          # Control plane + default team data plane
└── tenants/
    └── {team_id}/
        └── hitkeep.db                  # Team-specific data plane

Every data pathway — ingestion, API reads, background workers, exports — resolves the correct data plane database before touching analytics data. See Teams and Data Isolation for details.

Storage Boundary Diagram

flowchart TD
    Control["Shared control plane\nusers, sessions, teams,\nsite ownership, share links,\nAPI client metadata"]
    SiteMap["site_tenants mapping\nsite_id to tenant_id"]

    DefaultPlane["Default data plane\nshared hitkeep.db\nhits, events, goals,\nfunnels, rollups"]
    TeamA["Team data plane\n[data-path]/tenants/team-a/hitkeep.db\nhits, events, goals,\nfunnels, rollups"]
    TeamB["Team data plane\n[data-path]/tenants/team-b/hitkeep.db\nhits, events, goals,\nfunnels, rollups"]

    Control --> SiteMap
    SiteMap --> DefaultPlane
    SiteMap --> TeamA
    SiteMap --> TeamB

The important rule is simple: identity and ownership metadata live in the control plane; analytics rows live in the resolved data plane.

2. Ingestion Buffering: Embedded NSQ

Writing to a columnar database synchronously per HTTP request creates lock contention when traffic spikes. HitKeep avoids that by embedding NSQ, a distributed messaging platform, in-process, and by flushing queued analytics rows to DuckDB in short appender batches instead of issuing one SQL INSERT per message.

• Decoupling: The ingest HTTP handler validates and enqueues the hit in memory, completing the request in microseconds.
• Burst absorption: Traffic spikes (a site goes viral, a product launches) queue up without database pressure. The writer consumes at a steady, optimal pace.
• Store-local batching: The consumer resolves the destination data plane first, then groups queued hits and events per DuckDB store before flushing them.
• Columnar-friendly writes: DuckDB’s appender API bypasses per-row SQL parsing and matches the database’s preferred bulk-ingest path.
• Zero configuration: NSQ runs inside the process on configurable loopback ports. No Kafka cluster, no broker to manage.

NSQ’s TCP and HTTP interfaces bind to 127.0.0.1 by default and are not exposed externally.

In practice, the write path now looks like this:

1. HTTP handler validates and publishes the hit or event to NSQ.
2. One of the in-process consumers picks up the message.
3. The consumer resolves the correct tenant analytics store for the site_id.
4. Messages targeting the same store are coalesced into a small batch for a short time window.
5. The batch is flushed through DuckDB’s appender API.
6. Only after the flush succeeds are the corresponding NSQ messages acknowledged.

Read and Write Paths

The read path and write path both go through the same tenant resolution layer.

Read operations resolve ownership first, then open the correct analytics data plane:

flowchart TD
    ReadClient["Dashboard, API client,\nshare link, export,\nor worker"] --> ReadHTTP["Go HTTP server"]
    ReadHTTP --> ReadAuth["Validate identity\nand site permission"]
    ReadAuth --> ReadControl["Shared control plane\nresolve user, team, site"]
    ReadControl --> ReadResolver["Tenant Store Manager"]
    ReadResolver --> ReadData["Resolved DuckDB data plane"]
    ReadData --> ReadResponse["JSON response\nor export"]

Browser and server-side ingest writes acknowledge the HTTP request before the DuckDB batch flush:

flowchart TD
    WriteClient["Browser tracker\nor server-side caller"] --> WriteHTTP["Ingest handler"]
    WriteHTTP --> WriteAuth["Validate request\nand site access"]
    WriteAuth --> WriteQueue["Embedded NSQ"]
    WriteQueue --> WriteConsumer["Ingest consumer"]
    WriteConsumer --> WriteResolver["Tenant Store Manager"]
    WriteResolver --> WriteBatch["Appender micro-batch"]
    WriteBatch --> WriteData["Resolved DuckDB data plane"]

That applies to:

• dashboard reads
• API exports
• share link reads
• API clients
• ecommerce analytics
• opportunity detectors and saved recommendations
• background workers
• site transfer copy/delete operations

3. Clustering: Leader / Follower

For high availability, HitKeep supports a Leader/Follower topology using HashiCorp Memberlist (gossip protocol) for node discovery.

Roles

Role	Behavior
Leader	Opens the control-plane and tenant DuckDB files, runs embedded NSQ, consumers, workers, MCP, and stateful API handlers
Follower	Serves the HTTP process without stateful stores; proxies browser /ingest and /ingest/event requests to the leader

There is exactly one Leader at any time. If the Leader goes down, the cluster elects a new one — provided the PVC or data volume can be re-attached (Kubernetes StatefulSets handle this automatically).

Request Flow

Load Balancer → Follower → [proxy /ingest or /ingest/event] → Leader → NSQ → DuckDB
Load Balancer → Leader   → [dashboard/API/MCP/stateful handlers] → Tenant Store Manager → DuckDB

For single-server deployments (Docker Compose, systemd), the node acts as Leader implicitly.

Optional MCP Route

The optional MCP route is mounted on the main HitKeep HTTP server. It is disabled by default and is registered only on the leader after leader services are ready.

MCP requests use the Streamable HTTP transport and authenticate with existing API client bearer tokens. They do not accept dashboard cookies. Analytics reads pass through the same API client permission checks and tenant store resolution used by REST handlers.

The v1 MCP surface is read-only and aggregate-only. It can list visible sites, return overview, event, ecommerce, Web Vitals, AI visibility, saved Opportunities, and imported Search Console analytics, and expose local help resources. Where a report supports visitor context, MCP can return city/provider/ASN aggregate breakdowns without exposing raw IP addresses, user agents, session IDs, or visitor rows. Docs tools may fetch official HitKeep documentation as markdown from the configured docs origin, but analytics data is not sent to the docs site. Docs markdown is cached in a bounded in-memory LRU cache.

flowchart TD
    Assistant["MCP Client"] -->|Bearer token| MCP["Leader HTTP server\n/mcp route"]
    MCP --> Authz["API client auth\nsite.view checks\nAPI rate limiter"]
    Authz --> Resolver["Tenant Store Manager"]
    Resolver --> Data["Resolved DuckDB analytics store"]
    MCP -->|Accept: text/markdown\nonly on docs tool calls| Docs["Official docs origin"]

Optional AI Model Route

Optional AI provider calls are disabled by default. When enabled, HitKeep calls the configured provider, model, or OpenAI-compatible gateway route only for features that explicitly use AI.

The first feature is Opportunity Recommendations. Deterministic detectors read tenant-local analytics first and decide the opportunity type, evidence, impact, confidence, score, and status. HitKeep accepts only validated translation keys, params, cited evidence IDs, and safe structured output from AI enrichment. Raw prompts, raw provider payloads, and provider secrets are not persisted.

4. Security Layer

The HTTP server enforces multiple security controls before any data is processed:

• Rate limiting: Per-IP token bucket limiters on /ingest, /api/*, and /api/login. Configurable rate and burst.
• Sec-Fetch validation: Checks Sec-Fetch-* headers on state-changing requests.
• JWT authentication: HTTP-only cookies signed with a configurable secret. Short expiry by default.
• WebAuthn: FIDO2/Passkey challenge-response for passwordless login.
• TOTP: RFC 6238 time-based one-time passwords for second-factor auth.

Authentication Modes

HitKeep currently supports three main access modes into the same API surface:

flowchart TD
    subgraph Credentials["Access modes"]
        Session["Session cookie"]
        Personal["Personal API client"]
        Team["Team API client"]
        Share["Share token"]
    end

    Session --> Authz["Auth + permissions"]
    Personal --> Authz
    Team --> Authz
    Share --> ShareRead["Read-only share handlers"]

    Authz --> Control["Shared control plane"]
    Authz --> Resolver["Tenant Store Manager"]
    ShareRead --> Resolver

• Session cookies are for interactive users in the dashboard.
• Personal API clients are user-owned bearer tokens.
• Team API clients are tenant-owned bearer tokens that survive individual user departure.
• Share tokens are read-only public-style analytics access scoped to one shared site/dashboard view.

The optional MCP server reuses API client bearer tokens only. This makes MCP access revocable through the same API client lifecycle as other server-to-server integrations.

5. Frontend Architecture

The dashboard is a Single Page Application served from the same binary:

• Framework: Angular v22 with Signals for reactive state
• UI library: PrimeNG with Tailwind CSS
• API contract: All dashboard functionality uses the same JSON REST API as external clients
• Tracking snippet: hk.js is minified with esbuild and served from your instance. It uses sendBeacon() with a keepalive fetch fallback, in-memory retry and dedupe state, and only stores the existing opaque session tuple in sessionStorage. Web Vitals stay in a separate opt-in hk-vitals.js bundle. See Tracker Architecture.

The dashboard itself is now multi-context:

• active site
• active team
• user/session state
• permission state
• analytics filters and date range

Those contexts stay on the client, but authoritative ownership and access checks always happen server-side in the shared control plane before analytics data is read.

6. Data Lifecycle and Recovery

HitKeep now has three distinct storage lifecycle concepts and they should not be confused:

1. Live databases for current reads and writes
2. Retention archives for old analytics rows exported to Parquet
3. Backup snapshots / purge flows for disaster recovery and irreversible cleanup

flowchart TD
    Live["Live control plane + tenant DBs"] --> Retention["Retention worker\narchive old analytics to Parquet"]
    Live --> Backup["Backup worker\nEXPORT DATABASE snapshots"]
    ArchivedTeam["Archived team"] --> Purge["Admin purge\nDELETE /api/admin/teams/:team_id"]
    Purge --> Remove["Remove control-plane metadata\nand tenants/team_id/hitkeep.db"]

Important distinctions:

• -archive-path is for Parquet retention archives and related artifacts.
• -data-path is where live tenant DuckDB files live.
• archiving a team is a reversible control-plane state until you run the purge path.
• purging an archived team is irreversible and removes its live tenant database directory.

7. Ecommerce in the Architecture

Ecommerce is not a separate subsystem. It is an opinionated query layer over the same tenant-local events tables.

flowchart TD
    Tracker["hk.js / server ingest"] --> Events["events table"]
    Events --> Normalizer["GA4-inspired\nevent normalization"]
    Normalizer --> Queries["Revenue / products / sources queries"]
    Queries --> Page["Ecommerce dashboard page"]

That means:

• ecommerce data inherits team isolation automatically
• ecommerce filters reuse the same site/session attribution model
• ecommerce backups, restores, transfers, and retention follow the same tenant data-plane rules as other analytics data

What the Binary Contains

hitkeep (single executable)
├── Go HTTP server
├── Embedded NSQ broker + consumer
├── DuckDB engine + SQL migrations
├── Angular dashboard (compiled, embedded)
├── hk.js tracking snippet (embedded)
├── Optional MCP route (leader only)
├── Optional AI model route (disabled by default)
└── Background workers (retention, rollups, reports, imports, backups)

Every layer is auditable. The full source is on GitHub under the MIT license.

Related

• Configuration Reference
• Teams and Data Isolation
• Installation Guide
• Trusted Proxies
• Opportunity Recommendations

HitKeep Cloud runs this exact binary stack in managed EU (Frankfurt) or US (Virginia) infrastructure, with the same source-visible product foundation as self-hosted deployments. Start with HitKeep Cloud →