HitKeep Data Retention and Parquet Archiving

You decide how long your analytics data lives — not a cloud vendor’s pricing tier. HitKeep’s retention system follows one rule: data you choose to prune is archived to Parquet first, in an open format you own, before it’s removed from the live database.

Retention fact	HitKeep behavior
Raw retention	Configurable globally and per site
Archive format	Parquet
Archive trigger	Rows are archived before pruning from the live database
Backup format	DuckDB EXPORT DATABASE, including schema.sql and Parquet table files
Storage target	Local path or S3-compatible path when configured
Broader runtime facts	See Facts and Limits

Quick Start

# Keep raw hits and events for 365 days; archive older data to /var/lib/hitkeep/archive
export HITKEEP_DATA_RETENTION_DAYS=365
export HITKEEP_ARCHIVE_PATH=/var/lib/hitkeep/archive

./hitkeep

Or as startup flags:

./hitkeep --data-retention-days=365 --archive-path=/var/lib/hitkeep/archive

See the Configuration Reference for all options.

How It Works

The retention worker runs once daily. For each site with a configured retention policy it will:

1. Count expired native traffic, events, Web Vitals, AI fetches, QR opens, imported facts, Search Console facts, and rollups in the site’s tenant catalog.
2. Export those rows to a compressed Parquet file in the archive directory before touching the live database.
3. Prune the archived records and expired dirty-rollup buckets from the tenant database.
4. Leave activity maintenance independent. Activity-hourly rows follow their maintenance policy rather than analytics retention.

The result is two data tiers:

Tier	Location	What’s there	Query speed
Hot	tenants/{tenant_id}/hitkeep.db	Recent native, imported, Search Console, and rollup data within the retention window	Instant
Cold	Archive directory	Older analytics rows exported to Parquet	Fast (file scan)

Archived hit rows keep the same derived IP metadata columns as live hit exports: region, city, provider, ASN, and ASN organization. Raw visitor IP addresses are not added to retention archives.

Dashboard trend history follows the configured retention window; expired rollups are archived and pruned with their source data.

Per-Site Overrides

Different sites have different requirements. Override the default retention window per site via the API:

curl -X PUT https://your-hitkeep.example/api/sites/{site_id}/retention \
  -H "Content-Type: application/json" \
  -b "hk_token=YOUR_SESSION_COOKIE" \
  -d '{"days": 90}'

A high-traffic site may need only 90 days of raw data. A site subject to statutory record-keeping requirements may need seven years. You set the policy; HitKeep enforces it.

Archive Destination Overview

The retention worker and backup worker can write to local disk or any S3-compatible object store. Both share the same S3 credential configuration.

Archiving to S3

Instead of writing Parquet files to a local directory, HitKeep can archive directly to any S3-compatible object store. Set HITKEEP_ARCHIVE_PATH to an s3:// URL and configure credentials.

AWS S3 with Static Keys

export HITKEEP_ARCHIVE_PATH=s3://my-analytics-bucket/hitkeep/archive
export HITKEEP_S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export HITKEEP_S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export HITKEEP_S3_REGION=eu-west-1

./hitkeep

AWS S3 with IAM Role (Credential Chain)

On EC2, ECS, or Lambda with an attached IAM role, no explicit keys are needed. HitKeep falls back to the AWS SDK default credential chain automatically.

export HITKEEP_ARCHIVE_PATH=s3://my-analytics-bucket/hitkeep/archive
export HITKEEP_S3_REGION=eu-west-1

./hitkeep
# Logs: "S3 archive enabled" mode="credential chain" region="eu-west-1"

MinIO (Custom Endpoint)

export HITKEEP_ARCHIVE_PATH=s3://hitkeep-archive/data
export HITKEEP_S3_ACCESS_KEY_ID=minioadmin
export HITKEEP_S3_SECRET_ACCESS_KEY=minioadmin
export HITKEEP_S3_ENDPOINT=localhost:9000
export HITKEEP_S3_URL_STYLE=path
export HITKEEP_S3_USE_SSL=false
export HITKEEP_S3_REGION=us-east-1

./hitkeep

Cloudflare R2

export HITKEEP_ARCHIVE_PATH=s3://my-r2-bucket/hitkeep/archive
export HITKEEP_S3_ACCESS_KEY_ID=your-r2-access-key
export HITKEEP_S3_SECRET_ACCESS_KEY=your-r2-secret-key
export HITKEEP_S3_ENDPOINT=your-account-id.r2.cloudflarestorage.com
export HITKEEP_S3_REGION=auto

./hitkeep

See the Configuration Reference for the full list of S3 settings.

Querying Cold Data

Archived Parquet files are standard open-format files queryable with any compatible tool — no HitKeep license required.

# DuckDB CLI — count page views per month from the archive
duckdb -c "
  SELECT date_trunc('month', timestamp) AS month, count(*) AS hits
  FROM read_parquet('/var/lib/hitkeep/archive/site_*.parquet')
  GROUP BY 1 ORDER BY 1;
"

# Merge hot and cold data in a single query
duckdb -c "
  ATTACH 'hitkeep.db' AS hot;
  SELECT timestamp::date AS day, count(*) AS hits
  FROM (
    SELECT timestamp FROM hot.hits WHERE site_id = 'your-site-id'
    UNION ALL
    SELECT timestamp FROM read_parquet('/var/lib/hitkeep/archive/site_your-site-id_*.parquet')
  )
  GROUP BY 1 ORDER BY 1;
"

The archive naming convention is site_{site_id}_{unix_timestamp}.parquet. Each archival run writes one file per site that had data past the cutoff.

Database Backups

HitKeep includes a built-in backup worker that periodically exports all live databases to Parquet snapshots using DuckDB’s EXPORT DATABASE. This covers both the shared hitkeep.db and any per-tenant databases.

For a dedicated operational guide, see Backups and Restore and S3 Backups.

Backup Worker Lifecycle

Enabling Backups

# Local backups — every 60 minutes, keep 24 snapshots
export HITKEEP_BACKUP_PATH=/var/lib/hitkeep/backups

./hitkeep
# Logs: "Local backup enabled" path="/var/lib/hitkeep/backups" interval_min=60 retention=24

# S3 backups — every 30 minutes, keep 48 snapshots
export HITKEEP_BACKUP_PATH=s3://my-bucket/hitkeep/backups
export HITKEEP_BACKUP_INTERVAL=30
export HITKEEP_BACKUP_RETENTION=48

./hitkeep

The worker runs on the leader node only. The first backup is taken 30 seconds after startup, then at the configured interval.

Backup Layout

Each backup is a timestamped directory containing the output of DuckDB’s EXPORT DATABASE (a schema.sql file plus Parquet data files for each table).

Or as a directory tree:

{backup-path}/
├── shared/
│   ├── 2026-03-02T120000Z/    ← schema.sql + *.parquet
│   └── 2026-03-02T130000Z/
└── tenants/
    └── {tenant-id}/
        ├── 2026-03-02T120000Z/
        └── 2026-03-02T130000Z/

Snapshot Pruning

For local backups, snapshots beyond the retention count are automatically deleted (oldest first). For S3 backups, configure S3 lifecycle policies on your bucket to manage snapshot retention.

Restoring from a Backup

Use the hitkeep recover restore-backup command to import a snapshot into fresh databases. HitKeep must be stopped before running a restore (DuckDB allows only one writer at a time).

# Restore the latest local snapshot
./hitkeep recover restore-backup \
  -from /var/lib/hitkeep/backups \
  -yes

# Restore a specific snapshot
./hitkeep recover restore-backup \
  -from /var/lib/hitkeep/backups \
  -snapshot 2026-03-02T120000Z \
  -db /var/lib/hitkeep/data/hitkeep.db \
  -data-path /var/lib/hitkeep/data \
  -yes

# Restore from S3 (snapshot timestamp required)
./hitkeep recover restore-backup \
  -from s3://my-bucket/hitkeep/backups \
  -snapshot 2026-03-02T120000Z \
  -yes

The restore process:

1. Finds the requested snapshot (or the latest one for local sources).
2. Renames existing database files as a safety net (.pre-restore.{timestamp}).
3. Imports the snapshot into temporary DuckDB files, checkpoints them, and only then promotes them into place.
4. Discovers and restores any tenant databases from the backup.

On the next normal hitkeep startup, the migration system will apply any schema changes if HitKeep has been upgraded since the backup was taken.

See the Configuration Reference for all backup settings.

Backup Strategy (Manual)

For users who prefer external tooling, the complete HitKeep data footprint is:

• Live data tree: the full data-path directory, including the shared control plane and any tenant-local databases
• Archive: the configured archive directory (Parquet files)

A reliable backup is a periodic file copy of both:

# Example: nightly sync to S3-compatible storage with rclone
rclone sync /var/lib/hitkeep/data/ remote:my-bucket/hitkeep/data/
rclone sync /var/lib/hitkeep/archive/ remote:my-bucket/hitkeep/archive/

# Or with rsync to a remote host
rsync -az /var/lib/hitkeep/ backup-host:/backups/hitkeep/

Because HitKeep stores its live state on the local filesystem, you can also use filesystem-level snapshots (LVM, ZFS, APFS) for point-in-time consistency — but snapshot the full data-path, not just the root database file.

Complete Data Lifecycle

The following diagram shows how data moves through HitKeep — from ingestion to hot storage, through retention archiving to cold storage, and how backups and restores fit into the picture.

The key insight: retention archives and database backups serve different purposes. Retention exports are per-site, incremental Parquet files for long-term analytical querying. Database backups are full point-in-time snapshots for disaster recovery. Both can target local disk or S3.

Related

• Analytics with Open Exports
• Configuration Reference
• REST API Reference

HitKeep Cloud manages retention policies, automated Parquet archiving, and encrypted off-site backups in your chosen managed region (EU Frankfurt or US Virginia). Start with HitKeep Cloud →