---
title: "HitKeep Data Retention and Parquet Archiving | HitKeep"
description: "Configure how long HitKeep keeps raw analytics data globally or per site, then archive stale rows to open Parquet before pruning."
canonical: "https://hitkeep.com/guides/data/retention/"
---

# HitKeep Data Retention and Parquet Archiving

You decide how long your analytics data lives — not a cloud vendor’s pricing tier. HitKeep’s retention system follows one rule: data you choose to prune is **archived to Parquet first**, in an open format you own, before it’s removed from the live database.

| Retention fact | HitKeep behavior |
| --- | --- |
| Raw retention | Configurable globally and per site |
| Archive format | Parquet |
| Archive trigger | Rows are archived before pruning from the live database |
| Backup format | DuckDB EXPORT DATABASE, including schema.sql and Parquet table files |
| Storage target | Local path or S3-compatible path when configured |
| Broader runtime facts | See Facts and Limits |

## Quick Start

```
# Keep raw hits and events for 365 days; archive older data to /var/lib/hitkeep/archive
export HITKEEP_DATA_RETENTION_DAYS=365
export HITKEEP_ARCHIVE_PATH=/var/lib/hitkeep/archive

./hitkeep
```

Or as startup flags:

```
./hitkeep --data-retention-days=365 --archive-path=/var/lib/hitkeep/archive
```

See the [Configuration Reference](https://hitkeep.com/reference/configuration/#data-management) for all options.

## How It Works

The retention worker runs once daily. For each site with a configured retention policy it will:

1. **Count** expired native traffic, events, Web Vitals, AI fetches, QR opens, imported facts, Search Console facts, and rollups in the site’s tenant catalog.
2. **Export** those rows to a compressed Parquet file in the archive directory before touching the live database.
3. **Prune** the archived records and expired dirty-rollup buckets from the tenant database.
4. **Leave activity maintenance independent.** Activity-hourly rows follow their maintenance policy rather than analytics retention.

The result is two data tiers:

| Tier | Location | What’s there | Query speed |
| --- | --- | --- | --- |
| Hot | tenants/{tenant_id}/hitkeep.db | Recent native, imported, Search Console, and rollup data within the retention window | Instant |
| Cold | Archive directory | Older analytics rows exported to Parquet | Fast (file scan) |

Archived hit rows keep the same derived IP metadata columns as live hit exports: region, city, provider, ASN, and ASN organization. Raw visitor IP addresses are not added to retention archives.

Dashboard trend history follows the configured retention window; expired rollups are archived and pruned with their source data.

## Per-Site Overrides

Different sites have different requirements. Override the default retention window per site via the API:

```
curl -X PUT https://your-hitkeep.example/api/sites/{site_id}/retention \
  -H "Content-Type: application/json" \
  -b "hk_token=YOUR_SESSION_COOKIE" \
  -d '{"days": 90}'
```

A high-traffic site may need only 90 days of raw data. A site subject to statutory record-keeping requirements may need seven years. You set the policy; HitKeep enforces it.

## Archive Destination Overview

The retention worker and backup worker can write to local disk or any S3-compatible object store. Both share the same S3 credential configuration.

## Archiving to S3

Instead of writing Parquet files to a local directory, HitKeep can archive directly to any S3-compatible object store. Set `HITKEEP_ARCHIVE_PATH` to an `s3://` URL and configure credentials.

### AWS S3 with Static Keys

```
export HITKEEP_ARCHIVE_PATH=s3://my-analytics-bucket/hitkeep/archive
export HITKEEP_S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export HITKEEP_S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export HITKEEP_S3_REGION=eu-west-1

./hitkeep
```

### AWS S3 with IAM Role (Credential Chain)

On EC2, ECS, or Lambda with an attached IAM role, no explicit keys are needed. HitKeep falls back to the AWS SDK default credential chain automatically.

```
export HITKEEP_ARCHIVE_PATH=s3://my-analytics-bucket/hitkeep/archive
export HITKEEP_S3_REGION=eu-west-1

./hitkeep
# Logs: "S3 archive enabled" mode="credential chain" region="eu-west-1"
```

### MinIO (Custom Endpoint)

```
export HITKEEP_ARCHIVE_PATH=s3://hitkeep-archive/data
export HITKEEP_S3_ACCESS_KEY_ID=minioadmin
export HITKEEP_S3_SECRET_ACCESS_KEY=minioadmin
export HITKEEP_S3_ENDPOINT=localhost:9000
export HITKEEP_S3_URL_STYLE=path
export HITKEEP_S3_USE_SSL=false
export HITKEEP_S3_REGION=us-east-1

./hitkeep
```

### Cloudflare R2

```
export HITKEEP_ARCHIVE_PATH=s3://my-r2-bucket/hitkeep/archive
export HITKEEP_S3_ACCESS_KEY_ID=your-r2-access-key
export HITKEEP_S3_SECRET_ACCESS_KEY=your-r2-secret-key
export HITKEEP_S3_ENDPOINT=your-account-id.r2.cloudflarestorage.com
export HITKEEP_S3_REGION=auto

./hitkeep
```

See the [Configuration Reference](https://hitkeep.com/reference/configuration/#s3-archive-storage) for the full list of S3 settings.

## Querying Cold Data

Archived Parquet files are standard open-format files queryable with any compatible tool — no HitKeep license required.

```
# DuckDB CLI — count page views per month from the archive
duckdb -c "
  SELECT date_trunc('month', timestamp) AS month, count(*) AS hits
  FROM read_parquet('/var/lib/hitkeep/archive/site_*.parquet')
  GROUP BY 1 ORDER BY 1;
"
```

```
# Merge hot and cold data in a single query
duckdb -c "
  ATTACH 'hitkeep.db' AS hot;
  SELECT timestamp::date AS day, count(*) AS hits
  FROM (
    SELECT timestamp FROM hot.hits WHERE site_id = 'your-site-id'
    UNION ALL
    SELECT timestamp FROM read_parquet('/var/lib/hitkeep/archive/site_your-site-id_*.parquet')
  )
  GROUP BY 1 ORDER BY 1;
"
```

The archive naming convention is `site_{site_id}_{unix_timestamp}.parquet`. Each archival run writes one file per site that had data past the cutoff.

## Database Backups

HitKeep includes a built-in backup worker that periodically exports all live databases to Parquet snapshots using DuckDB’s `EXPORT DATABASE`. This covers both the shared `hitkeep.db` and any per-tenant databases.

For a dedicated operational guide, see [Backups and Restore](https://hitkeep.com/guides/data/backups-and-restore/) and [S3 Backups](https://hitkeep.com/guides/data/s3-backups/).

### Backup Worker Lifecycle

### Enabling Backups

```
# Local backups — every 60 minutes, keep 24 snapshots
export HITKEEP_BACKUP_PATH=/var/lib/hitkeep/backups

./hitkeep
# Logs: "Local backup enabled" path="/var/lib/hitkeep/backups" interval_min=60 retention=24
```

```
# S3 backups — every 30 minutes, keep 48 snapshots
export HITKEEP_BACKUP_PATH=s3://my-bucket/hitkeep/backups
export HITKEEP_BACKUP_INTERVAL=30
export HITKEEP_BACKUP_RETENTION=48

./hitkeep
```

The worker runs on the leader node only. The first backup is taken 30 seconds after startup, then at the configured interval.

### Backup Layout

Each backup is a timestamped directory containing the output of DuckDB’s `EXPORT DATABASE` (a `schema.sql` file plus Parquet data files for each table).

Or as a directory tree:

```
{backup-path}/
├── shared/
│   ├── 2026-03-02T120000Z/    ← schema.sql + *.parquet
│   └── 2026-03-02T130000Z/
└── tenants/
    └── {tenant-id}/
        ├── 2026-03-02T120000Z/
        └── 2026-03-02T130000Z/
```

### Snapshot Pruning

For local backups, snapshots beyond the retention count are automatically deleted (oldest first). For S3 backups, configure [S3 lifecycle policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) on your bucket to manage snapshot retention.

### Restoring from a Backup

Use the `hitkeep recover restore-backup` command to import a snapshot into fresh databases. HitKeep **must be stopped** before running a restore (DuckDB allows only one writer at a time).

```
# Restore the latest local snapshot
./hitkeep recover restore-backup \
  -from /var/lib/hitkeep/backups \
  -yes
```

```
# Restore a specific snapshot
./hitkeep recover restore-backup \
  -from /var/lib/hitkeep/backups \
  -snapshot 2026-03-02T120000Z \
  -db /var/lib/hitkeep/data/hitkeep.db \
  -data-path /var/lib/hitkeep/data \
  -yes
```

```
# Restore from S3 (snapshot timestamp required)
./hitkeep recover restore-backup \
  -from s3://my-bucket/hitkeep/backups \
  -snapshot 2026-03-02T120000Z \
  -yes
```

The restore process:

1. Finds the requested snapshot (or the latest one for local sources).
2. Renames existing database files as a safety net (`.pre-restore.{timestamp}`).
3. Imports the snapshot into temporary DuckDB files, checkpoints them, and only then promotes them into place.
4. Discovers and restores any tenant databases from the backup.

On the next normal `hitkeep` startup, the migration system will apply any schema changes if HitKeep has been upgraded since the backup was taken.

See the [Configuration Reference](https://hitkeep.com/reference/configuration/#database-backups) for all backup settings.

## Backup Strategy (Manual)

For users who prefer external tooling, the complete HitKeep data footprint is:

- **Live data tree:** the full `data-path` directory, including the shared control plane and any tenant-local databases
- **Archive:** the configured archive directory (Parquet files)

A reliable backup is a periodic file copy of both:

```
# Example: nightly sync to S3-compatible storage with rclone
rclone sync /var/lib/hitkeep/data/ remote:my-bucket/hitkeep/data/
rclone sync /var/lib/hitkeep/archive/ remote:my-bucket/hitkeep/archive/
```

```
# Or with rsync to a remote host
rsync -az /var/lib/hitkeep/ backup-host:/backups/hitkeep/
```

Because HitKeep stores its live state on the local filesystem, you can also use filesystem-level snapshots (LVM, ZFS, APFS) for point-in-time consistency — but snapshot the full `data-path`, not just the root database file.

## Complete Data Lifecycle

The following diagram shows how data moves through HitKeep — from ingestion to hot storage, through retention archiving to cold storage, and how backups and restores fit into the picture.

The key insight: **retention archives** and **database backups** serve different purposes. Retention exports are per-site, incremental Parquet files for long-term analytical querying. Database backups are full point-in-time snapshots for disaster recovery. Both can target local disk or S3.

## Related

- [Analytics with Open Exports](https://hitkeep.com/guides/data/takeout/)
- [Configuration Reference](https://hitkeep.com/reference/configuration/)
- [REST API Reference](https://hitkeep.com/api/)

HitKeep Cloud manages retention policies, automated Parquet archiving, and encrypted off-site backups in your chosen managed region (EU Frankfurt or US Virginia). [Start with HitKeep Cloud →](https://cloud.hitkeep.eu/signup?plan=free&billing=monthly&utm_source=hitkeep_docs&utm_medium=referral&utm_campaign=cloud_signup&utm_content=docs_inline)

[Previous S3 backups](https://hitkeep.com/guides/data/s3-backups/)[Next Disaster recovery](https://hitkeep.com/guides/data/disaster-recovery/)