HitKeep Disaster Recovery Runbook for Operators

Disaster recovery for HitKeep is straightforward if you treat the storage layout correctly.

The most common mistake is restoring only the shared control-plane database while forgetting tenant-local analytics files.

What You Need To Recover

At minimum:

the control-plane database at HITKEEP_DB_PATH
all tenant analytics databases in {data-path}/tenants/**, including the default tenant
the archive directory if you rely on retention archives for older raw data

If you use built-in backups, these are already exported into snapshot directories. If you use external tooling, your DR plan should capture the same boundary.

Recovery Scenarios

Plan for the failure you are most likely to face:

Scenario	Recovery source	Main risk
Bad deploy	Latest local backup	Restoring a snapshot from before a schema migration
Disk loss	Off-host backup or S3 backup	Missing tenant-local databases
Accidental team deletion	Backup from before purge	Retaining deleted tenant data longer than policy allows
Host migration	Latest verified snapshot	Forgetting archive and asset directories
Region outage	Object storage or external snapshot	Restore time and DNS cutover

Recovery Drill Checklist

Run this periodically on a disposable environment:

Provision an empty host or container.
Restore HitKeep from a recent snapshot.
Start the same HitKeep version, or a newer compatible one.
Log in as an admin.
Validate one default-tenant site.
Validate one non-default team site.
Validate goals, funnels, and ecommerce.
Confirm team membership and team switching still work.
Confirm retention archives are still present if you keep them separately.

If you cannot perform this drill successfully, you do not yet have a reliable recovery process.

Recommended Restore Command

./hitkeep recover restore-backup \
  -from /var/lib/hitkeep/backups \
  -snapshot 2026-03-08T120000Z \
  -db /var/lib/hitkeep/data/hitkeep.db \
  -data-path /var/lib/hitkeep/data \
  -yes

Restore is offline-only. Stop HitKeep before running it.

Automatic DuckDB Recovery

HitKeep recognizes two narrow DuckDB failures:

an explicit-index invalidation with the exact “failed to delete all rows from index” signature
a known WAL replay failure involving an ALTER default expression

For the index failure, HitKeep writes a compressed recovery bundle and removes the non-unique secondary indexes from the table whose mutation failed. If DuckDB reports only the later, context-free invalidation error, recovery falls back to removing all non-unique secondary indexes in that database. Primary-key and unique indexes remain intact. HitKeep then checkpoints the database and can resume every repair phase from a local marker.

If no matching non-unique index exists, recovery fails closed with database_needs_attention instead of reporting a successful zero-index repair. Upgrade migrations prevent the known Search Console and activity-summary variants by removing those tables’ non-unique indexes before normal writes resume. These indexes are query accelerators; site deletion, tenant cleanup, and retention use schema scope and policy rather than depending on them.

For the recognized WAL failure, HitKeep first writes a compressed recovery bundle under HITKEEP_DB_RECOVERY_PATH and then stops by default. The bundle contains the exact database and WAL state, checksums, and sanitized recovery metadata. Schema migrations use a narrower startup path: each pending migration commits independently, heavy analytics-index sections release their native DuckDB instance before the next section, and a durable guard records the checksum of the closed base database while the migration WAL remains authoritative. On reconnect, HitKeep verifies that checksum before replaying and checkpointing the migration WAL. The guard remains until the complete pending migration set succeeds, so partial progress resumes idempotently without exposing the database to workers.

Set HITKEEP_DB_AUTO_RECOVER_WAL=true only when your availability policy accepts losing committed application changes that exist solely in an unreplayable WAL. Verified migration-only recovery does not require this setting because the guarded migration runs before the database is published to application workers and the closed base-file checksum must match. If that checksum does not match, HitKeep preserves the live WAL and falls back to the normal recovery path. For other recognized WAL failures, the explicit opt-in lets HitKeep move the WAL aside, checkpoint the base database without it, and resume safely from each filesystem phase after interruption. Unknown DuckDB errors are never bypassed automatically.

On Unix-like systems, bundle creation fails closed unless the recovery filesystem has free space roughly equal to the database plus WAL plus the larger of 512 MiB or 10% of the database size. The live database and WAL are not mutated until the compressed artifacts and manifest have been written and synchronized.

If automatic WAL bypass is disabled and your incident policy approves continuing without WAL-only changes:

Keep HitKeep stopped.
Confirm that the retained bundle contains manifest.json, database.zst, and wal.zst.
Move the live hitkeep.db.wal out of the database directory into protected incident storage; do not delete the retained recovery bundle.
Start HitKeep again. The resumable marker recognizes that the replay-failing live WAL is absent, completes recovery, and checkpoints the base database.

If you are not prepared to accept that loss boundary, leave the live files unchanged and restore a known-good backup instead.

During recovery of the shared database or an open tenant database, /healthz remains a process-liveness check while /readyz returns 503, a stable recovery reason, and Retry-After: 5. API and ingest requests guarded by database availability also return 503 instead of being misreported as missing resources.

Recovery bundles are never deleted automatically. They contain the same sensitive data as the live database and must follow your normal access-control, retention, and deletion policy.

To roll back to a retained pre-recovery bundle, stop HitKeep and run:

./hitkeep recover restore-database-bundle \
  -from /var/lib/hitkeep/data/recovery/20260716T120000.000000000Z-0123456789abcdef \
  -db /var/lib/hitkeep/data/hitkeep.db \
  -yes

This command verifies every artifact checksum and restores the exact database and WAL files. That can intentionally restore the original failure, so use it for rollback or forensic work—not as a substitute for a healthy backup.

If recovery cannot safely continue, or an invalidated database cannot drain its open connections, HitKeep performs a controlled shutdown instead of serving partial results. Run it under a restart policy such as systemd, Docker Compose, or a Kubernetes controller, and investigate repeated restart or database_needs_attention states rather than bypassing the failure.

Team and Archive Lifecycle Considerations

Teams introduce two important operational facts:

archived teams can later be purged physically
tenant analytics may live outside the shared database

That means:

backups taken before a purge may still contain the purged tenant
backups taken after a purge should not
archive retention and backup retention are separate concerns

If you have GDPR or hard-deletion requirements, your DR runbooks should explicitly define how long old snapshots are retained and when they are expired.

What Success Looks Like

A good HitKeep DR posture means:

you know exactly where live data lives
you know exactly where backups are written
you have tested recover restore-backup
you have documented who may inspect or remove automatic-recovery bundles
you can restore both shared and tenant-local data
you are not depending on replaying a stale WAL to make a restore boot

How often to test

For small self-hosted installs, run a restore drill after changing backup storage or before a major version upgrade. For teams using HitKeep as client or business reporting infrastructure, run a scheduled drill at least quarterly and record the snapshot timestamp, HitKeep version, restore target, and validation result.