---
title: "HitKeep Disaster Recovery Runbook for Operators | HitKeep"
description: "Plan and verify HitKeep recovery for single-tenant and multiteam deployments, including backups, archives, and tenant-local analytics data."
canonical: "https://hitkeep.com/guides/data/disaster-recovery/"
---

# HitKeep Disaster Recovery Runbook for Operators

Disaster recovery for HitKeep is straightforward if you treat the storage layout correctly.

The most common mistake is restoring only the shared control-plane database while forgetting tenant-local analytics files.

## What You Need To Recover

At minimum:

- the control-plane database at `HITKEEP_DB_PATH`
- all tenant analytics databases in `{data-path}/tenants/**`, including the default tenant
- the archive directory if you rely on retention archives for older raw data

If you use built-in backups, these are already exported into snapshot directories. If you use external tooling, your DR plan should capture the same boundary.

## Recovery Scenarios

Plan for the failure you are most likely to face:

| Scenario | Recovery source | Main risk |
| --- | --- | --- |
| Bad deploy | Latest local backup | Restoring a snapshot from before a schema migration |
| Disk loss | Off-host backup or S3 backup | Missing tenant-local databases |
| Accidental team deletion | Backup from before purge | Retaining deleted tenant data longer than policy allows |
| Host migration | Latest verified snapshot | Forgetting archive and asset directories |
| Region outage | Object storage or external snapshot | Restore time and DNS cutover |

## Recovery Drill Checklist

Run this periodically on a disposable environment:

1. Provision an empty host or container.
2. Restore HitKeep from a recent snapshot.
3. Start the same HitKeep version, or a newer compatible one.
4. Log in as an admin.
5. Validate one default-tenant site.
6. Validate one non-default team site.
7. Validate goals, funnels, and ecommerce.
8. Confirm team membership and team switching still work.
9. Confirm retention archives are still present if you keep them separately.

If you cannot perform this drill successfully, you do not yet have a reliable recovery process.

## Recommended Restore Command

```
./hitkeep recover restore-backup \
  -from /var/lib/hitkeep/backups \
  -snapshot 2026-03-08T120000Z \
  -db /var/lib/hitkeep/data/hitkeep.db \
  -data-path /var/lib/hitkeep/data \
  -yes
```

Restore is offline-only. Stop HitKeep before running it.

## Automatic DuckDB Recovery

HitKeep recognizes two narrow DuckDB failures:

- an explicit-index invalidation with the exact “failed to delete all rows from index” signature
- a known WAL replay failure involving an `ALTER` default expression

For the index failure, HitKeep writes a compressed recovery bundle and removes the non-unique secondary indexes from the table whose mutation failed. If DuckDB reports only the later, context-free invalidation error, recovery falls back to removing all non-unique secondary indexes in that database. Primary-key and unique indexes remain intact. HitKeep then checkpoints the database and can resume every repair phase from a local marker.

If no matching non-unique index exists, recovery fails closed with `database_needs_attention` instead of reporting a successful zero-index repair. Upgrade migrations prevent the known Search Console and activity-summary variants by removing those tables’ non-unique indexes before normal writes resume. These indexes are query accelerators; site deletion, tenant cleanup, and retention use schema scope and policy rather than depending on them.

For the recognized WAL failure, HitKeep first writes a compressed recovery bundle under `HITKEEP_DB_RECOVERY_PATH` and then stops by default. The bundle contains the exact database and WAL state, checksums, and sanitized recovery metadata. Schema migrations use a narrower startup path: each pending migration commits independently, heavy analytics-index sections release their native DuckDB instance before the next section, and a durable guard records the checksum of the closed base database while the migration WAL remains authoritative. On reconnect, HitKeep verifies that checksum before replaying and checkpointing the migration WAL. The guard remains until the complete pending migration set succeeds, so partial progress resumes idempotently without exposing the database to workers.

Set `HITKEEP_DB_AUTO_RECOVER_WAL=true` only when your availability policy accepts losing committed application changes that exist solely in an unreplayable WAL. Verified migration-only recovery does not require this setting because the guarded migration runs before the database is published to application workers and the closed base-file checksum must match. If that checksum does not match, HitKeep preserves the live WAL and falls back to the normal recovery path. For other recognized WAL failures, the explicit opt-in lets HitKeep move the WAL aside, checkpoint the base database without it, and resume safely from each filesystem phase after interruption. Unknown DuckDB errors are never bypassed automatically.

On Unix-like systems, bundle creation fails closed unless the recovery filesystem has free space roughly equal to the database plus WAL plus the larger of 512 MiB or 10% of the database size. The live database and WAL are not mutated until the compressed artifacts and manifest have been written and synchronized.

If automatic WAL bypass is disabled and your incident policy approves continuing without WAL-only changes:

1. Keep HitKeep stopped.
2. Confirm that the retained bundle contains `manifest.json`, `database.zst`, and `wal.zst`.
3. Move the live `hitkeep.db.wal` out of the database directory into protected incident storage; do not delete the retained recovery bundle.
4. Start HitKeep again. The resumable marker recognizes that the replay-failing live WAL is absent, completes recovery, and checkpoints the base database.

If you are not prepared to accept that loss boundary, leave the live files unchanged and restore a known-good backup instead.

During recovery of the shared database or an open tenant database, `/healthz` remains a process-liveness check while `/readyz` returns `503`, a stable recovery reason, and `Retry-After: 5`. API and ingest requests guarded by database availability also return `503` instead of being misreported as missing resources.

Recovery bundles are never deleted automatically. They contain the same sensitive data as the live database and must follow your normal access-control, retention, and deletion policy.

To roll back to a retained pre-recovery bundle, stop HitKeep and run:

```
./hitkeep recover restore-database-bundle \
  -from /var/lib/hitkeep/data/recovery/20260716T120000.000000000Z-0123456789abcdef \
  -db /var/lib/hitkeep/data/hitkeep.db \
  -yes
```

This command verifies every artifact checksum and restores the exact database and WAL files. That can intentionally restore the original failure, so use it for rollback or forensic work—not as a substitute for a healthy backup.

If recovery cannot safely continue, or an invalidated database cannot drain its open connections, HitKeep performs a controlled shutdown instead of serving partial results. Run it under a restart policy such as systemd, Docker Compose, or a Kubernetes controller, and investigate repeated restart or `database_needs_attention` states rather than bypassing the failure.

## Team and Archive Lifecycle Considerations

Teams introduce two important operational facts:

- archived teams can later be purged physically
- tenant analytics may live outside the shared database

That means:

- backups taken before a purge may still contain the purged tenant
- backups taken after a purge should not
- archive retention and backup retention are separate concerns

If you have GDPR or hard-deletion requirements, your DR runbooks should explicitly define how long old snapshots are retained and when they are expired.

## What Success Looks Like

A good HitKeep DR posture means:

- you know exactly where live data lives
- you know exactly where backups are written
- you have tested `recover restore-backup`
- you have documented who may inspect or remove automatic-recovery bundles
- you can restore both shared and tenant-local data
- you are not depending on replaying a stale WAL to make a restore boot

## How often to test

For small self-hosted installs, run a restore drill after changing backup storage or before a major version upgrade. For teams using HitKeep as client or business reporting infrastructure, run a scheduled drill at least quarterly and record the snapshot timestamp, HitKeep version, restore target, and validation result.

## Related

- [Backups and Restore](https://hitkeep.com/guides/data/backups-and-restore/)
- [S3 Backups](https://hitkeep.com/guides/data/s3-backups/)
- [Teams](https://hitkeep.com/guides/admin/teams/)
- [Architecture](https://hitkeep.com/reference/architecture/)

[Previous Data retention and archiving](https://hitkeep.com/guides/data/retention/)[Next Open exports and takeout](https://hitkeep.com/guides/data/takeout/)