You are viewing archived documentation for v0.44. Go to latest →

Backup & Disaster Recovery

What survives a lost disk, and how to bring the stack back from a backup.

What is backed up

Tier What How Why
PostgreSQL detections, users, tenants, votes, annotations, calibration, audit, settings timestamped pg_dump (custom format), rotated the structured crown jewels
MinIO audio chunks, species images, model files, APKs incremental rclone sync mirror per bucket the field audio is irreplaceable
Mosquitto dynamic-security.json file copy losing it means re-registering every satellite

Redis is intentionally not backed up. The BullMQ queue rebuilds, and unsent chunks re-drain from the satellite outboxes, so there is nothing worth saving.

The backup sidecar (in docker-compose.yml, image built from config/backup/) writes everything to a local directory first (fast restore), then pushes the whole tree off-site with rclone (real disaster recovery).

Configuration

All via .env (see .env.example):

Var Default Notes
BACKUP_ENABLED false opt-in switch. Scheduled backups run only when true; otherwise the sidecar exits and stays Exited (no idle container). Manual runs work either way.
BNG_BACKUP_VOLUMES ./storage/backups local backup target. Point at a different disk than BNG_APP_VOLUMES for real DR.
BACKUP_REMOTE (empty) rclone remote+path for the off-site copy, e.g. offsite:birdnet. Empty = local only.
BNG_RCLONE_CONFIG ./config/backup/rclone.conf rclone config defining the offsite remote (credentials).
BACKUP_SCHEDULE 0 3 * * * cron expression (UTC).
BACKUP_KEEP_DAILY 7 most-recent dumps to keep.
BACKUP_KEEP_WEEKLY 4 additionally keep the newest dump from each of the last N ISO weeks.

The in-stack MinIO is reached over S3 via RCLONE_CONFIG_MINIO_* env wired in the compose service; do not configure it in rclone.conf. That file is only for the off-site offsite remote (templated with examples for S3 / B2 / SFTP).

Off-site setup

  1. Set BACKUP_ENABLED=true in .env (scheduled backups are off by default).
  2. Edit config/backup/rclone.conf (or point BNG_RCLONE_CONFIG elsewhere): add an [offsite] section for your provider.
  3. Set BACKUP_REMOTE=offsite:<bucket-or-path> in .env.
  4. docker compose up -d backup.

Without these, backups are local-only, which is not disaster recovery (a lost disk takes the backups with it).

Operating

The sidecar runs one backup at boot (surfacing config errors immediately) then on BACKUP_SCHEDULE. Run one on demand:

docker compose run --rm backup backup.sh

Inspect the latest run without unpacking anything:

cat "$BNG_BACKUP_VOLUMES/MANIFEST.txt"

Restore

Restoring is destructive: it overwrites the live database and MinIO buckets. Do it deliberately.

1. Stop the writers

The hub must not be running during a Postgres restore (it holds connections and would race migrations):

docker compose stop api dispatcher worker web

2. Restore Postgres + MinIO

# Restores the most recent local dump; pass a path to pick another.
docker compose run --rm backup restore.sh
# or:  docker compose run --rm backup restore.sh /backups/postgres/birdnet-YYYYMMDD-HHMMSS.dump

This pg_restore --cleans the database and rclone syncs each bucket mirror back into MinIO.

3. Restore the Mosquitto credentials (if lost)

With the broker stopped, copy the saved file back into its data volume, then start it:

docker compose stop mosquitto
cp "$BNG_BACKUP_VOLUMES/mosquitto/dynamic-security.json" "$BNG_APP_VOLUMES/mosquitto/data/"
docker compose up -d mosquitto

4. Bring the stack back

docker compose up -d

The hub runs migrations on startup; restoring a current dump leaves nothing to apply, but the step is idempotent and safe.

Restoring from off-site

If the local backup dir is gone too, pull it back first, then restore as above:

rclone sync "$BACKUP_REMOTE" "$BNG_BACKUP_VOLUMES"

Recovering onto a fresh host

  1. Install Docker, clone the repo, restore .env (keep it somewhere safe; it holds JWT_SECRET, MFA_ENCRYPTION_KEY, and DB/MinIO credentials, all of which must match the backup for sessions and encrypted secrets to keep working).
  2. rclone sync "$BACKUP_REMOTE" "$BNG_BACKUP_VOLUMES" to pull the backup local.
  3. docker compose up -d postgres minio mosquitto (infrastructure only).
  4. Run the restore steps above.
  5. docker compose up -d.

.env is not in any backup (it is the one secret-bearing file and is gitignored). Store it independently. Without the original MFA_ENCRYPTION_KEY the restored TOTP secrets and webhook signing secrets cannot be decrypted.

Verifying a backup

A backup nobody has restored is a guess. To check a dump restores cleanly without touching production, restore it into a scratch database and compare counts:

docker exec <postgres> psql -U birdnet -d postgres -c "CREATE DATABASE restore_check;"
PW=$(grep ^POSTGRES_PASSWORD= .env | cut -d= -f2-)
docker compose run --rm -e PGPASSWORD="$PW" backup \
  pg_restore --clean --if-exists --no-owner --no-privileges \
  -h postgres -U birdnet -d restore_check /backups/postgres/<dump>
docker exec <postgres> psql -U birdnet -d restore_check -c "SELECT count(*) FROM detections;"
docker exec <postgres> psql -U birdnet -d postgres -c "DROP DATABASE restore_check;"