Skip to main content
Local stack as a deployment-parity tool is the authoritative walkthrough of scripts/rehearse_product_backup.sh and the three named Docker volumes — read it for the full mechanics. This page is the backup-and-recovery reference proper: a condensed version of that local rehearsal, what the AWS and Azure Terraform/Bicep stacks actually configure for backup (only what deploy/ provisions, nothing assumed), the retention model shared across all three environments, the quarterly cloud restore drills, and what this repository does and does not commit to for RPO/RTO.

​
Local restore rehearsal

With the product Compose stack running (infra/docker-compose.product.yml):
1

Dump

pg_dump -U causeloop -d causeloop -Fc runs inside the running postgres container, writing a custom-format dump to a scratch path.
2

Restore into an isolated scratch database

A throwaway database named causeloop_restore_rehearsal_<pid> is created and the dump is restored into it with pg_restore — the live causeloop database is never touched, and the script refuses to run at all if its own generated scratch-database name doesn’t match the expected causeloop_restore_rehearsal_[0-9]* pattern.
3

Verify the control schema came back

SELECT to_regclass('control.tenants') IS NOT NULL against the restored copy must return t, proving the control-plane schema — tenants, jobs, audit log — is intact in the restore, not just that pg_restore exited zero.
4

Confirm object-store versioning

A one-off mc container checks mc stat local/causeloop-uploads and mc version info local/causeloop-uploads against MinIO, confirming the uploads bucket is reachable and versioning is enabled — the same property the S3/Blob backends are configured with in the cloud.
5

Clean up unconditionally

A trap ... EXIT HUP INT TERM drops the scratch database and deletes the dump file even if an earlier step failed, so a rehearsal run never leaves scratch state behind.
A green run is the local evidence that dump/restore/verify actually works, cheaply enough to run before every release rather than only once a quarter. It intentionally proves the same three things the cloud quarterly drills below prove — restore succeeds, control-plane data is intact, object versioning is enabled — just against a local Postgres/MinIO pair instead of RDS/Flexible Server and S3/Blob.

​
What’s backed up, and what isn’t

The rehearsal (and the cloud posture below) treats PostgreSQL as the sole backup target that matters for correctness, and object storage as versioned rather than separately snapshotted:
  • PostgreSQL holds every tenant record, platform job, audit log row, dataset/checkpoint/insight-collection row — the entire control and tenant-schema state. This is what gets dumped, restored, and verified.
  • Object storage (MinIO locally; S3 on AWS; Blob on Azure) holds raw uploaded files. It is never restored from a separate backup target in this tooling — instead, versioning is enabled and never disabled, so an overwritten or deleted object’s prior bytes remain retrievable by version id for as long as retention allows (see below). The rehearsal script checks that this property holds (mc version info) rather than restoring a snapshot.
  • Redis is backed up nowhere in any of the three environments, deliberately — see Monitoring and incidents for why: it holds only login rate-limit windows, never durable job or business state.

​
Named volume lifecycle (local)

Local product state lives in three named Docker volumes, not bind mounts — product_postgres, product_redis, product_minio (infra/docker-compose.product.yml). docker compose down removes containers but preserves these volumes; only docker compose down -v deletes them, with no confirmation prompt. Treat -v with the same caution as dropping a cloud RDS instance or emptying a versioned S3 bucket — see Local stack as a deployment-parity tool for the full detail.

​
Cloud backup configuration

Only what deploy/aws and deploy/azure actually provision — verified against main.tf / main.bicep, not assumed from the general pattern:
  • RDS PostgreSQL: storage_encrypted = true with a customer-managed KMS key, multi_az enabled in the production environment, backup_retention_period (default 14 days via var.backup_retention_days) giving automated backups with point-in-time recovery, deletion_protection (default true), and skip_final_snapshot = false with a named final snapshot on destroy.
  • S3: bucket versioning enabled (aws_s3_bucket_versioning), server-side KMS encryption, a lifecycle rule (aws_s3_bucket_lifecycle_configuration) expiring current objects after var.object_retention_days (default 365) and noncurrent versions after 30 days, plus abort_incomplete_multipart_upload cleanup.
  • ElastiCache Redis: at_rest_encryption_enabled and transit_encryption_enabled, with snapshot_retention_limit also tied to var.backup_retention_days — a convenience, not a business-data guarantee, since Redis holds no durable application state.

​
Why object versions, not object overwrites, are the recovery unit

docs/CLOUD_DEPLOYMENT.md is explicit that object creation in both clouds is conditional at the provider API, not merely collision-resistant by naming convention: S3 completes multipart writes with If-None-Match: *, and Azure commits uniquely named blocks with an if-missing condition. A duplicate key cannot silently replace the bytes whose SHA-256 is recorded in the PostgreSQL metadata row for that upload. This is exactly why the recovery drills above verify a restored object’s SHA-256 against the database record rather than just checking the object exists — versioning plus this conditional-write guarantee means “restore the correct version” and “prove it’s byte-identical to what the metadata claims” are the same check, not two separate leaps of faith.

​
Who can actually run a restore

The same credential boundary docs/CLOUD_DEPLOYMENT.md establishes for migrations applies to a restore drill’s validation step: the API and product worker never receive the owner/migration DSN. Only the one-off migration task/job and the provisioning worker can perform schema DDL. The quarterly drills above restore at the provider level — RDS/Flexible Server point-in-time restore into a new, isolated instance, not a CREATE DATABASE/pg_restore invocation against a live server — as a matter of isolation discipline, keeping every drill off the production instance entirely. But the drill’s own “run migrations in validation mode against the restored instance” step still needs the owner-level database credential, which the running application’s APP_DATABASE_URL deliberately never holds.

​
Retention model

All three environments — local, AWS, Azure — default raw object retention to 365 days, and this is one number kept consistent on purpose rather than three independently chosen defaults:
  • Local: minio-init’s mc ilm rule add --expire-days "$CAUSELOOP_OBJECT_RETENTION_DAYS" --noncurrent-expire-days 30 (CAUSELOOP_OBJECT_RETENTION_DAYS defaults to 365 in infra/docker-compose.product.yml).
  • AWS: var.object_retention_days (Terraform variable, default 365), with a 30-day noncurrent-version expiration matching the local rule exactly.
  • Azure: objectRetentionDays (Bicep parameter, default 365).
PostgreSQL backup retention is a separate, shorter number (14 days by default on both clouds) — that is the window for point-in-time recovery of the database, not for how long a raw uploaded object stays retrievable.

​
Quarterly cloud recovery drills

docs/CLOUD_OPERATIONS.md specifies the same five-step shape for both clouds — restore into an isolated target, prove the restore is structurally and referentially sound, then tear down only what the drill itself created:
1

Restore RDS to a new isolated instance

At a selected point-in-time timestamp, separate from the live instance.
2

Run migrations in validation mode

Against the restored instance, never the live one.
3

Verify tenant/control counts and one insight snapshot

Confirms referential integrity, not just that the restore process exited zero.
4

Retrieve a sampled upload by version ID

Verify its stored SHA-256 matches what PostgreSQL’s metadata row records for that object.
5

Destroy only the explicitly named rehearsal resources

After sign-off — never touch anything outside the drill’s own isolated instance.
These drills are the practiced, cloud-real form of the exact same discipline scripts/rehearse_product_backup.sh proves cheaply and locally: dump/restore into isolation, verify referential integrity, verify object retrievability by version, then clean up only what was created for the drill.

​
RPO/RTO honesty

This repository does not define or commit to formal RPO/RTO numbers anywhere in docs/ or deploy/, and this page will not invent ones. What is actually configured and verifiable:
  • Implied RPO ceiling: RDS/Flexible Server automated backups with point-in-time recovery bound data loss to whatever granularity the managed provider’s PITR log-shipping interval provides, within the backup_retention_period/backupRetentionDays window (14 days by default on both clouds) — but no drill in this repository measures or asserts an actual achieved RPO number.
  • No measured RTO: the quarterly drills above prove a restore works, not how long it takes end-to-end under a real incident’s load and coordination overhead. There is no documented target time-to-recovery, and no automated failover test exercises the full restart and recovery order against a live production-shaped incident.
  • What is real: PostgreSQL Multi-AZ (AWS, production only) and Azure’s geo-redundant backup reduce the likelihood of needing the restore path at all for a single-AZ/region failure, but they are not a substitute for the quarterly restore-drill discipline above, which is what actually proves the backup is usable rather than merely present.
Treat any RPO/RTO figure quoted to a client or in an internal SLA as a commitment layered on top of this infrastructure, not something this repository’s configuration already guarantees on its own.