> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup and recovery

> Backup posture end to end: the local restore rehearsal script, named-volume lifecycle, the AWS/Azure backup configuration deploy/ actually provisions, retention defaults, and recovery procedures — with an honest RPO/RTO statement.

[Local stack as a deployment-parity tool](/deployment/local-parity-stack) is the authoritative walkthrough of `scripts/rehearse_product_backup.sh` and the three named Docker volumes — read it for the full mechanics. This page is the backup-and-recovery reference proper: a condensed version of that local rehearsal, what the AWS and Azure Terraform/Bicep stacks actually configure for backup (only what `deploy/` provisions, nothing assumed), the retention model shared across all three environments, the quarterly cloud restore drills, and what this repository does and does not commit to for RPO/RTO.

## Local restore rehearsal

With the product Compose stack running (`infra/docker-compose.product.yml`):

```bash theme={null}
scripts/rehearse_product_backup.sh
```

<Steps>
  <Step title="Dump">
    `pg_dump -U causeloop -d causeloop -Fc` runs inside the running `postgres` container, writing a custom-format dump to a scratch path.
  </Step>

  <Step title="Restore into an isolated scratch database">
    A throwaway database named `causeloop_restore_rehearsal_<pid>` is created and the dump is restored into it with `pg_restore` — the live `causeloop` database is never touched, and the script refuses to run at all if its own generated scratch-database name doesn't match the expected `causeloop_restore_rehearsal_[0-9]*` pattern.
  </Step>

  <Step title="Verify the control schema came back">
    `SELECT to_regclass('control.tenants') IS NOT NULL` against the restored copy must return `t`, proving the control-plane schema — tenants, jobs, audit log — is intact in the restore, not just that `pg_restore` exited zero.
  </Step>

  <Step title="Confirm object-store versioning">
    A one-off `mc` container checks `mc stat local/causeloop-uploads` and `mc version info local/causeloop-uploads` against MinIO, confirming the uploads bucket is reachable and versioning is enabled — the same property the S3/Blob backends are configured with in the cloud.
  </Step>

  <Step title="Clean up unconditionally">
    A `trap ... EXIT HUP INT TERM` drops the scratch database and deletes the dump file even if an earlier step failed, so a rehearsal run never leaves scratch state behind.
  </Step>
</Steps>

A green run is the local evidence that dump/restore/verify actually works, cheaply enough to run before every release rather than only once a quarter. It intentionally proves the same three things the cloud quarterly drills below prove — restore succeeds, control-plane data is intact, object versioning is enabled — just against a local Postgres/MinIO pair instead of RDS/Flexible Server and S3/Blob.

## What's backed up, and what isn't

The rehearsal (and the cloud posture below) treats **PostgreSQL as the sole backup target that matters for correctness**, and object storage as versioned rather than separately snapshotted:

* **PostgreSQL** holds every tenant record, platform job, audit log row, dataset/checkpoint/insight-collection row — the entire control and tenant-schema state. This is what gets dumped, restored, and verified.
* **Object storage** (MinIO locally; S3 on AWS; Blob on Azure) holds raw uploaded files. It is never restored from a separate backup target in this tooling — instead, **versioning is enabled and never disabled**, so an overwritten or deleted object's prior bytes remain retrievable by version id for as long as retention allows (see below). The rehearsal script checks that this property holds (`mc version info`) rather than restoring a snapshot.
* **Redis** is backed up nowhere in any of the three environments, deliberately — see [Monitoring and incidents](/deployment/monitoring-and-incidents#restart-and-recovery-order) for why: it holds only login rate-limit windows, never durable job or business state.

## Named volume lifecycle (local)

Local product state lives in three named Docker volumes, not bind mounts — `product_postgres`, `product_redis`, `product_minio` (`infra/docker-compose.product.yml`). `docker compose down` removes containers but **preserves** these volumes; only `docker compose down -v` deletes them, with no confirmation prompt. Treat `-v` with the same caution as dropping a cloud RDS instance or emptying a versioned S3 bucket — see [Local stack as a deployment-parity tool](/deployment/local-parity-stack#volume-lifecycle) for the full detail.

## Cloud backup configuration

Only what `deploy/aws` and `deploy/azure` actually provision — verified against `main.tf` / `main.bicep`, not assumed from the general pattern:

<Tabs>
  <Tab title="AWS">
    * **RDS PostgreSQL**: `storage_encrypted = true` with a customer-managed KMS key, `multi_az` enabled in the `production` environment, `backup_retention_period` (default 14 days via `var.backup_retention_days`) giving automated backups with point-in-time recovery, `deletion_protection` (default `true`), and `skip_final_snapshot = false` with a named final snapshot on destroy.
    * **S3**: bucket versioning enabled (`aws_s3_bucket_versioning`), server-side KMS encryption, a lifecycle rule (`aws_s3_bucket_lifecycle_configuration`) expiring current objects after `var.object_retention_days` (default 365) and noncurrent versions after 30 days, plus `abort_incomplete_multipart_upload` cleanup.
    * **ElastiCache Redis**: `at_rest_encryption_enabled` and `transit_encryption_enabled`, with `snapshot_retention_limit` also tied to `var.backup_retention_days` — a convenience, not a business-data guarantee, since Redis holds no durable application state.
  </Tab>

  <Tab title="Azure">
    * **PostgreSQL Flexible Server**: `backupRetentionDays` (default 14 via `backupRetentionDays` parameter) with `geoRedundantBackup` enabled specifically in the `production` environment.
    * **Blob Storage**: `isVersioningEnabled: true`, plus both `containerDeleteRetentionPolicy` and `deleteRetentionPolicy` (soft delete) configured, and a lifecycle-management rule (`tenant-object-retention`) tied to `objectRetentionDays` (default 365).
    * **Azure Managed Redis**: used instead of the retiring Azure Cache for Redis, with the classic client protocol so the application's standard `redis-py` client works unchanged — again, no durable business data lives here.
  </Tab>
</Tabs>

## Why object versions, not object overwrites, are the recovery unit

`docs/CLOUD_DEPLOYMENT.md` is explicit that object creation in both clouds is **conditional at the provider API, not merely collision-resistant by naming convention**: S3 completes multipart writes with `If-None-Match: *`, and Azure commits uniquely named blocks with an if-missing condition. A duplicate key cannot silently replace the bytes whose SHA-256 is recorded in the PostgreSQL metadata row for that upload. This is exactly why the recovery drills above verify a restored object's SHA-256 against the database record rather than just checking the object exists — versioning plus this conditional-write guarantee means "restore the correct version" and "prove it's byte-identical to what the metadata claims" are the same check, not two separate leaps of faith.

## Who can actually run a restore

The same credential boundary `docs/CLOUD_DEPLOYMENT.md` establishes for migrations applies to a restore drill's validation step: **the API and product worker never receive the owner/migration DSN.** Only the one-off migration task/job and the provisioning worker can perform schema DDL. The quarterly drills above restore at the **provider level** — RDS/Flexible Server point-in-time restore into a new, isolated instance, not a `CREATE DATABASE`/`pg_restore` invocation against a live server — as a matter of isolation discipline, keeping every drill off the production instance entirely. But the drill's own "run migrations in validation mode against the restored instance" step still needs the owner-level database credential, which the running application's `APP_DATABASE_URL` deliberately never holds.

## Retention model

All three environments — local, AWS, Azure — default raw object retention to **365 days**, and this is one number kept consistent on purpose rather than three independently chosen defaults:

* Local: `minio-init`'s `mc ilm rule add --expire-days "$CAUSELOOP_OBJECT_RETENTION_DAYS" --noncurrent-expire-days 30` (`CAUSELOOP_OBJECT_RETENTION_DAYS` defaults to `365` in `infra/docker-compose.product.yml`).
* AWS: `var.object_retention_days` (Terraform variable, default `365`), with a 30-day noncurrent-version expiration matching the local rule exactly.
* Azure: `objectRetentionDays` (Bicep parameter, default `365`).

PostgreSQL backup retention is a separate, shorter number (14 days by default on both clouds) — that is the window for point-in-time recovery of the *database*, not for how long a raw uploaded object stays retrievable.

## Quarterly cloud recovery drills

`docs/CLOUD_OPERATIONS.md` specifies the same five-step shape for both clouds — restore into an isolated target, prove the restore is structurally and referentially sound, then tear down only what the drill itself created:

<Tabs>
  <Tab title="AWS">
    <Steps>
      <Step title="Restore RDS to a new isolated instance">At a selected point-in-time timestamp, separate from the live instance.</Step>
      <Step title="Run migrations in validation mode">Against the restored instance, never the live one.</Step>
      <Step title="Verify tenant/control counts and one insight snapshot">Confirms referential integrity, not just that the restore process exited zero.</Step>
      <Step title="Retrieve a sampled upload by version ID">Verify its stored SHA-256 matches what PostgreSQL's metadata row records for that object.</Step>
      <Step title="Destroy only the explicitly named rehearsal resources">After sign-off — never touch anything outside the drill's own isolated instance.</Step>
    </Steps>
  </Tab>

  <Tab title="Azure">
    <Steps>
      <Step title="Restore the server to a new explicitly named rehearsal server">Separate from the production Flexible Server.</Step>
      <Step title="Attach a temporary migration/smoke job inside the VNet">Scoped to the rehearsal server only.</Step>
      <Step title="Verify tenant/control counts and one insight snapshot">Same referential-integrity check as AWS.</Step>
      <Step title="Retrieve a sampled prior blob version and compare SHA-256">Confirms Blob versioning is not just enabled but actually restorable.</Step>
      <Step title="Remove only the rehearsal server/job">After sign-off.</Step>
    </Steps>
  </Tab>
</Tabs>

These drills are the practiced, cloud-real form of the exact same discipline `scripts/rehearse_product_backup.sh` proves cheaply and locally: dump/restore into isolation, verify referential integrity, verify object retrievability by version, then clean up only what was created for the drill.

## RPO/RTO honesty

This repository does not define or commit to formal RPO/RTO numbers anywhere in `docs/` or `deploy/`, and this page will not invent ones. What is actually configured and verifiable:

* **Implied RPO ceiling**: RDS/Flexible Server automated backups with point-in-time recovery bound data loss to whatever granularity the managed provider's PITR log-shipping interval provides, within the `backup_retention_period`/`backupRetentionDays` window (14 days by default on both clouds) — but no drill in this repository measures or asserts an actual achieved RPO number.
* **No measured RTO**: the quarterly drills above prove a restore *works*, not how long it takes end-to-end under a real incident's load and coordination overhead. There is no documented target time-to-recovery, and no automated failover test exercises the full [restart and recovery order](/deployment/monitoring-and-incidents#restart-and-recovery-order) against a live production-shaped incident.
* **What is real**: PostgreSQL Multi-AZ (AWS, `production` only) and Azure's geo-redundant backup reduce the likelihood of needing the restore path at all for a single-AZ/region failure, but they are not a substitute for the quarterly restore-drill discipline above, which is what actually proves the backup is usable rather than merely present.

Treat any RPO/RTO figure quoted to a client or in an internal SLA as a commitment layered on top of this infrastructure, not something this repository's configuration already guarantees on its own.

## Related

* [Local stack as a deployment-parity tool](/deployment/local-parity-stack)
* [Monitoring and incidents](/deployment/monitoring-and-incidents)
* [AWS deployment](/deployment/aws)
* [Azure deployment](/deployment/azure)