> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Production promotion

> The causegraph evidence-gate: proving the research engine's runtime backends are wired to managed Postgres, Neo4j, object storage, a scheduler, and observability sinks before promotion is trusted.

<Warning>
  This page documents a governance system on the **legacy research surface** (`services/api/main.py`, the \~258-endpoint workbench API that is explicitly not deployed as the product — see [API reference overview](/api-reference/overview)), not the deployed product application. If you are deploying the actual Causeloop product (`services/api/product_app.py`, the frontend, and the two worker processes) to a cloud account, use [AWS deployment](/deployment/aws) or [Azure deployment](/deployment/azure) instead — those pages cover the real release/rollback sequence for `product_app.py`. This page covers a separate, narrower question: has the `causegraph` research engine's own runtime — its repository backends, graph reads, object-storage writes, external scheduler, and lineage/experiment-tracking sinks — actually been switched from local file-backed defaults to managed production services, with evidence to prove it.
</Warning>

## What production promotion proves

`causegraph` (`packages/causegraph/src/causegraph`) can run entirely against local, file-backed state: a disk-backed embedding store, an in-memory graph, local JSON operational storage. That mode is what the "local refactor readiness" gate (`local_refactor_ready_production_hardening_remaining`, reported as `overall_status` until every stage below closes) certifies as internally consistent. **Production promotion** is the separate, stricter gate that certifies the engine has been switched onto real backends and has structured evidence to prove it — not just that the switch was flipped, but that a validation surface actually exercised the managed service and recorded a `ready` result.

The gate is exposed two ways:

* **HTTP**, on the research API — `POST /runtime/production-promotion/preflight` (`services/api/main.py`).
* **CLI**, as `python -m causegraph.evidence_gate` (`packages/causegraph/src/causegraph/evidence_gate.py`), which can read the same preflight endpoint or run entirely offline against a `--configured-checks-file`.

<Note>
  The preflight endpoint is **read-only**. It does not switch backends, promote artifacts, start a scheduler, or configure an observability sink. It only reports `promotion_blockers` and `required_external_actions` — the checklist of what an operator still has to do, and where.
</Note>

## Preconditions

```bash theme={null}
uv sync --group promotion
```

installs the same clients exposed by the `causegraph[promotion]` extra — `psycopg[binary]`, `neo4j`, `boto3`, and `mlflow` (`packages/causegraph/pyproject.toml`) — required by the configured-check validation drivers below. You also need a running instance of the research API (`services/api/main.py`, not `product_app.py`) reachable at the URL you pass as `--preflight-url` / `PREFLIGHT_URL`.

## Capture the current gate

```bash theme={null}
curl -s -X POST http://127.0.0.1:8000/runtime/production-promotion/preflight \
  -H 'content-type: application/json' \
  -d '{"limit":500,"validate_configured":false}'
```

`validate_configured:false` is a safe local read — it still enumerates every skipped configured check by id, it just doesn't call out to Postgres/Neo4j/object storage/the scheduler to validate them. Set `validate_configured:true` only once those backends are actually configured; otherwise every configured check reports not-configured rather than failed.

For the same information reshaped as an ordered operator queue:

```bash theme={null}
curl -s http://127.0.0.1:8000/runtime/production-promotion/checklist?limit=500
```

Read `summary.next_stage_id` / `summary.next_stage` as the current action. Each stage in the checklist carries `promotion_area`, `blocked_by` (the complete unresolved blocker list — active runtime/storage blockers plus any skipped, missing, or failed configured checks), and ordered `validation_steps` mapping each operator command to a `configured_check_id`.

## Promotion sequence

The five external stages run in a fixed order; `overall_status` only reaches `promotion_ready` once every stage's blockers clear and `promotion_blockers` is empty.

```mermaid theme={null}
flowchart LR
    A[1. Postgres runtime repositories] --> B[2. Neo4j graph reads]
    B --> C[3. Object storage]
    C --> D[4. External scheduler execution]
    D --> E[5. Observability sinks]
    E --> F{Final gate}
    F -->|promotion_blockers empty| Ready[promotion_ready]
    F -->|blockers remain| Blocked[promotion_blocked]
```

| Stage                            | Required configuration                                                                                                                                                   | Key validation surfaces                                                                                                                                                             | Done when                                                                                                                                                                |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1. Postgres runtime repositories | `CAUSELOOP_POSTGRES_DSN`; `CAUSELOOP_RUNTIME_BACKEND=postgres` (or `CAUSELOOP_REPOSITORY_BACKEND=postgres`)                                                              | `POST /db/postgres/read-validation`, `.../bulk-ingest-read-validation`, `.../backend-activation-dry-run`, `.../live-backend-switch-rehearsal`, `.../live-repository-adapter-status` | The Postgres repository adapter reports active and all configured Postgres checks are `ready`.                                                                           |
| 2. Neo4j graph reads             | `CAUSELOOP_NEO4J_URI`; `CAUSELOOP_GRAPH_BACKEND=neo4j` (or `CAUSELOOP_NEO4J_GRAPH_BACKEND=neo4j`)                                                                        | `GET /lineage/neo4j`, `POST /db/neo4j/read-validation`                                                                                                                              | The Neo4j graph repository reports active and configured read validation is `ready`.                                                                                     |
| 3. Object storage                | `CAUSELOOP_OBJECT_BUCKET`; object-storage credentials; `CAUSELOOP_ARTIFACT_WRITE_BACKEND=object_storage` (or `CAUSELOOP_OBJECT_STORAGE_WRITE_MODE=object_storage_first`) | `POST /artifacts/object-storage/activate-primary`, `.../read-validation`, `.../lifecycle-validation`; `GET /artifacts/object-storage/primary-readiness`                             | Current concrete artifacts are object-storage primary and future writes are object-storage-first.                                                                        |
| 4. External scheduler execution  | A managed Dagster/Prefect/Temporal (or equivalent) deployment; `CAUSELOOP_EXTERNAL_SCHEDULER_HEALTH_URL`                                                                 | `GET /runtime/orchestration/deployment`, `POST /runtime/orchestration/execution-receipts`, `POST /runtime/orchestration/scheduler-validation`                                       | A valid run-backed execution receipt exists and scheduler health reports the current deployment package.                                                                 |
| 5. Observability sinks           | `CAUSELOOP_OPENLINEAGE_URL`; `CAUSELOOP_MLFLOW_TRACKING_URI`; managed Postgres/Neo4j/object-storage credentials                                                          | `POST /lineage/openlineage/emit`, `POST /lineage/mlflow/log`                                                                                                                        | OpenLineage emitted-event count matches source export with zero failures, and MLflow logged-run count matches source export with the required logging methods available. |

<Warning>
  Stage 4's scheduler check is not health-only: even when the managed scheduler health endpoint reports `healthy` and the current deployment package id, `/runtime/orchestration/scheduler-validation` stays incomplete until a valid run-backed execution receipt exists, and if the health payload reports `last_receipt_id` it must match that receipt.
</Warning>

## The CLI-driven workflow

For a single operator or CI run, the recommended path is bootstrap → collect → gate, all through `causegraph.evidence_gate`:

<Steps>
  <Step title="Emit the bootstrap script">
    ```bash theme={null}
    python -m causegraph.evidence_gate \
      --emit-promotion-bootstrap-script \
      --preflight-url http://127.0.0.1:8000/runtime/production-promotion/preflight \
      > promotion-bootstrap.sh

    RUN_UV_SYNC=1 bash promotion-bootstrap.sh
    ```

    This writes the editable preflight request, environment template, validation plan, stage manifest, status/checklist reports, and the full command bundle/handoff package — without collecting configured checks, recording a receipt, or running the gate. `RUN_UV_SYNC=1` opts into installing the promotion dependency group; it is otherwise skipped.
  </Step>

  <Step title="Emit and run the command bundle">
    ```bash theme={null}
    python -m causegraph.evidence_gate \
      --emit-promotion-command-script \
      --preflight-url http://127.0.0.1:8000/runtime/production-promotion/preflight \
      > promotion-command-bundle.sh
    chmod +x promotion-command-bundle.sh
    ./promotion-command-bundle.sh
    ```

    The generated script collects configured-check evidence from live preflight, writes before/after status, checklist, and validation-plan snapshots, records a diagnostic (`--gate none`) evidence receipt, validates its own output directory, and only then runs the real gate — so a failed run still leaves one coherent, inspectable evidence trail rather than a partial one.
  </Step>

  <Step title="Record a durable receipt">
    ```bash theme={null}
    curl -s -X POST http://127.0.0.1:8000/runtime/production-promotion/evidence-receipts \
      -H 'content-type: application/json' \
      -d '{"limit":50,"validate_configured":true,"gate":"promotion","source":"ci"}'
    ```

    Archives the exact filled configured checks, coverage, and gate summary so they survive process reloads and can be reused (`use_latest_receipt:true`) by later CI steps instead of re-collecting evidence.
  </Step>
</Steps>

For stage-by-stage execution instead of one all-in-one run — useful when different teams own Postgres, the scheduler, and observability — emit the stage evidence index, then run each stage's generated script from the operator handoff package's `scripts/` directory (e.g. `scripts/01-activate-postgres-runtime-repository.sh`, with every stage's path listed in `promotion-stage-evidence-index.json`) — each script writes its evidence into `promotion-stage-evidence/<stage>/` — then merge:

```bash theme={null}
python -m causegraph.evidence_gate \
  --emit-promotion-stage-evidence-index \
  --preflight-url http://127.0.0.1:8000/runtime/production-promotion/preflight \
  --pretty > promotion-stage-evidence-index.json

bash promotion-stage-output-merge.sh
```

The merge script combines each stage's `configured-checks.json`; an invalid merge report (any stage directory that fails validation, or merge issues) is rejected as configured-check input rather than silently used as partial evidence. It always runs the promotion gate and captures its exit code, then writes `promotion-final-review.json`/`.md` regardless of whether the gate passed, exiting with the gate's own status — the same failed-run-still-leaves-a-review guarantee as the command bundle above.

## Verification gates

Two gates matter, both driven by the same `--gate` flag on `causegraph.evidence_gate` (`Literal["local", "promotion", "none"]`):

* **`--gate local`** (or `python -m causegraph.evidence_gate --gate local --pretty`) checks only the local refactor invariants — including `active_learning_local_gate`, which requires a `closed_loop_active` learning loop, at least one lineage-valid reviewed-training consumer, zero invalid consumers, and zero unconsumed label/evaluation artifacts. Status-only labels do not satisfy it.
* **`--gate promotion`** additionally requires `configured_check_coverage.coverage_status == "complete_ready"`: every required configured check present, `ready`, carrying a non-empty `source_report_id`, current `validated_at`/`collected_at` evidence, and a summary that satisfies that check's generated `summary_requirements`. A manually filled `{"status": "ready"}` with nothing else is reported as **invalid ready evidence**, not proof.
* **`--gate none`** still records a receipt for audit purposes but is explicitly non-promotable.

Before trusting either gate, run the external evidence intake review — it independently scans for missing/non-ready checks, placeholder leftovers, stale or missing timestamps on evidence claiming to be `ready`, and secret-looking values, and both the promotion gate and receipt recording reject critical findings:

```bash theme={null}
python -m causegraph.evidence_gate \
  --emit-promotion-external-evidence-intake-review \
  --configured-checks-dir promotion-external-evidence \
  --pretty > promotion-external-evidence-intake-review.json
```

`overall_status` on the preflight/checklist/evidence-bundle responses moves through three observable values: `local_refactor_ready_production_hardening_remaining` (local gate clean, external stages still open), `promotion_blocked` (at least one stage still has an open blocker), and `promotion_ready` (every stage closed, `promotion_blockers` empty). Only the last one means the engine's production backends are proven, not merely configured.

## Rollback

There is no destructive step to roll back from running preflight, the checklist, or any `--emit-*` command — every one of them is read-only against the research API and writes only local report files. The only state-changing action in this whole flow is a stage operator setting the environment variables in the table above (`CAUSELOOP_RUNTIME_BACKEND`, `CAUSELOOP_GRAPH_BACKEND`, `CAUSELOOP_ARTIFACT_WRITE_BACKEND`, and similar) on the running research-API process, and recording a durable evidence receipt.

If a stage's live validation fails or a managed backend proves unreliable, "rollback" is exactly that: revert the stage's environment variables to their local values and restart the process, which returns that stage to `local_refactor_ready_production_hardening_remaining` rather than `promotion_ready`. This is precisely why the validation surfaces are named `*-dry-run` and `*-switch-rehearsal` for the highest-risk stage (Postgres): they let an operator prove a backend switch is safe **before** flipping the environment variable that makes it live, rather than needing to undo a live switch after the fact. Stale or invalid evidence receipts (`receipt_stale`, `receipt_invalid`) are never silently reused — they must be recollected, which is the same failure-safe default applied to any other evidence gap.

## Related

* [AWS deployment](/deployment/aws)
* [Azure deployment](/deployment/azure)
* [API reference overview](/api-reference/overview)
* [Migrations](/deployment/migrations)