Skip to main content
This page documents a governance system on the legacy research surface (services/api/main.py, the ~258-endpoint workbench API that is explicitly not deployed as the product — see API reference overview), not the deployed product application. If you are deploying the actual Causeloop product (services/api/product_app.py, the frontend, and the two worker processes) to a cloud account, use AWS deployment or Azure deployment instead — those pages cover the real release/rollback sequence for product_app.py. This page covers a separate, narrower question: has the causegraph research engine’s own runtime — its repository backends, graph reads, object-storage writes, external scheduler, and lineage/experiment-tracking sinks — actually been switched from local file-backed defaults to managed production services, with evidence to prove it.

​
What production promotion proves

causegraph (packages/causegraph/src/causegraph) can run entirely against local, file-backed state: a disk-backed embedding store, an in-memory graph, local JSON operational storage. That mode is what the “local refactor readiness” gate (local_refactor_ready_production_hardening_remaining, reported as overall_status until every stage below closes) certifies as internally consistent. Production promotion is the separate, stricter gate that certifies the engine has been switched onto real backends and has structured evidence to prove it — not just that the switch was flipped, but that a validation surface actually exercised the managed service and recorded a ready result. The gate is exposed two ways:
  • HTTP, on the research API — POST /runtime/production-promotion/preflight (services/api/main.py).
  • CLI, as python -m causegraph.evidence_gate (packages/causegraph/src/causegraph/evidence_gate.py), which can read the same preflight endpoint or run entirely offline against a --configured-checks-file.
The preflight endpoint is read-only. It does not switch backends, promote artifacts, start a scheduler, or configure an observability sink. It only reports promotion_blockers and required_external_actions — the checklist of what an operator still has to do, and where.

​
Preconditions

installs the same clients exposed by the causegraph[promotion] extra — psycopg[binary], neo4j, boto3, and mlflow (packages/causegraph/pyproject.toml) — required by the configured-check validation drivers below. You also need a running instance of the research API (services/api/main.py, not product_app.py) reachable at the URL you pass as --preflight-url / PREFLIGHT_URL.

​
Capture the current gate

validate_configured:false is a safe local read — it still enumerates every skipped configured check by id, it just doesn’t call out to Postgres/Neo4j/object storage/the scheduler to validate them. Set validate_configured:true only once those backends are actually configured; otherwise every configured check reports not-configured rather than failed. For the same information reshaped as an ordered operator queue:
Read summary.next_stage_id / summary.next_stage as the current action. Each stage in the checklist carries promotion_area, blocked_by (the complete unresolved blocker list — active runtime/storage blockers plus any skipped, missing, or failed configured checks), and ordered validation_steps mapping each operator command to a configured_check_id.

​
Promotion sequence

The five external stages run in a fixed order; overall_status only reaches promotion_ready once every stage’s blockers clear and promotion_blockers is empty.
Stage 4’s scheduler check is not health-only: even when the managed scheduler health endpoint reports healthy and the current deployment package id, /runtime/orchestration/scheduler-validation stays incomplete until a valid run-backed execution receipt exists, and if the health payload reports last_receipt_id it must match that receipt.

​
The CLI-driven workflow

For a single operator or CI run, the recommended path is bootstrap → collect → gate, all through causegraph.evidence_gate:
1

Emit the bootstrap script

This writes the editable preflight request, environment template, validation plan, stage manifest, status/checklist reports, and the full command bundle/handoff package — without collecting configured checks, recording a receipt, or running the gate. RUN_UV_SYNC=1 opts into installing the promotion dependency group; it is otherwise skipped.
2

Emit and run the command bundle

The generated script collects configured-check evidence from live preflight, writes before/after status, checklist, and validation-plan snapshots, records a diagnostic (--gate none) evidence receipt, validates its own output directory, and only then runs the real gate — so a failed run still leaves one coherent, inspectable evidence trail rather than a partial one.
3

Record a durable receipt

Archives the exact filled configured checks, coverage, and gate summary so they survive process reloads and can be reused (use_latest_receipt:true) by later CI steps instead of re-collecting evidence.
For stage-by-stage execution instead of one all-in-one run — useful when different teams own Postgres, the scheduler, and observability — emit the stage evidence index, then run each stage’s generated script from the operator handoff package’s scripts/ directory (e.g. scripts/01-activate-postgres-runtime-repository.sh, with every stage’s path listed in promotion-stage-evidence-index.json) — each script writes its evidence into promotion-stage-evidence/<stage>/ — then merge:
The merge script combines each stage’s configured-checks.json; an invalid merge report (any stage directory that fails validation, or merge issues) is rejected as configured-check input rather than silently used as partial evidence. It always runs the promotion gate and captures its exit code, then writes promotion-final-review.json/.md regardless of whether the gate passed, exiting with the gate’s own status — the same failed-run-still-leaves-a-review guarantee as the command bundle above.

​
Verification gates

Two gates matter, both driven by the same --gate flag on causegraph.evidence_gate (Literal["local", "promotion", "none"]):
  • --gate local (or python -m causegraph.evidence_gate --gate local --pretty) checks only the local refactor invariants — including active_learning_local_gate, which requires a closed_loop_active learning loop, at least one lineage-valid reviewed-training consumer, zero invalid consumers, and zero unconsumed label/evaluation artifacts. Status-only labels do not satisfy it.
  • --gate promotion additionally requires configured_check_coverage.coverage_status == "complete_ready": every required configured check present, ready, carrying a non-empty source_report_id, current validated_at/collected_at evidence, and a summary that satisfies that check’s generated summary_requirements. A manually filled {"status": "ready"} with nothing else is reported as invalid ready evidence, not proof.
  • --gate none still records a receipt for audit purposes but is explicitly non-promotable.
Before trusting either gate, run the external evidence intake review — it independently scans for missing/non-ready checks, placeholder leftovers, stale or missing timestamps on evidence claiming to be ready, and secret-looking values, and both the promotion gate and receipt recording reject critical findings:
overall_status on the preflight/checklist/evidence-bundle responses moves through three observable values: local_refactor_ready_production_hardening_remaining (local gate clean, external stages still open), promotion_blocked (at least one stage still has an open blocker), and promotion_ready (every stage closed, promotion_blockers empty). Only the last one means the engine’s production backends are proven, not merely configured.

​
Rollback

There is no destructive step to roll back from running preflight, the checklist, or any --emit-* command — every one of them is read-only against the research API and writes only local report files. The only state-changing action in this whole flow is a stage operator setting the environment variables in the table above (CAUSELOOP_RUNTIME_BACKEND, CAUSELOOP_GRAPH_BACKEND, CAUSELOOP_ARTIFACT_WRITE_BACKEND, and similar) on the running research-API process, and recording a durable evidence receipt. If a stage’s live validation fails or a managed backend proves unreliable, “rollback” is exactly that: revert the stage’s environment variables to their local values and restart the process, which returns that stage to local_refactor_ready_production_hardening_remaining rather than promotion_ready. This is precisely why the validation surfaces are named *-dry-run and *-switch-rehearsal for the highest-risk stage (Postgres): they let an operator prove a backend switch is safe before flipping the environment variable that makes it live, rather than needing to undo a live switch after the fact. Stale or invalid evidence receipts (receipt_stale, receipt_invalid) are never silently reused — they must be recollected, which is the same failure-safe default applied to any other evidence gap.