/metrics, and the job queue are built. This page is the on-call companion: given a paged alert or a support ticket, what does each signal actually mean in production, and what do you run next.
Readiness vs. liveness
GET /health / GET /health/live (services/api/product_app.py) only proves the API process can serve a request — it has no dependency checks and returns {"status": "ok"} unconditionally. Never route traffic, and never gate a deployment or rollback decision, on liveness alone — a live-but-not-ready API will happily accept traffic it cannot correctly serve (queued jobs that never process, uploads that 503, sign-ins that silently never send their reset link).
GET /health/ready is the real gate. It returns 503 the instant any one of six independent checks is false, and names exactly which one:
Reading GET /metrics
services/api/metrics_api.py exposes a Prometheus-format scrape endpoint, gated behind employee authentication (Depends(get_current_employee)) because it carries cross-tenant operational data — point your scraper at it with a service-account employee session, not an anonymous target.
A single tenant’s per-tenant metrics query failing (a transient connection blip) is caught and skipped rather than blanking the whole scrape (
except Exception: continue in _tenant_metrics) — a gap for one tenant_slug in causeloop_ingest_jobs_total between two scrapes is not itself an incident.
Audit log forensics
Two separate audit logs exist, and an incident investigation usually needs both:control.audit_log(ControlAuditLog,services/api/db/models.py) — control-plane/employee actions only:tenant_created,checkpoint_activated,dataset_ingested,tenant_go_live, and unconditionally, everyemployee_impersonation. Columns:actor_type,actor_id,tenant_id,action,resource_type,resource_id,request_id,detail(JSONB),created_at.- The per-tenant
audit_logtable (TenantAuditLog, one pertenant_<slug>schema) — every CAP/review mutation a tenant user makes, exposed read-only throughGET /remediation/audit-log(resource_type/resource_idfilters,limitclamped to[1, 200]). See Remediation for the full schema and query parameters.
control.audit_log directly:
GET /remediation/audit-log endpoint over a direct schema query where possible — it already carries actor_email, session_id, and ip_address alongside the action, which the control-plane log does not.
control.platform_jobs triage queries
Every asynchronous effect in the product — provisioning, training, materialization, email — is one row in control.platform_jobs. These queries (see Workers and jobs for the full schema) cover the triage questions that come up during an incident:
error_detail is deliberately generic for send_email jobs ("Email delivery failed; provider detail withheld.") — job_failure_detail() (services/pipeline/jobs.py) withholds the real provider exception specifically so a recipient address or token fragment never lands in a job row or a worker log. Do not expect provider-level detail from this column for email failures; check the email provider’s own delivery logs (SES/ACS console) instead.running job whose locked_at is older than CAUSELOOP_JOB_STALE_AFTER_SECONDS (900s / 15 minutes default) is not stuck forever on its own — recover_stale_jobs() runs once at every worker process startup and requeues it (or fails it terminally if attempts >= max_attempts) automatically. If you see a job stuck well past that threshold with no worker process restart in sight, that itself is the incident: the recovery sweep only runs at startup, not on a timer, so a long-lived worker process with a genuinely wedged claim will not self-heal until it restarts.
Restart and recovery order
When integrity is uncertain,docs/CLOUD_OPERATIONS.md’s incident priorities are the practiced order — the same one echoed in Migrations for a deploy-rollback specifically, restated here for a live incident:
1
Stop new writes
Scale the API and both workers to zero if data integrity is in question. This is a blunt but safe first move — no new jobs enqueue, no new writes land, and nothing already durable in PostgreSQL is at risk from stopping traffic.
2
Preserve evidence
Leave PostgreSQL job rows, logs, request IDs, and object versions exactly as they are. Do not truncate
control.platform_jobs or delete object versions while triaging — they are the only record of what was in flight.3
Restore PostgreSQL first
It is the metadata and business authority — tenant records, job state, and dataset/checkpoint metadata all live there. Nothing else can be trusted to reconcile until this is correct. See Backup and recovery for the restore procedure itself.
4
Restore or select the correct object versions next
Only after PostgreSQL is trustworthy, because the metadata rows are what tell you which object version is the correct one to restore or select.
5
Bring services back in order
Provisioner worker, then product worker, then API, then frontend. Workers should be running and heartbeating before the API is trusted to report
ready, since the worker readiness check depends on it.6
Require readiness, then reconcile
Wait for
/health/ready to report clean, then reconcile any queued/running jobs left over from before the incident — the triage queries above are exactly how — before reopening traffic to real users.Redis is explicitly not a business-data or job authority in this ordering. Losing it resets login rate-limit windows; it never loses a dataset, an upload, or a job, because none of those are stored there. Restoring Redis is not on the critical path to reopening traffic.
Alerting
Only AWS currently provisions alarms as infrastructure.deploy/aws/main.tf creates an SNS topic (aws_sns_topic.alerts, KMS-encrypted with alias/aws/sns) with an optional email subscription (var.alert_email), and wires three CloudWatch alarms to it: