Skip to main content
Observability and Workers and jobs explain how request logging, /metrics, and the job queue are built. This page is the on-call companion: given a paged alert or a support ticket, what does each signal actually mean in production, and what do you run next.

Readiness vs. liveness

GET /health / GET /health/live (services/api/product_app.py) only proves the API process can serve a request — it has no dependency checks and returns {"status": "ok"} unconditionally. Never route traffic, and never gate a deployment or rollback decision, on liveness alone — a live-but-not-ready API will happily accept traffic it cannot correctly serve (queued jobs that never process, uploads that 503, sign-ins that silently never send their reset link). GET /health/ready is the real gate. It returns 503 the instant any one of six independent checks is false, and names exactly which one:
A 503 with one check false is strictly more useful than a bare connection-refused error from a load balancer health check misconfigured against /health/live instead of /health/ready — always point your platform’s health-check target at /health/ready.

Reading GET /metrics

services/api/metrics_api.py exposes a Prometheus-format scrape endpoint, gated behind employee authentication (Depends(get_current_employee)) because it carries cross-tenant operational data — point your scraper at it with a service-account employee session, not an anonymous target. A single tenant’s per-tenant metrics query failing (a transient connection blip) is caught and skipped rather than blanking the whole scrape (except Exception: continue in _tenant_metrics) — a gap for one tenant_slug in causeloop_ingest_jobs_total between two scrapes is not itself an incident.

Audit log forensics

Two separate audit logs exist, and an incident investigation usually needs both:
  • control.audit_log (ControlAuditLog, services/api/db/models.py) — control-plane/employee actions only: tenant_created, checkpoint_activated, dataset_ingested, tenant_go_live, and unconditionally, every employee_impersonation. Columns: actor_type, actor_id, tenant_id, action, resource_type, resource_id, request_id, detail (JSONB), created_at.
  • The per-tenant audit_log table (TenantAuditLog, one per tenant_<slug> schema) — every CAP/review mutation a tenant user makes, exposed read-only through GET /remediation/audit-log (resource_type/resource_id filters, limit clamped to [1, 200]). See Remediation for the full schema and query parameters.
control.audit_log.request_id exists in the schema (added for correlating an audit row back to the structured HTTP log’s request_id from Observability) but no current code path populates it — every ControlAuditLog insert in onboarding_api.py and auth/employee_auth.py / auth/tenant_auth.py’s employee-facing actions omits it. Do not expect to join an audit row to a specific HTTP request by request_id today; correlate by actor_id + tenant_id + a tight created_at window instead.
For a cross-tenant incident (e.g. “who impersonated tenant X, and when”), query control.audit_log directly:
For a single tenant’s own activity trail, prefer the GET /remediation/audit-log endpoint over a direct schema query where possible — it already carries actor_email, session_id, and ip_address alongside the action, which the control-plane log does not.

control.platform_jobs triage queries

Every asynchronous effect in the product — provisioning, training, materialization, email — is one row in control.platform_jobs. These queries (see Workers and jobs for the full schema) cover the triage questions that come up during an incident:
error_detail is deliberately generic for send_email jobs ("Email delivery failed; provider detail withheld.") — job_failure_detail() (services/pipeline/jobs.py) withholds the real provider exception specifically so a recipient address or token fragment never lands in a job row or a worker log. Do not expect provider-level detail from this column for email failures; check the email provider’s own delivery logs (SES/ACS console) instead.
A running job whose locked_at is older than CAUSELOOP_JOB_STALE_AFTER_SECONDS (900s / 15 minutes default) is not stuck forever on its own — recover_stale_jobs() runs once at every worker process startup and requeues it (or fails it terminally if attempts >= max_attempts) automatically. If you see a job stuck well past that threshold with no worker process restart in sight, that itself is the incident: the recovery sweep only runs at startup, not on a timer, so a long-lived worker process with a genuinely wedged claim will not self-heal until it restarts.

Restart and recovery order

When integrity is uncertain, docs/CLOUD_OPERATIONS.md’s incident priorities are the practiced order — the same one echoed in Migrations for a deploy-rollback specifically, restated here for a live incident:
1

Stop new writes

Scale the API and both workers to zero if data integrity is in question. This is a blunt but safe first move — no new jobs enqueue, no new writes land, and nothing already durable in PostgreSQL is at risk from stopping traffic.
2

Preserve evidence

Leave PostgreSQL job rows, logs, request IDs, and object versions exactly as they are. Do not truncate control.platform_jobs or delete object versions while triaging — they are the only record of what was in flight.
3

Restore PostgreSQL first

It is the metadata and business authority — tenant records, job state, and dataset/checkpoint metadata all live there. Nothing else can be trusted to reconcile until this is correct. See Backup and recovery for the restore procedure itself.
4

Restore or select the correct object versions next

Only after PostgreSQL is trustworthy, because the metadata rows are what tell you which object version is the correct one to restore or select.
5

Bring services back in order

Provisioner worker, then product worker, then API, then frontend. Workers should be running and heartbeating before the API is trusted to report ready, since the worker readiness check depends on it.
6

Require readiness, then reconcile

Wait for /health/ready to report clean, then reconcile any queued/running jobs left over from before the incident — the triage queries above are exactly how — before reopening traffic to real users.
Redis is explicitly not a business-data or job authority in this ordering. Losing it resets login rate-limit windows; it never loses a dataset, an upload, or a job, because none of those are stored there. Restoring Redis is not on the critical path to reopening traffic.

Alerting

Only AWS currently provisions alarms as infrastructure. deploy/aws/main.tf creates an SNS topic (aws_sns_topic.alerts, KMS-encrypted with alias/aws/sns) with an optional email subscription (var.alert_email), and wires three CloudWatch alarms to it:
deploy/azure/main.bicep provisions Log Analytics for log aggregation but does not provision an equivalent set of metric alerts or an action group. Until that gap is closed, Azure deployments need an operator-configured alerting layer on top of Log Analytics to match the coverage AWS gets for free from Terraform — do not assume parity between the two clouds here.