> ## Documentation Index > Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring and incidents > Operating the deployed product: what each /health/ready gate failure means and how to fix it, reading /metrics, audit-log forensics, control.platform_jobs triage queries, and restart/recovery order. [Observability](/architecture/observability) and [Workers and jobs](/architecture/workers-and-jobs) explain *how* request logging, `/metrics`, and the job queue are built. This page is the on-call companion: given a paged alert or a support ticket, what does each signal actually mean in production, and what do you run next. ## Readiness vs. liveness `GET /health` / `GET /health/live` (`services/api/product_app.py`) only proves the API process can serve a request — it has no dependency checks and returns `{"status": "ok"}` unconditionally. **Never route traffic, and never gate a deployment or rollback decision, on liveness alone** — a live-but-not-ready API will happily accept traffic it cannot correctly serve (queued jobs that never process, uploads that 503, sign-ins that silently never send their reset link). `GET /health/ready` is the real gate. It returns `503` the instant any one of six independent checks is `false`, and names exactly which one: | Check | What it verifies | Common production root cause | Where to look next | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `database` | A live `SELECT 1` against the control schema, inside the same call that reads worker heartbeats | Connection pool exhaustion, RDS/Flexible Server failover in progress, or a network path change between the API and Postgres | RDS/Flexible Server events; `CAUSELOOP_DB_POOL_MAX_SIZE` if the pool itself is undersized | | `worker` | Every component in `CAUSELOOP_REQUIRED_WORKERS` (default `product-worker,provisioner-worker`) has a `control.runtime_heartbeats` row within `CAUSELOOP_WORKER_READY_SECONDS` (45s default), each advertising at least the job types `product_app.py`'s `REQUIRED_WORKER_JOB_TYPES` requires for that component (a subset check — advertising extra job types beyond what's required still passes) (`product-worker` → `materialize_insights`, `send_email`, `train_tenant_model`; `provisioner-worker` → `provision_tenant`) | **Missing heartbeat**: the worker process crashed, was scaled to zero, or its last heartbeat is simply older than 45s under load. **Capability mismatch**: the worker is running with a `CAUSELOOP_WORKER_JOB_TYPES` set that doesn't cover the required types for its component — most often `send_email` silently dropped because `validate_worker_configuration()` (`services/pipeline/job_worker.py`) refuses to let a worker claim it while email or credential-encryption readiness is broken, so the worker raises and never starts heartbeating at all | `SELECT * FROM control.runtime_heartbeats;` for `heartbeat_at` and the `detail->'job_types'` array; worker process logs for a startup `RuntimeError` from `validate_worker_configuration` | | `object_storage` | The configured bucket/container is reachable (`head_bucket` on S3, `get_container_properties` on Blob) | Credentials rotated without redeploying the runtime identity; bucket/container renamed; VPC endpoint or private-endpoint DNS broken | `services/storage/object_store.py`'s `readiness()` returns `{"ready": false, "detail": }` — that exception name (e.g. `ClientError`, `HttpResponseError`) is your first lead | | `credential_encryption` | The configured secret provider (`fernet` local-only / `aws-kms` / `azure-key-vault`) reports itself ready | In production this should never be `fernet` — `secret_encryption_readiness()` (`services/api/secrets.py`) reports `fernet` as not-ready whenever `CAUSELOOP_ENVIRONMENT=production`. For KMS/Key Vault, the usual cause is the key id env var (`AWS_KMS_KEY_ID` / `AZURE_KEY_VAULT_KEY_ID`) pointing at a key the runtime identity's IAM/RBAC role can no longer decrypt with | Confirm the key id matches the one actually granted to the runtime role in [AWS deployment](/deployment/aws) / [Azure deployment](/deployment/azure) | | `email_delivery` | The configured provider (`aws-ses` / `azure-communication` in production) has its required env vars present and is not a local provider | Sender address (`CAUSELOOP_EMAIL_FROM`) or region/endpoint env var missing after a redeploy; **this check never sends a probe message**, so it can be green while a real client's invitation still fails to arrive | See [Client onboarding runbook](/deployment/client-onboarding-runbook#production-email-delivery-vs-mailpit) for the SES-sandbox / sender-verification caveat this check cannot see | | `redis` | A successful `PING`, required whenever `CAUSELOOP_REDIS_REQUIRED` is `true` (defaults to `true` whenever `CAUSELOOP_ENVIRONMENT=production`) | ElastiCache/Azure Managed Redis failover or auth-token rotation | Redis is not a business-data or job authority — see [Restart and recovery order](#restart-and-recovery-order) below for why this is lower-priority than the other five | A `503` with one check `false` is strictly more useful than a bare connection-refused error from a load balancer health check misconfigured against `/health/live` instead of `/health/ready` — always point your platform's health-check target at `/health/ready`. ## Reading `GET /metrics` `services/api/metrics_api.py` exposes a Prometheus-format scrape endpoint, gated behind employee authentication (`Depends(get_current_employee)`) because it carries cross-tenant operational data — point your scraper at it with a service-account employee session, not an anonymous target. | Metric | Labels | Read it as | | --------------------------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `causeloop_platform_jobs_total` | `job_type`, `status` | A rising `failed` count for one `job_type` while others stay flat points at a specific job handler (ingestion, training, provisioning, email), not a general outage. | | `causeloop_platform_job_duration_seconds_avg` | `job_type` | A step change here (not a gradual drift) usually means an external dependency that job type calls got slower — an LLM provider for `train_tenant_model`, the email provider for `send_email`. | | `causeloop_tenants_total` | `lifecycle_state` | Cross-check against expected onboarding volume; a `draft` count that only grows and never drains into `onboarding` means provisioning itself is stuck platform-wide, not per-tenant. | | `causeloop_ingest_jobs_total` | `tenant_slug`, `status` | Per-tenant — useful for confirming a specific client's complaint ("my upload never processed") against real state rather than trusting their description of what happened. | | `causeloop_llm_tokens_used_this_month` | `tenant_slug` | Compare against each tenant's `llm_monthly_token_budget` (set via [training configuration](/features/client-onboarding-pipeline#4-training-configuration-not-itself-a-lifecycle-transition)) before assuming a training failure is a code bug rather than a budget ceiling. | | `causeloop_pipeline_queue_depth` | — | The single best "is the queue backing up" number: `COUNT(*)` of `control.platform_jobs` where `status='queued'`. Recomputed fresh from PostgreSQL on every scrape, so it's correct across multiple API replicas — no in-process counter to reconcile. | A single tenant's per-tenant metrics query failing (a transient connection blip) is caught and skipped rather than blanking the whole scrape (`except Exception: continue` in `_tenant_metrics`) — a gap for one `tenant_slug` in `causeloop_ingest_jobs_total` between two scrapes is not itself an incident. ## Audit log forensics Two separate audit logs exist, and an incident investigation usually needs both: * **`control.audit_log`** (`ControlAuditLog`, `services/api/db/models.py`) — control-plane/employee actions only: `tenant_created`, `checkpoint_activated`, `dataset_ingested`, `tenant_go_live`, and unconditionally, every `employee_impersonation`. Columns: `actor_type`, `actor_id`, `tenant_id`, `action`, `resource_type`, `resource_id`, `request_id`, `detail` (JSONB), `created_at`. * **The per-tenant `audit_log` table** (`TenantAuditLog`, one per `tenant_` schema) — every CAP/review mutation a tenant user makes, exposed read-only through `GET /remediation/audit-log` (`resource_type`/`resource_id` filters, `limit` clamped to `[1, 200]`). See [Remediation](/features/remediation) for the full schema and query parameters. `control.audit_log.request_id` exists in the schema (added for correlating an audit row back to the structured HTTP log's `request_id` from [Observability](/architecture/observability#request-observability-middleware)) but no current code path populates it — every `ControlAuditLog` insert in `onboarding_api.py` and `auth/employee_auth.py` / `auth/tenant_auth.py`'s employee-facing actions omits it. Do not expect to join an audit row to a specific HTTP request by `request_id` today; correlate by `actor_id` + `tenant_id` + a tight `created_at` window instead. For a cross-tenant incident (e.g. "who impersonated tenant X, and when"), query `control.audit_log` directly: ```sql theme={null} SELECT actor_id, action, resource_type, resource_id, detail, created_at FROM control.audit_log WHERE tenant_id = :tenant_id ORDER BY created_at DESC LIMIT 100; ``` For a single tenant's own activity trail, prefer the `GET /remediation/audit-log` endpoint over a direct schema query where possible — it already carries `actor_email`, `session_id`, and `ip_address` alongside the action, which the control-plane log does not. ## `control.platform_jobs` triage queries Every asynchronous effect in the product — provisioning, training, materialization, email — is one row in `control.platform_jobs`. These queries (see [Workers and jobs](/architecture/workers-and-jobs) for the full schema) cover the triage questions that come up during an incident: ```sql theme={null} -- Current backlog by type and status (the same numbers /metrics reports, but filterable) SELECT job_type, status, count(*) FROM control.platform_jobs GROUP BY job_type, status ORDER BY job_type, status; -- Jobs claimed but not making progress -- candidates for a stuck worker SELECT id, tenant_id, job_type, locked_by, locked_at, attempts, max_attempts FROM control.platform_jobs WHERE status = 'running' AND locked_at < now() - interval '5 minutes' ORDER BY locked_at; -- Jobs approaching their retry ceiling -- likely to fail terminally soon SELECT id, tenant_id, job_type, attempts, max_attempts, error_detail, available_at FROM control.platform_jobs WHERE status = 'queued' AND attempts >= max_attempts - 1 ORDER BY available_at; -- Failed jobs in the last hour, grouped by cause SELECT job_type, error_detail, count(*) FROM control.platform_jobs WHERE status = 'failed' AND finished_at > now() - interval '1 hour' GROUP BY job_type, error_detail ORDER BY count(*) DESC; ``` `error_detail` is deliberately generic for `send_email` jobs (`"Email delivery failed; provider detail withheld."`) — `job_failure_detail()` (`services/pipeline/jobs.py`) withholds the real provider exception specifically so a recipient address or token fragment never lands in a job row or a worker log. Do not expect provider-level detail from this column for email failures; check the email provider's own delivery logs (SES/ACS console) instead. A `running` job whose `locked_at` is older than `CAUSELOOP_JOB_STALE_AFTER_SECONDS` (900s / 15 minutes default) is not stuck forever on its own — `recover_stale_jobs()` runs once at every worker process startup and requeues it (or fails it terminally if `attempts >= max_attempts`) automatically. If you see a job stuck well past that threshold with no worker process restart in sight, that itself is the incident: the recovery sweep only runs at startup, not on a timer, so a long-lived worker process with a genuinely wedged claim will not self-heal until it restarts. ## Restart and recovery order When integrity is uncertain, `docs/CLOUD_OPERATIONS.md`'s incident priorities are the practiced order — the same one echoed in [Migrations](/deployment/migrations#rollback-posture) for a deploy-rollback specifically, restated here for a live incident: Scale the API and both workers to zero if data integrity is in question. This is a blunt but safe first move — no new jobs enqueue, no new writes land, and nothing already durable in PostgreSQL is at risk from stopping traffic. Leave PostgreSQL job rows, logs, request IDs, and object versions exactly as they are. Do not truncate `control.platform_jobs` or delete object versions while triaging — they are the only record of what was in flight. It is the metadata and business authority — tenant records, job state, and dataset/checkpoint metadata all live there. Nothing else can be trusted to reconcile until this is correct. See [Backup and recovery](/deployment/backup-and-recovery) for the restore procedure itself. Only after PostgreSQL is trustworthy, because the metadata rows are what tell you which object version is the correct one to restore or select. Provisioner worker, then product worker, then API, then frontend. Workers should be running and heartbeating before the API is trusted to report `ready`, since the `worker` readiness check depends on it. Wait for `/health/ready` to report clean, then reconcile any `queued`/`running` jobs left over from before the incident — the triage queries above are exactly how — before reopening traffic to real users. Redis is explicitly **not** a business-data or job authority in this ordering. Losing it resets login rate-limit windows; it never loses a dataset, an upload, or a job, because none of those are stored there. Restoring Redis is not on the critical path to reopening traffic. ## Alerting Only AWS currently provisions alarms as infrastructure. `deploy/aws/main.tf` creates an SNS topic (`aws_sns_topic.alerts`, KMS-encrypted with `alias/aws/sns`) with an optional email subscription (`var.alert_email`), and wires three CloudWatch alarms to it: | Alarm | Fires when | | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `${name}-frontend-5xx` | More than 5 `HTTPCode_Target_5XX_Count` on the frontend ALB target group over two 5-minute periods | | `${name}-database-low-storage` | RDS `FreeStorageSpace` drops below 5 GB, averaged over two 5-minute periods | | `${name}--not-running` (one per ECS service: frontend, API, product-worker, provisioner-worker) | `RunningTaskCount` drops below 1 for two consecutive 1-minute periods, only once `activate_services=true`; missing data is treated as breaching, not healthy | `deploy/azure/main.bicep` provisions Log Analytics for log aggregation but does not provision an equivalent set of metric alerts or an action group. Until that gap is closed, Azure deployments need an operator-configured alerting layer on top of Log Analytics to match the coverage AWS gets for free from Terraform — do not assume parity between the two clouds here. ## Related * [Observability](/architecture/observability) * [Workers and jobs](/architecture/workers-and-jobs) * [Backup and recovery](/deployment/backup-and-recovery) * [Security model](/architecture/security)