> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> The request observability middleware, GET /metrics, worker heartbeats as capability advertisement, health endpoint semantics, and where logs live locally.

Observability in the product API is deliberately small and provider-neutral: one structured request log, one Prometheus scrape endpoint, and a heartbeat mechanism workers use to advertise their own capability rather than the API assuming it. There is no bundled APM agent or vendor SDK to configure — every signal here is either a stdlib `logging` call or a plain database row.

## Request observability middleware

`services/api/observability.py`'s `install_request_observability()` is mounted on the product app in `create_product_app()` (`services/api/product_app.py`) as one ASGI middleware, applied to every request regardless of route:

<Steps>
  <Step title="Request ID resolution">
    Reads `x-request-id` from the inbound request. If it matches `^[A-Za-z0-9._:-]{1,128}$`, it's trusted and reused (so a caller — or the frontend proxy — can correlate a request across hops); otherwise a fresh `uuid4().hex` is generated. Either way, `request.state.request_id` is set for the rest of the request's lifetime and the same value is echoed back in the response's `x-request-id` header.
  </Step>

  <Step title="Timing">
    `time.perf_counter()` brackets the call to `call_next(request)`.
  </Step>

  <Step title="One structured completion log">
    In a `finally` block (so it fires even on an unhandled exception, where `status_code` defaults to `500`), it logs one JSON line to the `causeloop.http` logger: `{"event": "http_request", "request_id", "method", "path", "status_code", "duration_ms"}`, keys sorted, compact separators — built for a log aggregator to parse, not for humans to read inline.
  </Step>
</Steps>

This is the only per-request logging the product API does — there's no separate access-log format to reconcile with it.

## `GET /metrics`

`services/api/metrics_api.py` exposes a Prometheus-format scrape endpoint, computed fresh from PostgreSQL on every scrape rather than accumulated in-process. That's a deliberate tradeoff: it's correct under multiple API replicas (an in-process counter would only reflect one replica's traffic), at the cost of a few extra queries per scrape — acceptable at Prometheus's usual 15–30s interval.

<Warning>
  `/metrics` is **employee-only** (`metrics_router = APIRouter(dependencies=[Depends(get_current_employee)])`) — it exposes cross-tenant operational data (per-tenant job counts, durations, LLM token usage), which is exactly the kind of cross-tenant visibility the platform otherwise restricts to authenticated staff. Scraping isn't treated as a per-request user action worth an `audit_log` row (unlike impersonation); it's gated by employee auth alone.
</Warning>

Metrics published, all computed inside one `CollectorRegistry` built per request:

| Metric                                        | Labels                  | Source                                                                       |
| --------------------------------------------- | ----------------------- | ---------------------------------------------------------------------------- |
| `causeloop_platform_jobs_total`               | `job_type`, `status`    | `COUNT(*) GROUP BY` over `control.platform_jobs`                             |
| `causeloop_platform_job_duration_seconds_avg` | `job_type`              | Average `updated_at - created_at` over succeeded jobs, per type              |
| `causeloop_tenants_total`                     | `lifecycle_state`       | `COUNT(*) GROUP BY` over `control.tenants`                                   |
| `causeloop_ingest_jobs_total`                 | `tenant_slug`, `status` | Per-tenant `ingest_jobs` count, one `tenant_uow()` per non-offboarded tenant |
| `causeloop_llm_tokens_used_this_month`        | `tenant_slug`           | `llm_settings.tokens_used_this_month`, per tenant                            |
| `causeloop_pipeline_queue_depth`              | —                       | `COUNT(*)` of `control.platform_jobs` where `status='queued'`                |

A single tenant's per-tenant metrics failing (e.g. a transient connection issue) is caught and skipped (`except Exception: continue`) rather than blanking the entire scrape — one bad tenant shouldn't take down cross-tenant visibility into every other tenant.

## Worker heartbeats as capability advertisement

`control.runtime_heartbeats` (one row per `component`, upserted on conflict) is not a liveness ping in the trivial sense — it's how a worker tells the API **what it can currently do**, not just that it's alive. `record_worker_heartbeat()` (`services/pipeline/jobs.py`) writes `component`, `instance_id`, `heartbeat_at`, and a `detail` JSONB blob of `{"job_types": [...]}` reflecting the actual set of job types that worker process was started with (`CAUSELOOP_WORKER_JOB_TYPES`) and passed its own startup validation for (see below). The job worker loop (`services/pipeline/job_worker.py`) heartbeats every 10 seconds (a `loop.time()` check inside its poll loop, not a separate timer task) under `CAUSELOOP_WORKER_COMPONENT` (default `product-worker`; the provisioner worker sets this to `provisioner-worker` in `infra/docker-compose.product.yml`).

This distinction matters concretely for email: `validate_worker_configuration()` refuses to let a worker heartbeat `send_email` as a supported job type unless `email_delivery_readiness()` and `secret_encryption_readiness()` both report ready — a worker that can't actually send email (misconfigured provider, no encryption key) never advertises that capability, rather than heartbeating healthy and then failing every email job it claims.

## Health endpoint semantics

Three health routes are registered directly on the app inside `create_product_app()` (`services/api/product_app.py`) — outside the `OBSERVABILITY_OPERATIONS`/`AUTH_OPERATIONS` allowlist mechanism that gates the rest of the published surface — and all three are in the frontend proxy's `PUBLIC_PATHS` — reachable without a session, since an external deployment smoke check should only need the frontend's public URL:

* **`GET /health`, `GET /health/live`** — process liveness only: `{"status": "ok"}`, no dependency checks. This is what an orchestrator's liveness probe should hit — a slow database must never make the process look dead and get killed.
* **`GET /health/ready`** — the full dependency readiness check, for a readiness probe (should this instance receive traffic yet?). Returns `503` if any check fails.

`/health/ready`'s checks, each an independent entry in the response's `checks` object:

```mermaid theme={null}
flowchart TD
    R["GET /health/ready"] --> DB[database: SELECT 1]
    R --> W[worker: heartbeat freshness + job-type coverage]
    R --> OS[object_storage: object_store_readiness]
    R --> CE[credential_encryption: secret_encryption_readiness]
    R --> ED[email_delivery: email_delivery_readiness]
    R --> RD[redis: PING, only if REDIS_URL set]
    DB --> Overall{all ready?}
    W --> Overall
    OS --> Overall
    CE --> Overall
    ED --> Overall
    RD --> Overall
    Overall -->|yes| OK["200 status: ready"]
    Overall -->|no| NR["503 status: not_ready"]
```

The **worker** check is the most involved: it reads `CAUSELOOP_REQUIRED_WORKERS` (default `product-worker,provisioner-worker`), looks up `control.runtime_heartbeats` rows for exactly those components with `heartbeat_at` within the last `CAUSELOOP_WORKER_READY_SECONDS` (default 45) seconds, and additionally checks that each one's advertised `job_types` (from its heartbeat `detail`) is a superset of that component's hardcoded required set (`REQUIRED_WORKER_JOB_TYPES`: `product-worker` must advertise `materialize_insights, send_email, train_tenant_model`; `provisioner-worker` must advertise `provision_tenant`). A worker process that's running but hasn't heartbeated its full contract — or has silently disabled `send_email` because email delivery isn't configured — makes readiness fail, not just a missing process.

The **redis** check only runs a live `PING` if `REDIS_URL` is set; otherwise it reports `ready: not redis_required` — so a deployment that hasn't configured Redis is only unready if `CAUSELOOP_REDIS_REQUIRED` says it must be (which defaults to `true` in production, `false` elsewhere — see [Security](/architecture/security)).

## `control.audit_log` and per-tenant `audit_log`

Two separate audit trails exist, at two different scopes — see [Tenancy and data model](/architecture/tenancy-and-data-model) for their table definitions:

* **`control.audit_log`** — employee/control-plane actions only: `employee_login`, `employee_impersonation`, and other staff actions on tenants. `actor_type` is `employee`, `tenant_user`, or `system`.
* **`<tenant schema>.audit_log`** — every human interaction on that tenant's own reviews, CAPs, and exports (`_write_audit_log()` in `services/api/remediation_api.py`), recording `actor_email`, `session_id`, `action`, `resource_type`/`resource_id`, a `detail` JSONB blob, and `ip_address`. `GET /remediation/audit-log` (tenant-authenticated, optionally filtered by `resource_type`/`resource_id`, capped at 200 rows) is how the console's own audit views read it back — this is application-level observability of *who did what*, distinct from the request-level logging above.

Both are ordinary Postgres tables, queryable directly for incident investigation — there's no separate audit pipeline or external sink to reach for.

## Where logs live locally

The containerized product stack (`infra/docker-compose.product.yml`) runs the API, both workers, and the frontend as separate Compose services (`api`, `product-worker`, `provisioner-worker`, `frontend`, plus `postgres`, `redis`, `minio`, `mailpit`). Every process's stdout/stderr — including the structured `causeloop.http` request-completion lines above — goes to Docker's own log driver, read with:

```bash theme={null}
docker compose -f infra/docker-compose.product.yml logs -f api
docker compose -f infra/docker-compose.product.yml logs -f product-worker
docker compose -f infra/docker-compose.product.yml logs -f provisioner-worker
```

Omit `-f` for a one-shot dump instead of following, or omit the service name entirely to interleave every service's output in one stream.

## Related

* [Security model](/architecture/security)
* [Tenancy and data model](/architecture/tenancy-and-data-model)
* [Workers and jobs](/architecture/workers-and-jobs)
* [Monitoring and incidents](/deployment/monitoring-and-incidents)