Skip to main content
Observability in the product API is deliberately small and provider-neutral: one structured request log, one Prometheus scrape endpoint, and a heartbeat mechanism workers use to advertise their own capability rather than the API assuming it. There is no bundled APM agent or vendor SDK to configure — every signal here is either a stdlib logging call or a plain database row.

​
Request observability middleware

services/api/observability.py’s install_request_observability() is mounted on the product app in create_product_app() (services/api/product_app.py) as one ASGI middleware, applied to every request regardless of route:
1

Request ID resolution

Reads x-request-id from the inbound request. If it matches ^[A-Za-z0-9._:-]{1,128}$, it’s trusted and reused (so a caller — or the frontend proxy — can correlate a request across hops); otherwise a fresh uuid4().hex is generated. Either way, request.state.request_id is set for the rest of the request’s lifetime and the same value is echoed back in the response’s x-request-id header.
2

Timing

time.perf_counter() brackets the call to call_next(request).
3

One structured completion log

In a finally block (so it fires even on an unhandled exception, where status_code defaults to 500), it logs one JSON line to the causeloop.http logger: {"event": "http_request", "request_id", "method", "path", "status_code", "duration_ms"}, keys sorted, compact separators — built for a log aggregator to parse, not for humans to read inline.
This is the only per-request logging the product API does — there’s no separate access-log format to reconcile with it.

​
GET /metrics

services/api/metrics_api.py exposes a Prometheus-format scrape endpoint, computed fresh from PostgreSQL on every scrape rather than accumulated in-process. That’s a deliberate tradeoff: it’s correct under multiple API replicas (an in-process counter would only reflect one replica’s traffic), at the cost of a few extra queries per scrape — acceptable at Prometheus’s usual 15–30s interval.
/metrics is employee-only (metrics_router = APIRouter(dependencies=[Depends(get_current_employee)])) — it exposes cross-tenant operational data (per-tenant job counts, durations, LLM token usage), which is exactly the kind of cross-tenant visibility the platform otherwise restricts to authenticated staff. Scraping isn’t treated as a per-request user action worth an audit_log row (unlike impersonation); it’s gated by employee auth alone.
Metrics published, all computed inside one CollectorRegistry built per request: A single tenant’s per-tenant metrics failing (e.g. a transient connection issue) is caught and skipped (except Exception: continue) rather than blanking the entire scrape — one bad tenant shouldn’t take down cross-tenant visibility into every other tenant.

​
Worker heartbeats as capability advertisement

control.runtime_heartbeats (one row per component, upserted on conflict) is not a liveness ping in the trivial sense — it’s how a worker tells the API what it can currently do, not just that it’s alive. record_worker_heartbeat() (services/pipeline/jobs.py) writes component, instance_id, heartbeat_at, and a detail JSONB blob of {"job_types": [...]} reflecting the actual set of job types that worker process was started with (CAUSELOOP_WORKER_JOB_TYPES) and passed its own startup validation for (see below). The job worker loop (services/pipeline/job_worker.py) heartbeats every 10 seconds (a loop.time() check inside its poll loop, not a separate timer task) under CAUSELOOP_WORKER_COMPONENT (default product-worker; the provisioner worker sets this to provisioner-worker in infra/docker-compose.product.yml). This distinction matters concretely for email: validate_worker_configuration() refuses to let a worker heartbeat send_email as a supported job type unless email_delivery_readiness() and secret_encryption_readiness() both report ready — a worker that can’t actually send email (misconfigured provider, no encryption key) never advertises that capability, rather than heartbeating healthy and then failing every email job it claims.

​
Health endpoint semantics

Three health routes are registered directly on the app inside create_product_app() (services/api/product_app.py) — outside the OBSERVABILITY_OPERATIONS/AUTH_OPERATIONS allowlist mechanism that gates the rest of the published surface — and all three are in the frontend proxy’s PUBLIC_PATHS — reachable without a session, since an external deployment smoke check should only need the frontend’s public URL:
  • GET /health, GET /health/live — process liveness only: {"status": "ok"}, no dependency checks. This is what an orchestrator’s liveness probe should hit — a slow database must never make the process look dead and get killed.
  • GET /health/ready — the full dependency readiness check, for a readiness probe (should this instance receive traffic yet?). Returns 503 if any check fails.
/health/ready’s checks, each an independent entry in the response’s checks object: The worker check is the most involved: it reads CAUSELOOP_REQUIRED_WORKERS (default product-worker,provisioner-worker), looks up control.runtime_heartbeats rows for exactly those components with heartbeat_at within the last CAUSELOOP_WORKER_READY_SECONDS (default 45) seconds, and additionally checks that each one’s advertised job_types (from its heartbeat detail) is a superset of that component’s hardcoded required set (REQUIRED_WORKER_JOB_TYPES: product-worker must advertise materialize_insights, send_email, train_tenant_model; provisioner-worker must advertise provision_tenant). A worker process that’s running but hasn’t heartbeated its full contract — or has silently disabled send_email because email delivery isn’t configured — makes readiness fail, not just a missing process. The redis check only runs a live PING if REDIS_URL is set; otherwise it reports ready: not redis_required — so a deployment that hasn’t configured Redis is only unready if CAUSELOOP_REDIS_REQUIRED says it must be (which defaults to true in production, false elsewhere — see Security).

​
control.audit_log and per-tenant audit_log

Two separate audit trails exist, at two different scopes — see Tenancy and data model for their table definitions:
  • control.audit_log — employee/control-plane actions only: employee_login, employee_impersonation, and other staff actions on tenants. actor_type is employee, tenant_user, or system.
  • <tenant schema>.audit_log — every human interaction on that tenant’s own reviews, CAPs, and exports (_write_audit_log() in services/api/remediation_api.py), recording actor_email, session_id, action, resource_type/resource_id, a detail JSONB blob, and ip_address. GET /remediation/audit-log (tenant-authenticated, optionally filtered by resource_type/resource_id, capped at 200 rows) is how the console’s own audit views read it back — this is application-level observability of who did what, distinct from the request-level logging above.
Both are ordinary Postgres tables, queryable directly for incident investigation — there’s no separate audit pipeline or external sink to reach for.

​
Where logs live locally

The containerized product stack (infra/docker-compose.product.yml) runs the API, both workers, and the frontend as separate Compose services (api, product-worker, provisioner-worker, frontend, plus postgres, redis, minio, mailpit). Every process’s stdout/stderr — including the structured causeloop.http request-completion lines above — goes to Docker’s own log driver, read with:
Omit -f for a one-shot dump instead of following, or omit the service name entirely to interleave every service’s output in one stream.