> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Backend in depth
> How services/api/product_app.py assembles the deployed API: the route allowlist, router inventory, request observability, and the /health/ready model.
`services/api/product_app.py` is the executable target for the deployed backend. Its own module docstring states the reason it exists as a separate file from `services/api/main.py`: the historical app "also hosts the local research workbench, legacy file-backed client onboarding, and process-global `/mvp1` routes. Importing that app therefore creates stores and exposes operations the customer/staff product does not need." `product_app.py` builds a second, deliberately smaller FastAPI app from the same route modules, and lists every operation it publishes explicitly.
## `create_product_app()`: the assembly
```python theme={null}
def create_product_app() -> FastAPI:
product = FastAPI(
title="CL MVP Product API",
version="1.0.0",
description="Allowlisted customer and staff product surface.",
)
install_request_observability(product)
product.include_router(_selected_router(auth_router, AUTH_OPERATIONS, label="auth"))
product.include_router(_selected_router(onboarding_router, ONBOARDING_OPERATIONS, label="onboarding"))
product.include_router(_selected_router(insights_router, INSIGHT_OPERATIONS, label="insights"))
product.include_router(_selected_router(sources_router, SOURCE_OPERATIONS, label="sources"))
product.include_router(_selected_router(members_router, MEMBER_OPERATIONS, label="members"))
product.include_router(_selected_router(remediation_router, REMEDIATION_OPERATIONS, label="remediation"))
product.include_router(_selected_router(metrics_router, OBSERVABILITY_OPERATIONS, label="observability"))
# /health, /health/live, /health/ready registered directly on `product`
return product
```
`app = create_product_app()` at module scope is the object `uvicorn services.api.product_app:app` (the repo's `Dockerfile` `CMD`) actually serves. There is no additional route registration anywhere else in the deployed process.
## The route-allowlist pattern
Every one of the seven `include_router` calls above passes the *full* router object from its source module (`auth_router`, `onboarding_router`, etc.) through `_selected_router()`, which filters it down to an explicit `frozenset` of `(METHOD, path)` tuples — `AUTH_OPERATIONS`, `ONBOARDING_OPERATIONS`, and so on. `product_app.py` also defines `TENANT_OPERATIONS` (the union of `INSIGHT_OPERATIONS`, `MEMBER_OPERATIONS`, `SOURCE_OPERATIONS`, and `REMEDIATION_OPERATIONS`) and `MVP_ROUTE_OPERATIONS` — the union of `AUTH_OPERATIONS`, `ONBOARDING_OPERATIONS`, `TENANT_OPERATIONS`, and `OBSERVABILITY_OPERATIONS` — as the single frozenset tests and contract checks treat as the full published surface. This is the load-bearing design decision in this file:
* Source routers hold more routes than the product ever exposes. For example, `onboarding_router` also defines `/{tenant_id}/rename`, `/{tenant_id}/materialize-now`, `/{tenant_id}/run-migrations`, and `/{tenant_id}/import-checkpoint` — each one an operator/rehearsal escape hatch documented in its own `TEMPORARY:` code comment (e.g. `run-migrations` exists because this deployment has no direct external Postgres access from a local dev machine, so an already-provisioned tenant can be migrated to head over HTTPS instead) — and none of them appear in `ONBOARDING_OPERATIONS`, so `_selected_router` drops them. `auth_router` similarly defines `/admin/sessions`, `/admin/password-reset/*`, `/password-reset/*`, and `/dev/mailbox` (a local-only mailbox reader) that never reach the product surface.
* `_selected_router` is defensive in two directions, both enforced with a `RuntimeError` at import time (i.e., app boot fails loudly, not silently at request time):
* **Missing operations**: if an allowlisted `(method, path)` pair isn't actually registered on the source router, boot fails with `"{label} MVP operations are not registered"`. You cannot allowlist an endpoint that doesn't exist yet.
* **Mixed methods on one path**: if a path has some HTTP methods allowlisted and others not (e.g., a route registered for both `GET` and `DELETE` where only `GET` is approved), boot fails with `"{label} route {path!r} mixes allowed and unapproved methods"`. A path is all-or-nothing per method — there is no way to partially publish a route.
Adding an endpoint to a router file (`onboarding_api.py`, `sources_api.py`, etc.) does **not** publish it. You must also add its `(METHOD, path)` tuple to the matching `*_OPERATIONS` frozenset in `product_app.py`, or `_selected_router` will silently exclude it — silently from the caller's perspective, though the app would still boot fine since nothing requires that path. The reverse mistake — allowlisting a path before the router defines it — fails loudly at boot, which is the safer failure mode by design.
## Published surface: 44 operations
The published surface is exactly **44 HTTP operations**: 41 allowlisted business operations plus 3 always-on system endpoints (`/health`, `/health/live`, `/health/ready`) registered directly on `product` outside the allowlist mechanism, since they carry no tenant or employee authorization and must always answer.
| Router (label) | Operations | Representative paths |
| --------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `auth` | 8 | `GET /workspaces`, `POST /login`, `POST /admin/login`, `GET /me`, `GET /admin/me`, `POST /accept-invite`, `POST /logout`, `POST /admin/logout` |
| `onboarding` | 20 | `GET`/`POST /admin/tenants`, `.../provision`, `.../ingest/upload`, `.../ingest/commit`, `.../training-config`, `.../data-policy`, `.../sources/seed`, `.../train`, `.../checkpoints`, `.../retrain-recommendation`, `.../activate`, `.../invite`, `.../invitations`, `.../go-live` |
| `insights` | 1 | `GET /insights/snapshot` |
| `sources` | 3 | `GET`/`POST /sources`, `POST /sources/{source_id}/upload` |
| `members` | 1 | `GET /members` |
| `remediation` | 7 | `GET`/`POST /remediation/caps`, `PATCH /remediation/caps/{cap_id}`, `GET /remediation/reviews`, `PATCH /remediation/reviews/{review_id}`, `GET /remediation/audit-log`, `GET /remediation/export.xlsx` |
| `observability` | 1 | `GET /metrics` |
| **Total** | **41** | plus 3 health endpoints = **44** |
Each router's own module docstring explains its authorization model and is the right place to read next:
* `auth_api.py` — employee (Onboarding Portal) and tenant-user (Client Portal) session auth. Every mutating route depends on `require_csrf`; login/accept-invite are exempt because there is no session yet to protect.
* `onboarding_api.py` — the staff-facing tenant lifecycle and checklist pipeline. Every route requires an authenticated employee; mutations additionally require the `onboarding_admin` control role (`support_readonly` employees can read but not act).
* `insights_api.py` — tenant-facing insight reads backed only by materialized `insight_collections`; never computes on request.
* `members_api.py` — tenant-facing member directory.
* `metrics_api.py` — Prometheus-format observability endpoint (job durations, queue depth, LLM usage), gated behind `get_current_employee`.
* `remediation_api.py` — the real, RLS-isolated, per-tenant CAP/review/audit-log system built on `caps`/`reviews`/`audit_log` tables, replacing the legacy process-global `/mvp1/caps` path. Every mutation writes a tenant `audit_log` row.
* `sources_api.py` — tenant-facing source connector management. Credentials are write-only: encrypted on submission, never returned after creation (only a last-4-style hint).
See [Tenancy and data model](/architecture/tenancy-and-data-model) for how `TenantPrincipal`/`EmployeePrincipal` and RLS enforce these boundaries, and [API reference overview](/api-reference/overview) for request/response contracts.
## Request observability
`install_request_observability()` (`services/api/observability.py`) is the only middleware the product app installs. It is deliberately small and provider-neutral:
* Accepts an inbound `x-request-id` header only if it matches `^[A-Za-z0-9._:-]{1,128}$`; otherwise generates a fresh `uuid4().hex`. This prevents a client from injecting arbitrary content into structured logs via the header.
* Stores the resolved ID on `request.state.request_id`, echoes it back as the `x-request-id` response header, and emits exactly one structured JSON log line per request (`event: "http_request"`, method, path, status code, duration in milliseconds) to the `causeloop.http` logger — in the `finally` block, so it fires even when the handler raises.
There is no request/response body logging, no tracing SDK, and no external APM call in this module — it is the floor every deployment gets for free, and the natural place other export/monitoring integrations should hook if you add them later. See [Observability](/architecture/observability) for what consumes these logs in each environment.
## The `/health/ready` model
The product app exposes three health routes, all defined directly in `create_product_app()`:
* **`GET /health`** and **`GET /health/live`** — trivial liveness: always return `{"status": "ok"}`. Suitable for a container orchestrator's liveness probe; they do not touch the database.
* **`GET /health/ready`** — the real readiness gate. It runs five independent checks and returns HTTP `503` with `{"status": "not_ready", ...}` the moment any one of them fails; `200` with `{"status": "ready", ...}` only when all pass.
Runs `SELECT 1` against the control-plane connection (fail-fast if Postgres is unreachable), then checks `control.runtime_heartbeats` for a row per `CAUSELOOP_REQUIRED_WORKERS` (default `product-worker,provisioner-worker`) heartbeated within the last `CAUSELOOP_WORKER_READY_SECONDS` (default 45s). This alone would only prove a worker process is *alive* — `_worker_capabilities_ready()` goes further and cross-checks each heartbeat's advertised `detail.job_types` against `REQUIRED_WORKER_JOB_TYPES` (`product-worker` must advertise all of `materialize_insights`, `send_email`, `train_tenant_model`; `provisioner-worker` must advertise `provision_tenant`). A `product-worker` process that is up but was started with an incomplete `CAUSELOOP_WORKER_JOB_TYPES` env var fails readiness even though its heartbeat row exists — readiness is capability-gated, not just liveness-gated.
`object_store_readiness()` (`services/storage/object_store.py`) constructs the configured provider (`CAUSELOOP_OBJECT_PROVIDER`: `s3`, `azure`, or test-only `memory`) and does a cheap connectivity check — `head_bucket` for S3/MinIO, `get_container_properties` for Azure Blob — without reading or writing an object.
`secret_encryption_readiness()` (`services/api/secrets.py`) reports whether the configured `CAUSELOOP_SECRET_PROVIDER` (`fernet`, `aws-kms`, `azure-key-vault`) is actually usable: for `fernet` this means `SOURCE_CREDS_KEY` is set *and* the environment is not `production` (Fernet is local-only by policy — production must select KMS or Key Vault); for the cloud providers it means the corresponding key identifier env var is set.
`email_delivery_readiness()` (`services/api/auth/mailbox.py`) validates the email provider contract (`CAUSELOOP_EMAIL_PROVIDER` plus its required env vars) without sending anything — see [Workers and jobs](/architecture/workers-and-jobs) for how this same check gates whether `product-worker` will even claim `send_email` jobs.
If `REDIS_URL` is set, pings it directly. If not, readiness for this check is simply `not redis_required` — where `redis_required` defaults to `true` in production (`CAUSELOOP_ENVIRONMENT=production`) and `false` otherwise, overridable via `CAUSELOOP_REDIS_REQUIRED`.
All five checks run every time `/health/ready` is called — nothing is cached — so a load balancer or orchestrator polling this endpoint gets a live, current answer, not a stale one computed at boot.
## Authorization on mutating routes
Two independent dependencies gate a mutating request, and both are visible directly in each route's decorator rather than hidden in middleware:
* **Role/session dependency** — e.g. `_MUTATE = Depends(require_employee_role("onboarding_admin"))` in `onboarding_api.py`. A `support_readonly` employee can call every `GET` in that router but is rejected on every mutation; there is no separate read-only mode toggle, the role requirement is per-route.
* **`Depends(require_csrf)`** — `services/api/auth/csrf.py` implements double-submit-cookie CSRF protection: a random token is set as a *readable* (non-`httpOnly`) cookie at login, and the frontend proxy echoes it back as the `X-CSRF-Token` header on every mutating request. `require_csrf` is a no-op for `GET`/`HEAD`/`OPTIONS` and otherwise does a constant-time (`hmac.compare_digest`) comparison of cookie versus header, raising `403` on any mismatch or absence. No server-side token storage is needed — a cross-site attacker's form submission can make the browser send the cookie automatically, but same-origin JavaScript is required to *read* the cookie and set the matching header, which is exactly the property this defends.
You'll see both composed together on nearly every `POST`/`PUT`/`PATCH` in `onboarding_api.py`: `dependencies=[_MUTATE, Depends(require_csrf)]`. Login and `accept-invite` routes are the deliberate exception — there is no session yet to protect, so they carry no CSRF dependency.
`remediation_api.py` is tenant-facing, not employee-facing, so its role dependency is different: mutations depend on `Depends(require_permission("caps.write"))` (CAP create/update) or `Depends(require_permission("reviews.write"))` (review update) against the authenticated `TenantPrincipal`'s permissions, each paired with `Depends(require_csrf)` the same way. There is no `onboarding_admin` employee-role check anywhere in this router.
## Related
* [System overview](/architecture/overview)
* [Workers and jobs](/architecture/workers-and-jobs)
* [Observability](/architecture/observability)
* [API reference overview](/api-reference/overview)