Skip to main content
services/api/product_app.py is the executable target for the deployed backend. Its own module docstring states the reason it exists as a separate file from services/api/main.py: the historical app “also hosts the local research workbench, legacy file-backed client onboarding, and process-global /mvp1 routes. Importing that app therefore creates stores and exposes operations the customer/staff product does not need.” product_app.py builds a second, deliberately smaller FastAPI app from the same route modules, and lists every operation it publishes explicitly.

​
create_product_app(): the assembly

app = create_product_app() at module scope is the object uvicorn services.api.product_app:app (the repo’s Dockerfile CMD) actually serves. There is no additional route registration anywhere else in the deployed process.

​
The route-allowlist pattern

Every one of the seven include_router calls above passes the full router object from its source module (auth_router, onboarding_router, etc.) through _selected_router(), which filters it down to an explicit frozenset of (METHOD, path) tuples — AUTH_OPERATIONS, ONBOARDING_OPERATIONS, and so on. product_app.py also defines TENANT_OPERATIONS (the union of INSIGHT_OPERATIONS, MEMBER_OPERATIONS, SOURCE_OPERATIONS, and REMEDIATION_OPERATIONS) and MVP_ROUTE_OPERATIONS — the union of AUTH_OPERATIONS, ONBOARDING_OPERATIONS, TENANT_OPERATIONS, and OBSERVABILITY_OPERATIONS — as the single frozenset tests and contract checks treat as the full published surface. This is the load-bearing design decision in this file:
  • Source routers hold more routes than the product ever exposes. For example, onboarding_router also defines /{tenant_id}/rename, /{tenant_id}/materialize-now, /{tenant_id}/run-migrations, and /{tenant_id}/import-checkpoint — each one an operator/rehearsal escape hatch documented in its own TEMPORARY: code comment (e.g. run-migrations exists because this deployment has no direct external Postgres access from a local dev machine, so an already-provisioned tenant can be migrated to head over HTTPS instead) — and none of them appear in ONBOARDING_OPERATIONS, so _selected_router drops them. auth_router similarly defines /admin/sessions, /admin/password-reset/*, /password-reset/*, and /dev/mailbox (a local-only mailbox reader) that never reach the product surface.
  • _selected_router is defensive in two directions, both enforced with a RuntimeError at import time (i.e., app boot fails loudly, not silently at request time):
    • Missing operations: if an allowlisted (method, path) pair isn’t actually registered on the source router, boot fails with "{label} MVP operations are not registered". You cannot allowlist an endpoint that doesn’t exist yet.
    • Mixed methods on one path: if a path has some HTTP methods allowlisted and others not (e.g., a route registered for both GET and DELETE where only GET is approved), boot fails with "{label} route {path!r} mixes allowed and unapproved methods". A path is all-or-nothing per method — there is no way to partially publish a route.
Adding an endpoint to a router file (onboarding_api.py, sources_api.py, etc.) does not publish it. You must also add its (METHOD, path) tuple to the matching *_OPERATIONS frozenset in product_app.py, or _selected_router will silently exclude it — silently from the caller’s perspective, though the app would still boot fine since nothing requires that path. The reverse mistake — allowlisting a path before the router defines it — fails loudly at boot, which is the safer failure mode by design.

​
Published surface: 44 operations

The published surface is exactly 44 HTTP operations: 41 allowlisted business operations plus 3 always-on system endpoints (/health, /health/live, /health/ready) registered directly on product outside the allowlist mechanism, since they carry no tenant or employee authorization and must always answer. Each router’s own module docstring explains its authorization model and is the right place to read next:
  • auth_api.py — employee (Onboarding Portal) and tenant-user (Client Portal) session auth. Every mutating route depends on require_csrf; login/accept-invite are exempt because there is no session yet to protect.
  • onboarding_api.py — the staff-facing tenant lifecycle and checklist pipeline. Every route requires an authenticated employee; mutations additionally require the onboarding_admin control role (support_readonly employees can read but not act).
  • insights_api.py — tenant-facing insight reads backed only by materialized insight_collections; never computes on request.
  • members_api.py — tenant-facing member directory.
  • metrics_api.py — Prometheus-format observability endpoint (job durations, queue depth, LLM usage), gated behind get_current_employee.
  • remediation_api.py — the real, RLS-isolated, per-tenant CAP/review/audit-log system built on caps/reviews/audit_log tables, replacing the legacy process-global /mvp1/caps path. Every mutation writes a tenant audit_log row.
  • sources_api.py — tenant-facing source connector management. Credentials are write-only: encrypted on submission, never returned after creation (only a last-4-style hint).
See Tenancy and data model for how TenantPrincipal/EmployeePrincipal and RLS enforce these boundaries, and API reference overview for request/response contracts.

​
Request observability

install_request_observability() (services/api/observability.py) is the only middleware the product app installs. It is deliberately small and provider-neutral:
  • Accepts an inbound x-request-id header only if it matches ^[A-Za-z0-9._:-]{1,128}$; otherwise generates a fresh uuid4().hex. This prevents a client from injecting arbitrary content into structured logs via the header.
  • Stores the resolved ID on request.state.request_id, echoes it back as the x-request-id response header, and emits exactly one structured JSON log line per request (event: "http_request", method, path, status code, duration in milliseconds) to the causeloop.http logger — in the finally block, so it fires even when the handler raises.
There is no request/response body logging, no tracing SDK, and no external APM call in this module — it is the floor every deployment gets for free, and the natural place other export/monitoring integrations should hook if you add them later. See Observability for what consumes these logs in each environment.

​
The /health/ready model

The product app exposes three health routes, all defined directly in create_product_app():
  • GET /health and GET /health/live — trivial liveness: always return {"status": "ok"}. Suitable for a container orchestrator’s liveness probe; they do not touch the database.
  • GET /health/ready — the real readiness gate. It runs five independent checks and returns HTTP 503 with {"status": "not_ready", ...} the moment any one of them fails; 200 with {"status": "ready", ...} only when all pass.
1

database + worker heartbeat

Runs SELECT 1 against the control-plane connection (fail-fast if Postgres is unreachable), then checks control.runtime_heartbeats for a row per CAUSELOOP_REQUIRED_WORKERS (default product-worker,provisioner-worker) heartbeated within the last CAUSELOOP_WORKER_READY_SECONDS (default 45s). This alone would only prove a worker process is alive — _worker_capabilities_ready() goes further and cross-checks each heartbeat’s advertised detail.job_types against REQUIRED_WORKER_JOB_TYPES (product-worker must advertise all of materialize_insights, send_email, train_tenant_model; provisioner-worker must advertise provision_tenant). A product-worker process that is up but was started with an incomplete CAUSELOOP_WORKER_JOB_TYPES env var fails readiness even though its heartbeat row exists — readiness is capability-gated, not just liveness-gated.
2

object storage

object_store_readiness() (services/storage/object_store.py) constructs the configured provider (CAUSELOOP_OBJECT_PROVIDER: s3, azure, or test-only memory) and does a cheap connectivity check — head_bucket for S3/MinIO, get_container_properties for Azure Blob — without reading or writing an object.
3

credential encryption

secret_encryption_readiness() (services/api/secrets.py) reports whether the configured CAUSELOOP_SECRET_PROVIDER (fernet, aws-kms, azure-key-vault) is actually usable: for fernet this means SOURCE_CREDS_KEY is set and the environment is not production (Fernet is local-only by policy — production must select KMS or Key Vault); for the cloud providers it means the corresponding key identifier env var is set.
4

email delivery

email_delivery_readiness() (services/api/auth/mailbox.py) validates the email provider contract (CAUSELOOP_EMAIL_PROVIDER plus its required env vars) without sending anything — see Workers and jobs for how this same check gates whether product-worker will even claim send_email jobs.
5

Redis

If REDIS_URL is set, pings it directly. If not, readiness for this check is simply not redis_required — where redis_required defaults to true in production (CAUSELOOP_ENVIRONMENT=production) and false otherwise, overridable via CAUSELOOP_REDIS_REQUIRED.
All five checks run every time /health/ready is called — nothing is cached — so a load balancer or orchestrator polling this endpoint gets a live, current answer, not a stale one computed at boot.

​
Authorization on mutating routes

Two independent dependencies gate a mutating request, and both are visible directly in each route’s decorator rather than hidden in middleware:
  • Role/session dependency — e.g. _MUTATE = Depends(require_employee_role("onboarding_admin")) in onboarding_api.py. A support_readonly employee can call every GET in that router but is rejected on every mutation; there is no separate read-only mode toggle, the role requirement is per-route.
  • Depends(require_csrf) — services/api/auth/csrf.py implements double-submit-cookie CSRF protection: a random token is set as a readable (non-httpOnly) cookie at login, and the frontend proxy echoes it back as the X-CSRF-Token header on every mutating request. require_csrf is a no-op for GET/HEAD/OPTIONS and otherwise does a constant-time (hmac.compare_digest) comparison of cookie versus header, raising 403 on any mismatch or absence. No server-side token storage is needed — a cross-site attacker’s form submission can make the browser send the cookie automatically, but same-origin JavaScript is required to read the cookie and set the matching header, which is exactly the property this defends.
You’ll see both composed together on nearly every POST/PUT/PATCH in onboarding_api.py: dependencies=[_MUTATE, Depends(require_csrf)]. Login and accept-invite routes are the deliberate exception — there is no session yet to protect, so they carry no CSRF dependency. remediation_api.py is tenant-facing, not employee-facing, so its role dependency is different: mutations depend on Depends(require_permission("caps.write")) (CAP create/update) or Depends(require_permission("reviews.write")) (review update) against the authenticated TenantPrincipal’s permissions, each paired with Depends(require_csrf) the same way. There is no onboarding_admin employee-role check anywhere in this router.