Skip to main content
The backend repo holds two things at once: a small, deliberately-scoped product boundary (what’s actually deployed to AWS/Azure and fronted by the Next.js console), and a much larger legacy research codebase — the original single-tenant workbench and relational analysis pipeline the product was extracted from. Nothing in this repo has been deleted during that extraction, so most top-level directories contain both kinds of code side by side. This page tells you which is which before you start editing.

​
Directory map at a glance

The one authoritative signal for “is this in the product?” is services/api/product_app.py. It builds the deployed FastAPI app by explicitly allow-listing route operations out of the routers below — if an operation isn’t in one of its frozensets, it is not reachable in production, no matter what the router defines. See Architecture overview for how the product boundary is composed and Legacy Surfaces for the research API.

​
services/ — application code

  • product_app.py — the product boundary described above. Read this file first.
  • main.py (~249 KB, the largest file in services/) — the legacy research application: the full local/research backend with roughly 258 endpoints, the embedded file-backed client-onboarding path, and the /mvp1 strict-analysis surface. Not deployed.
  • clients_api.py, mvp1_contract.py — legacy-only routers used by main.py (per-client file-backed onboarding via ClientAssetStore, and the MVP1 analysis contract).
  • auth_api.py, onboarding_api.py, insights_api.py, members_api.py, metrics_api.py, remediation_api.py, sources_api.py — the product routers. Each defines a superset of operations; product_app.py selects only the allow-listed subset from each. onboarding_api.py is the largest (staff tenant-onboarding: provisioning, ingest, training, checkpoints, invitations, go-live).
  • auth/ — session and credential internals: tenant_auth.py/employee_auth.py (login flows), cookies.py, csrf.py, tokens.py, hashing.py (Argon2), rate_limit.py (Redis-backed), client_ip.py, pii_masking.py, mailbox.py (invitation/reset email readiness).
  • tenancy/ — multi-tenant plumbing: db.py (tenant storage-prefix helpers), deps.py (TenantRecord and FastAPI dependencies), provisioner.py (new-tenant provisioning job).
  • db/ — models.py (SQLAlchemy models, including RuntimeHeartbeat used by worker readiness) and session.py (control_uow/tenant_uow unit-of-work session factories).
  • reports/branded_export.py — the tenant remediation workbook export (GET /remediation/export.xlsx).
  • secrets.py — credential-encryption readiness (Fernet locally, KMS/Key Vault in cloud).
  • observability.py — request-metrics middleware installed into the product app (product_app.py); the legacy main.py does not install it.
pipeline.py and uploads.py implement the product’s tenant ingest path: uploads.py’s store_raw_upload streams a file straight to the configured object store (services.storage.object_store.get_object_store()) and records its metadata transactionally — never to local disk. connectors/ holds the pluggable source-connector registry (registry.py, base.py) and per-kind connectors (file_upload.py, database.py, s3.py, kafka.py), each with its own Pydantic config schema; registry.py’s get_connector_class raises a plain ValueError for an unknown kind. The 409 {"status": "not_enabled"} stub response documented for connector types that exist but aren’t wired up lives in the legacy clients_api.py path (driven by causegraph’s ConnectorNotEnabledError), not in this product ingest path.
jobs.py and job_worker.py implement the durable, PostgreSQL-backed job queue that both product workers (product-worker, provisioner-worker) run via scripts/run_pipeline_worker.py. materialize.py builds the tenant insight snapshot; training_job.py runs tenant model training. See Workers and jobs.
object_store.py is the one interface business code uses for immutable uploads: an S3ObjectStore (used for both real S3 and local MinIO, selected by CAUSELOOP_OBJECT_PROVIDER=s3), an AzureBlobObjectStore, and a test-only MemoryObjectStore that get_object_store() only ever returns when CAUSELOOP_ENVIRONMENT is not production — selecting CAUSELOOP_OBJECT_PROVIDER=memory in production raises instead. No code path writes uploads to local disk.

​
workers/ — background entrypoints

Two very different generations live here:
  • Product: nothing in workers/ itself is part of the deployed product — the product’s two long-running workers (product-worker, provisioner-worker) both run scripts/run_pipeline_worker.py, which imports services.pipeline.job_worker.run_job_worker(). workers/ is entirely legacy research tooling.
  • Legacy research: ingest/ holds the CLI entrypoints behind make ingest-helpdesk and make ingest-bpic13 (cli.py), plus one-off dataset/lifecycle scripts (acquire_public_datasets.py, process_imported_datasets.py, run_desired_lifecycle_workflow.py, continuous_embedding_loop.py, run_theme_intelligence.py, train_subset_sota_models.py, activate_bpic13.py). run_banking_relational_e2e.py, run_homecare_relational.py, run_homecare_relational_e2e.py, and migrate_homecare_checkpoint.py are the relational-mode pipeline runners described in Research modes. artifacts/build_mvp1_clustering_workbook.mjs is the Node script that builds the reviewed Excel workbook via @oai/artifact-tool (see Prerequisites).

​
migrations/ — Alembic

env.py plus versions/. Filenames encode which schema a migration targets:
  • c0xx_* — control-plane schema (staff/employee accounts, tenant registry, durable platform jobs — e.g. 20260722_c001_control_schema.py, 20260727_c003_durable_platform_jobs.py).
  • t00x_* — tenant-template schema, applied per-tenant (20260722_t001_tenant_template_schema.py through 20260727_t005_source_uploads.py: audit log, source checkpoints, source uploads).
  • Everything dated 202607{08,11,12,13,14}_* without a c/t prefix (mvp_postgres_embedding_store, mvp1_relational_runtime, the fishbone-related migrations) is legacy MVP1 relational-pipeline schema, not part of the product’s control/tenant split.
alembic.ini at the repo root points here. Run migrations with uv run alembic upgrade head (relational/legacy mode) or, for the product control schema, scripts/migrate_product.py (see below) — never both against the same database without checking which schema you mean.

​
infra/ — local infrastructure

  • docker-compose.product.yml — the canonical local stack: mailpit, postgres (pgvector image), redis, minio + minio-init, a one-off migrate service, a one-off idempotent local-admin bootstrap service, product-worker, provisioner-worker, api, and frontend (built from the sibling ../../frontend checkout). See Local product stack for the full walkthrough.
  • docker-compose.yml — the legacy research infrastructure stack: postgres (port 55432, distinct from product’s 55433), redis (6379), minio/minio-init, plus two profiles: ["research"]-gated services (openlineage-receiver, scheduler-health) that only start when that Compose profile is explicitly selected.
  • causeloop.local.env.example / causeloop.local.env (gitignored) — relational-mode environment; NEON.md — notes on using Neon-hosted Postgres for relational mode.
  • neo4j-schema.cypher, postgres-init.sql — legacy research schema bootstrap.
  • local_openlineage_receiver.py, local_scheduler_health.py — local stand-ins for the legacy research stack’s lineage/scheduler integrations, run by the research-profile Compose services above.

​
deploy/ — cloud infrastructure as code

  • aws/ — Terraform (main.tf, variables.tf, outputs.tf, versions.tf) deploying the same containers as docker-compose.product.yml onto ECS/Fargate, RDS PostgreSQL, ElastiCache-equivalent Redis, S3, and SES, with per-task IAM execution-role boundaries (the migration task, admin-bootstrap task, provisioner, and normal runtime each get different secret access). run-migration.sh / run-admin-bootstrap.sh are the one-off task runners.
  • azure/ — Bicep (main.bicep, main.bicepparam, acr-pull.bicep) deploying the Container Apps equivalent: PostgreSQL Flexible Server, Azure Managed Redis (EnterpriseCluster policy, classic protocol — Azure Cache for Redis is deliberately not used), private Blob Storage, Azure Communication Services Email, and Key Vault-backed secrets.
Both READMEs are ground truth for their respective cloud target — see AWS deployment and Azure deployment.

​
scripts/ — operational and developer scripts

1

Local setup and verification

check_local_setup.py (the doctor — see Prerequisites), smoke_product.py (post-deploy/post-Compose HTTP smoke check against /health/live, /health/ready, and /login), bootstrap_env.py (control-schema migration + app-role grant helpers used by migrate_product.py).
2

Product database and admin bootstrap

migrate_product.py runs control-plane migrations and grants the least-privilege app_rw role — this is the migrate Compose service’s command. create_employee.py creates a staff account (used by the local-admin Compose service to bootstrap admin@causeloop.local). migrate_all_tenants.py and offboard_tenant.py are tenant-lifecycle operational scripts.
3

Product runtime entrypoints

run_pipeline_worker.py runs the durable job worker (both product-worker and provisioner-worker use this same entrypoint). The two differ by more than job types: they also carry a distinct CAUSELOOP_WORKER_COMPONENT name (used for heartbeats) and a distinct credential scope — the provisioner runs with owner-level MIGRATIONS_DATABASE_URL credentials and no Redis/email environment at all, a deliberate security boundary docs/LOCAL_SETUP.md calls out. rehearse_product_backup.sh exercises the backup/restore path against the product stack.
4

Legacy research scripts

run_local.py (starts the legacy basic-mode API + workbench), kafka_produce_sample.py, load_test_pipeline.py — all support the research modes in Research modes, not the product.

​
apps/ — legacy research workbench

apps/web/ is a Vite + React 18 SPA (@causeloop/web) — the UI behind basic/mvp1/relational research modes. Its src/components/ (ProductControlPanel.tsx, DeveloperTaskRunner.tsx, BenchmarkPanel.tsx, WorkflowGraph.tsx, ArtifactRegistryPanel.tsx, and others) render the research pipeline’s operator/developer tooling against services/api/main.py. It is not related to the production Next.js console in the sibling frontend/ repo — see Frontend repo tour.

​
packages/causegraph/ — the shared intelligence engine

A uv workspace member ([tool.uv.workspace] members = ["packages/causegraph"] in the root pyproject.toml, resolved from [tool.uv.sources]). causegraph is a normal runtime dependency of the root project ([project.dependencies]), not a dev-only one, so any uv sync installs it editable — it isn’t gated behind --dev. This is not legacy-only: both the product routers (onboarding_api.py, insights_api.py) and the legacy services/api/main.py/clients_api.py import from it. Its src/causegraph/ tree holds the domain graph schema (graph_schema.py, models.py), retrieval/embedding/clustering providers (providers/: embeddings.py, clusterers.py, labelers.py, bm25.py, predictive.py, recommendations.py, and MVP1-specific providers like mvp1_fishbone.py, mvp1_issue_intelligence.py, mvp1_llm.py), storage adapters (storage/: mvp1_postgres.py, embeddings.py, operational.py, repositories.py), and the onboarding domain logic product routers actually call (onboarding/assets.py, onboarding/connectors.py, onboarding/ingestion.py, onboarding/insights.py, onboarding/training.py, onboarding/narrative.py). The optional promotion extra (causegraph[promotion], also mirrored as a root dependency-groups.promotion) adds boto3, mlflow, neo4j, and psycopg[binary] for the legacy promotion-handoff path (promotion_handoff.py).

​
tests/ — pytest suite

54 test_*.py files directly under tests/ (57 .py files total, counting conftest.py, db_support.py, and __init__.py), run with uv run pytest. Product-surface tests are named after the router or cross-cutting concern they cover: test_auth_flows.py, test_onboarding_api.py, test_onboarding_insights*.py, test_onboarding_training.py, test_sources_api.py, test_remediation_api.py, test_tenancy_isolation.py, test_product_app_contract.py (asserts the exact allow-listed operation set in product_app.py), test_cloud_runtime_boundaries.py, test_durable_jobs.py, test_email_delivery.py, test_metrics_api.py, test_pipeline_materialize.py, test_sqlalchemy_runtime_boundary.py, test_local_setup_tools.py. Everything prefixed test_mvp1_*, plus test_homecare_*, test_banking_relational_runner.py, test_theme_intelligence.py, test_loop_discovery.py, test_clusterers.py, test_narrative_adapters.py, test_process_discovery_providers.py, and test_repositories.py (~647 KB — the largest file under tests/) cover the legacy research pipeline and causegraph providers. conftest.py and db_support.py provide shared fixtures; fixtures/ holds test data files.

​
docs/ — prose documentation and legacy dataset artifacts

Start with LOCAL_SETUP.md (mirrored into this docs site — see Local product stack and Research modes), CLOUD_DEPLOYMENT.md/CLOUD_OPERATIONS.md, MVP_TARGET_ARCHITECTURE_AUDIT.md (the auditor’s record of why the product boundary is shaped the way it is), and PRODUCTION_PROMOTION_RUNBOOK.md. Everything prefixed MVP1_* documents the legacy relational pipeline’s architecture and contracts. docs/issue-set-2-records.json is the tracked, synthetic 250-record banking dataset the doctor validates and the relational/mvp1 modes analyze by default — the other issue-set-2-* files (.xlsx, .json, .md, several multi-megabyte) are generated outputs from past legacy pipeline runs, checked in as reference artifacts rather than regenerated on demand. docs/superpowers/ and the assorted root-level design-exploration Markdown files (dataset-aligned-graph-model.md, failure-informed-data-model-design.md, etc.) are working notes, not shipped documentation.

​
data/ — local and generated state

  • data/clients/ — per-client file-backed state (sources/, model/, cache/, manifest.json) written by the legacy main.py’s ClientAssetStore, rooted at CAUSELOOP_CLIENTS_ROOT (default data/clients). Gitignored.
  • data/tenant_uploads/ — local per-upload artifact directories. Gitignored.
  • data/artifacts/ — legacy research pipeline state. Five state/ JSON files are tracked in git (unlike the two directories above): state/manifest.json, state/runs.json, state/task_run_artifacts.json, state/artifact_manifests.json, state/product_control.json. state/dataset_snapshots.json and state/records.jsonl are locally generated and untracked. The directory’s other top-level child is embeddings/; feature_snapshots/ and bm25_indexes/ are subdirectories under state/, not siblings of it. Expect the tracked files to show as modified after running legacy pipeline commands locally.
  • data/raw/ — placeholder (.gitkeep only) for datasets acquired by workers/ingest/acquire_public_datasets.py.

​
Where to start

1

Read the boundary

services/api/product_app.py, then the router it selects from that matches the feature you’re touching.
2

Read the contract test

tests/test_product_app_contract.py — it will fail loudly if you add a route without updating the allowlist.
3

Read the shared engine, if relevant

If the router calls into causegraph.onboarding.*, read the matching module in packages/causegraph/src/causegraph/onboarding/.