> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Backend repo tour

> What lives in every top-level directory of the backend repo, and which parts are the deployed product versus legacy research code.

The backend repo holds two things at once: a small, deliberately-scoped **product boundary** (what's actually deployed to AWS/Azure and fronted by the Next.js console), and a much larger **legacy research codebase** — the original single-tenant workbench and relational analysis pipeline the product was extracted from. Nothing in this repo has been deleted during that extraction, so most top-level directories contain both kinds of code side by side. This page tells you which is which before you start editing.

## Directory map at a glance

| Directory              | Contains                                                                         | Product or legacy?                                  |
| ---------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------- |
| `services/`            | HTTP API, ingest pipeline, durable job execution, object storage                 | Both — split at the file level, see below           |
| `workers/`             | Background CLI/pipeline entrypoints                                              | Legacy research only                                |
| `migrations/`          | Alembic migrations (`c0xx`/`t0xx` = product control/tenant; unprefixed = legacy) | Both                                                |
| `infra/`               | Local Docker Compose stacks                                                      | Both — two separate Compose files                   |
| `deploy/`              | AWS Terraform, Azure Bicep                                                       | Product only                                        |
| `scripts/`             | Setup, migration, worker, and legacy dev scripts                                 | Both                                                |
| `apps/`                | Legacy Vite research workbench                                                   | Legacy only                                         |
| `packages/causegraph/` | Shared domain-graph/retrieval/embedding engine                                   | Both — imported by product and legacy routers alike |
| `tests/`               | pytest suite (59 files)                                                          | Both — named per surface                            |
| `docs/`                | Prose docs plus legacy dataset artifacts                                         | Both                                                |
| `data/`                | Local/generated runtime state                                                    | Legacy-heavy; some tracked, some gitignored         |

<Note>
  The one authoritative signal for "is this in the product?" is `services/api/product_app.py`. It builds the deployed FastAPI app by explicitly allow-listing route operations out of the routers below — if an operation isn't in one of its frozensets, it is not reachable in production, no matter what the router defines. See [Architecture overview](/architecture/overview) for how the product boundary is composed and [Legacy Surfaces](/api-reference/research-api) for the research API.
</Note>

## `services/` — application code

<AccordionGroup>
  <Accordion title="services/api/ — HTTP layer">
    * `product_app.py` — the product boundary described above. Read this file first.
    * `main.py` (\~249 KB, the largest file in `services/`) — the legacy research application: the full local/research backend with roughly 258 endpoints, the embedded file-backed client-onboarding path, and the `/mvp1` strict-analysis surface. **Not deployed.**
    * `clients_api.py`, `mvp1_contract.py` — legacy-only routers used by `main.py` (per-client file-backed onboarding via `ClientAssetStore`, and the MVP1 analysis contract).
    * `auth_api.py`, `onboarding_api.py`, `insights_api.py`, `members_api.py`, `metrics_api.py`, `remediation_api.py`, `sources_api.py` — the product routers. Each defines a superset of operations; `product_app.py` selects only the allow-listed subset from each. `onboarding_api.py` is the largest (staff tenant-onboarding: provisioning, ingest, training, checkpoints, invitations, go-live).
    * `auth/` — session and credential internals: `tenant_auth.py`/`employee_auth.py` (login flows), `cookies.py`, `csrf.py`, `tokens.py`, `hashing.py` (Argon2), `rate_limit.py` (Redis-backed), `client_ip.py`, `pii_masking.py`, `mailbox.py` (invitation/reset email readiness).
    * `tenancy/` — multi-tenant plumbing: `db.py` (tenant storage-prefix helpers), `deps.py` (`TenantRecord` and FastAPI dependencies), `provisioner.py` (new-tenant provisioning job).
    * `db/` — `models.py` (SQLAlchemy models, including `RuntimeHeartbeat` used by worker readiness) and `session.py` (`control_uow`/`tenant_uow` unit-of-work session factories).
    * `reports/branded_export.py` — the tenant remediation workbook export (`GET /remediation/export.xlsx`).
    * `secrets.py` — credential-encryption readiness (Fernet locally, KMS/Key Vault in cloud).
    * `observability.py` — request-metrics middleware installed into the product app (`product_app.py`); the legacy `main.py` does not install it.
  </Accordion>

  <Accordion title="services/ingest/ — upload and connector pipeline">
    `pipeline.py` and `uploads.py` implement the product's tenant ingest path: `uploads.py`'s `store_raw_upload` streams a file straight to the configured object store (`services.storage.object_store.get_object_store()`) and records its metadata transactionally — never to local disk. `connectors/` holds the pluggable source-connector registry (`registry.py`, `base.py`) and per-kind connectors (`file_upload.py`, `database.py`, `s3.py`, `kafka.py`), each with its own Pydantic config schema; `registry.py`'s `get_connector_class` raises a plain `ValueError` for an unknown kind. The `409 {"status": "not_enabled"}` stub response documented for connector types that exist but aren't wired up lives in the **legacy** `clients_api.py` path (driven by `causegraph`'s `ConnectorNotEnabledError`), not in this product ingest path.
  </Accordion>

  <Accordion title="services/pipeline/ — durable job execution">
    `jobs.py` and `job_worker.py` implement the durable, PostgreSQL-backed job queue that both product workers (`product-worker`, `provisioner-worker`) run via `scripts/run_pipeline_worker.py`. `materialize.py` builds the tenant insight snapshot; `training_job.py` runs tenant model training. See [Workers and jobs](/architecture/workers-and-jobs).
  </Accordion>

  <Accordion title="services/storage/ — object storage boundary">
    `object_store.py` is the one interface business code uses for immutable uploads: an `S3ObjectStore` (used for both real S3 and local MinIO, selected by `CAUSELOOP_OBJECT_PROVIDER=s3`), an `AzureBlobObjectStore`, and a test-only `MemoryObjectStore` that `get_object_store()` only ever returns when `CAUSELOOP_ENVIRONMENT` is not `production` — selecting `CAUSELOOP_OBJECT_PROVIDER=memory` in production raises instead. No code path writes uploads to local disk.
  </Accordion>
</AccordionGroup>

## `workers/` — background entrypoints

Two very different generations live here:

* **Product**: nothing in `workers/` itself is part of the deployed product — the product's two long-running workers (`product-worker`, `provisioner-worker`) both run `scripts/run_pipeline_worker.py`, which imports `services.pipeline.job_worker.run_job_worker()`. `workers/` is entirely legacy research tooling.
* **Legacy research**: `ingest/` holds the CLI entrypoints behind `make ingest-helpdesk` and `make ingest-bpic13` (`cli.py`), plus one-off dataset/lifecycle scripts (`acquire_public_datasets.py`, `process_imported_datasets.py`, `run_desired_lifecycle_workflow.py`, `continuous_embedding_loop.py`, `run_theme_intelligence.py`, `train_subset_sota_models.py`, `activate_bpic13.py`). `run_banking_relational_e2e.py`, `run_homecare_relational.py`, `run_homecare_relational_e2e.py`, and `migrate_homecare_checkpoint.py` are the relational-mode pipeline runners described in [Research modes](/onboarding/research-modes). `artifacts/build_mvp1_clustering_workbook.mjs` is the Node script that builds the reviewed Excel workbook via `@oai/artifact-tool` (see [Prerequisites](/onboarding/prerequisites)).

## `migrations/` — Alembic

`env.py` plus `versions/`. Filenames encode which schema a migration targets:

* `c0xx_*` — control-plane schema (staff/employee accounts, tenant registry, durable platform jobs — e.g. `20260722_c001_control_schema.py`, `20260727_c003_durable_platform_jobs.py`).
* `t00x_*` — tenant-template schema, applied per-tenant (`20260722_t001_tenant_template_schema.py` through `20260727_t005_source_uploads.py`: audit log, source checkpoints, source uploads).
* Everything dated `202607{08,11,12,13,14}_*` without a `c`/`t` prefix (`mvp_postgres_embedding_store`, `mvp1_relational_runtime`, the fishbone-related migrations) is legacy MVP1 relational-pipeline schema, not part of the product's control/tenant split.

`alembic.ini` at the repo root points here. Run migrations with `uv run alembic upgrade head` (relational/legacy mode) or, for the product control schema, `scripts/migrate_product.py` (see below) — never both against the same database without checking which schema you mean.

## `infra/` — local infrastructure

* `docker-compose.product.yml` — the canonical local stack: `mailpit`, `postgres` (pgvector image), `redis`, `minio` + `minio-init`, a one-off `migrate` service, a one-off idempotent `local-admin` bootstrap service, `product-worker`, `provisioner-worker`, `api`, and `frontend` (built from the sibling `../../frontend` checkout). See [Local product stack](/onboarding/local-product-stack) for the full walkthrough.
* `docker-compose.yml` — the legacy research infrastructure stack: `postgres` (port `55432`, distinct from product's `55433`), `redis` (`6379`), `minio`/`minio-init`, plus two `profiles: ["research"]`-gated services (`openlineage-receiver`, `scheduler-health`) that only start when that Compose profile is explicitly selected.
* `causeloop.local.env.example` / `causeloop.local.env` (gitignored) — relational-mode environment; `NEON.md` — notes on using Neon-hosted Postgres for relational mode.
* `neo4j-schema.cypher`, `postgres-init.sql` — legacy research schema bootstrap.
* `local_openlineage_receiver.py`, `local_scheduler_health.py` — local stand-ins for the legacy research stack's lineage/scheduler integrations, run by the `research`-profile Compose services above.

## `deploy/` — cloud infrastructure as code

* `aws/` — Terraform (`main.tf`, `variables.tf`, `outputs.tf`, `versions.tf`) deploying the same containers as `docker-compose.product.yml` onto ECS/Fargate, RDS PostgreSQL, ElastiCache-equivalent Redis, S3, and SES, with per-task IAM execution-role boundaries (the migration task, admin-bootstrap task, provisioner, and normal runtime each get different secret access). `run-migration.sh` / `run-admin-bootstrap.sh` are the one-off task runners.
* `azure/` — Bicep (`main.bicep`, `main.bicepparam`, `acr-pull.bicep`) deploying the Container Apps equivalent: PostgreSQL Flexible Server, Azure Managed Redis (`EnterpriseCluster` policy, classic protocol — Azure Cache for Redis is deliberately not used), private Blob Storage, Azure Communication Services Email, and Key Vault-backed secrets.

Both READMEs are ground truth for their respective cloud target — see [AWS deployment](/deployment/aws) and [Azure deployment](/deployment/azure).

## `scripts/` — operational and developer scripts

<Steps>
  <Step title="Local setup and verification">
    `check_local_setup.py` (the doctor — see [Prerequisites](/onboarding/prerequisites)), `smoke_product.py` (post-deploy/post-Compose HTTP smoke check against `/health/live`, `/health/ready`, and `/login`), `bootstrap_env.py` (control-schema migration + app-role grant helpers used by `migrate_product.py`).
  </Step>

  <Step title="Product database and admin bootstrap">
    `migrate_product.py` runs control-plane migrations and grants the least-privilege `app_rw` role — this is the `migrate` Compose service's command. `create_employee.py` creates a staff account (used by the `local-admin` Compose service to bootstrap `admin@causeloop.local`). `migrate_all_tenants.py` and `offboard_tenant.py` are tenant-lifecycle operational scripts.
  </Step>

  <Step title="Product runtime entrypoints">
    `run_pipeline_worker.py` runs the durable job worker (both `product-worker` and `provisioner-worker` use this same entrypoint). The two differ by more than job types: they also carry a distinct `CAUSELOOP_WORKER_COMPONENT` name (used for heartbeats) and a distinct credential scope — the provisioner runs with owner-level `MIGRATIONS_DATABASE_URL` credentials and no Redis/email environment at all, a deliberate security boundary `docs/LOCAL_SETUP.md` calls out. `rehearse_product_backup.sh` exercises the backup/restore path against the product stack.
  </Step>

  <Step title="Legacy research scripts">
    `run_local.py` (starts the legacy `basic`-mode API + workbench), `kafka_produce_sample.py`, `load_test_pipeline.py` — all support the research modes in [Research modes](/onboarding/research-modes), not the product.
  </Step>
</Steps>

## `apps/` — legacy research workbench

`apps/web/` is a Vite + React 18 SPA (`@causeloop/web`) — the UI behind `basic`/`mvp1`/`relational` research modes. Its `src/components/` (`ProductControlPanel.tsx`, `DeveloperTaskRunner.tsx`, `BenchmarkPanel.tsx`, `WorkflowGraph.tsx`, `ArtifactRegistryPanel.tsx`, and others) render the research pipeline's operator/developer tooling against `services/api/main.py`. It is not related to the production Next.js console in the sibling `frontend/` repo — see [Frontend repo tour](/onboarding/frontend-repo-tour).

## `packages/causegraph/` — the shared intelligence engine

A `uv` workspace member (`[tool.uv.workspace] members = ["packages/causegraph"]` in the root `pyproject.toml`, resolved from `[tool.uv.sources]`). `causegraph` is a normal runtime dependency of the root project (`[project.dependencies]`), not a dev-only one, so any `uv sync` installs it editable — it isn't gated behind `--dev`. This is **not** legacy-only: both the product routers (`onboarding_api.py`, `insights_api.py`) and the legacy `services/api/main.py`/`clients_api.py` import from it. Its `src/causegraph/` tree holds the domain graph schema (`graph_schema.py`, `models.py`), retrieval/embedding/clustering providers (`providers/`: `embeddings.py`, `clusterers.py`, `labelers.py`, `bm25.py`, `predictive.py`, `recommendations.py`, and MVP1-specific providers like `mvp1_fishbone.py`, `mvp1_issue_intelligence.py`, `mvp1_llm.py`), storage adapters (`storage/`: `mvp1_postgres.py`, `embeddings.py`, `operational.py`, `repositories.py`), and the onboarding domain logic product routers actually call (`onboarding/assets.py`, `onboarding/connectors.py`, `onboarding/ingestion.py`, `onboarding/insights.py`, `onboarding/training.py`, `onboarding/narrative.py`). The optional `promotion` extra (`causegraph[promotion]`, also mirrored as a root `dependency-groups.promotion`) adds `boto3`, `mlflow`, `neo4j`, and `psycopg[binary]` for the legacy promotion-handoff path (`promotion_handoff.py`).

## `tests/` — pytest suite

54 `test_*.py` files directly under `tests/` (57 `.py` files total, counting `conftest.py`, `db_support.py`, and `__init__.py`), run with `uv run pytest`. Product-surface tests are named after the router or cross-cutting concern they cover: `test_auth_flows.py`, `test_onboarding_api.py`, `test_onboarding_insights*.py`, `test_onboarding_training.py`, `test_sources_api.py`, `test_remediation_api.py`, `test_tenancy_isolation.py`, `test_product_app_contract.py` (asserts the exact allow-listed operation set in `product_app.py`), `test_cloud_runtime_boundaries.py`, `test_durable_jobs.py`, `test_email_delivery.py`, `test_metrics_api.py`, `test_pipeline_materialize.py`, `test_sqlalchemy_runtime_boundary.py`, `test_local_setup_tools.py`. Everything prefixed `test_mvp1_*`, plus `test_homecare_*`, `test_banking_relational_runner.py`, `test_theme_intelligence.py`, `test_loop_discovery.py`, `test_clusterers.py`, `test_narrative_adapters.py`, `test_process_discovery_providers.py`, and `test_repositories.py` (\~647 KB — the largest file under `tests/`) cover the legacy research pipeline and `causegraph` providers. `conftest.py` and `db_support.py` provide shared fixtures; `fixtures/` holds test data files.

## `docs/` — prose documentation and legacy dataset artifacts

Start with `LOCAL_SETUP.md` (mirrored into this docs site — see [Local product stack](/onboarding/local-product-stack) and [Research modes](/onboarding/research-modes)), `CLOUD_DEPLOYMENT.md`/`CLOUD_OPERATIONS.md`, `MVP_TARGET_ARCHITECTURE_AUDIT.md` (the auditor's record of why the product boundary is shaped the way it is), and `PRODUCTION_PROMOTION_RUNBOOK.md`. Everything prefixed `MVP1_*` documents the legacy relational pipeline's architecture and contracts. `docs/issue-set-2-records.json` is the tracked, synthetic 250-record banking dataset the doctor validates and the relational/mvp1 modes analyze by default — the other `issue-set-2-*` files (`.xlsx`, `.json`, `.md`, several multi-megabyte) are generated outputs from past legacy pipeline runs, checked in as reference artifacts rather than regenerated on demand. `docs/superpowers/` and the assorted root-level design-exploration Markdown files (`dataset-aligned-graph-model.md`, `failure-informed-data-model-design.md`, etc.) are working notes, not shipped documentation.

## `data/` — local and generated state

* `data/clients/` — per-client file-backed state (`sources/`, `model/`, `cache/`, `manifest.json`) written by the legacy `main.py`'s `ClientAssetStore`, rooted at `CAUSELOOP_CLIENTS_ROOT` (default `data/clients`). Gitignored.
* `data/tenant_uploads/` — local per-upload artifact directories. Gitignored.
* `data/artifacts/` — legacy research pipeline state. Five `state/` JSON files are **tracked in git** (unlike the two directories above): `state/manifest.json`, `state/runs.json`, `state/task_run_artifacts.json`, `state/artifact_manifests.json`, `state/product_control.json`. `state/dataset_snapshots.json` and `state/records.jsonl` are locally generated and untracked. The directory's other top-level child is `embeddings/`; `feature_snapshots/` and `bm25_indexes/` are subdirectories under `state/`, not siblings of it. Expect the tracked files to show as modified after running legacy pipeline commands locally.
* `data/raw/` — placeholder (`.gitkeep` only) for datasets acquired by `workers/ingest/acquire_public_datasets.py`.

## Where to start

<Steps>
  <Step title="Read the boundary">
    `services/api/product_app.py`, then the router it selects from that matches the feature you're touching.
  </Step>

  <Step title="Read the contract test">
    `tests/test_product_app_contract.py` — it will fail loudly if you add a route without updating the allowlist.
  </Step>

  <Step title="Read the shared engine, if relevant">
    If the router calls into `causegraph.onboarding.*`, read the matching module in `packages/causegraph/src/causegraph/onboarding/`.
  </Step>
</Steps>

## Related

* [Prerequisites](/onboarding/prerequisites)
* [Frontend repo tour](/onboarding/frontend-repo-tour)
* [Architecture overview](/architecture/overview)
* [Research modes](/onboarding/research-modes)