> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Real failure modes across the product stack and the historical research modes, with the exact probe or log command that tells you which one you're looking at.

Most of these are covered by a mode's setup doctor first. Run `python3 scripts/check_local_setup.py --mode <mode>` (or `uv run python scripts/check_local_setup.py --mode <mode>` for the historical modes) and follow its first failing check before working through this page — the doctor is local, non-mutating, and never calls a remote service or reveals a secret value.

<AccordionGroup>
  <Accordion title="A stale process is already bound to :8000">
    `services/api/main.py` (the `basic`/`mvp1` research app) defaults to `127.0.0.1:8000`. If a previous `uv run uvicorn` or `scripts/run_local.py` invocation didn't shut down cleanly, the new one will either fail to bind or you'll be talking to the old process without realizing it. Probe the port directly:

    ```bash theme={null}
    curl -s http://127.0.0.1:8000/connectors
    ```

    `GET /connectors` is implemented in `services/api/clients_api.py` and needs no authentication, so it's a cheap way to confirm something Causeloop-shaped is actually listening. If it doesn't respond, find and stop the stale process before starting a new one; if it responds but the rest of the app behaves oddly, restart it anyway — an old `--reload` process can get into a broken state after enough file changes.

    This is a different port from the product stack's API — that one is `18000` (see [Local product stack](/onboarding/local-product-stack)), and its own liveness probe is `GET /health` or `GET /health/live` (both implemented in `services/api/product_app.py`), not the legacy `GET /health` on `services/api/main.py` this accordion is about.
  </Accordion>

  <Accordion title="404 from the MVP1 latest report on a fresh clone">
    `GET /mvp1/issue-intelligence/latest` (`services/api/main.py`, backed by `Mvp1IssueIntelligenceService.latest_strict_issue_intelligence`) only returns a report once a complete strict run has actually been persisted — it looks for a succeeded run whose artifact matches the exact strict analysis contract and whose provider status is `openai_llm_complete`. A fresh clone has never run that analysis, so `404` here is expected, not a setup failure. Run the strict analysis in **mvp1** mode ([Research modes](/onboarding/research-modes)) if you need a real report.
  </Accordion>

  <Accordion title="Postgres is not ready immediately after `docker compose up`">
    Both the relational research stack (`infra/docker-compose.yml postgres`) and the product stack's `postgres` service (`infra/docker-compose.product.yml`) start the container before Postgres has finished initializing. The product service has a `pg_isready` healthcheck (3-second interval, 30 retries) that gates the `migrate` service, so under product Compose this mostly self-heals — `migrate` simply waits. Under the relational mode there is no such gate in front of your own `alembic upgrade head`; if it fails immediately after `up -d postgres`, wait a few seconds and rerun:

    ```bash theme={null}
    uv run python scripts/check_local_setup.py --mode relational
    ```
  </Accordion>

  <Accordion title="The relational workbook check fails: '@oai/artifact-tool' not found">
    `scripts/check_local_setup.py`'s workbook-runtime check resolves `CAUSELOOP_NODE_PATH` (or whatever `node` is on `PATH` if unset) and looks for `@oai/artifact-tool` next to `workers/artifacts/node_modules` or next to that Node binary's own `node_modules`. If neither location has the package, it fails with "Node runtime does not expose `@oai/artifact-tool`." Set `CAUSELOOP_NODE_PATH` to the bundled Node executable discovered through the workspace-dependency tool — do not point it at an arbitrary Node install that doesn't carry the package, and don't install an unreviewed replacement package to work around it.
  </Accordion>

  <Accordion title="The frontend build stalls or OOMs on a low-memory Docker Desktop">
    On Docker Desktop instances limited to roughly 1 GB of memory, the Next.js compiler competes with the running data plane (Postgres, Redis, MinIO, both workers) for memory if you bring everything up in one shot. Build the images first, then start the stack:

    ```bash theme={null}
    docker compose -f infra/docker-compose.product.yml build
    docker compose -f infra/docker-compose.product.yml up -d
    ```
  </Accordion>

  <Accordion title="Invitation email never arrives in Mailpit">
    Invitations are asynchronous by design: the API atomically queues an encrypted outbox job, and the separate `product-worker` decrypts and sends it over SMTP to Mailpit — the job type is `send_email`, one of the three job types (`materialize_insights, send_email, train_tenant_model`) that `product-worker` polls for. If a message doesn't show up at `http://127.0.0.1:18025`:

    ```bash theme={null}
    docker compose -f infra/docker-compose.product.yml logs product-worker
    ```

    Look for a `job complete id=... type=send_email` line (or a `job ... status=...` failure) from the `causeloop.job_worker` logger. If you don't see the job attempted at all, confirm `product-worker` is actually running and has passed its own readiness heartbeat — see the next entry. Mailpit itself is an ephemeral capture sink with no application state; it is not where you debug why a job was never queued in the first place, only whether it was sent.
  </Accordion>

  <Accordion title="`/health/ready` stays `not_ready` on worker heartbeats">
    The product API's `GET /health/ready` (`services/api/product_app.py`) checks, among other things, that both `product-worker` and `provisioner-worker` have written a `RuntimeHeartbeat` row within the last `CAUSELOOP_WORKER_READY_SECONDS` seconds (default `45`) — `CAUSELOOP_REQUIRED_WORKERS` (default `product-worker,provisioner-worker`) controls which component names are required to heartbeat at all. Each heartbeating worker must also advertise the job types the hardcoded `REQUIRED_WORKER_JOB_TYPES` mapping expects of it (`product-worker`: `materialize_insights, send_email, train_tenant_model`; `provisioner-worker`: `provision_tenant`). If the `worker` check stays `false`:

    ```bash theme={null}
    docker compose -f infra/docker-compose.product.yml ps
    docker compose -f infra/docker-compose.product.yml logs product-worker provisioner-worker
    ```

    Confirm both containers are actually `Up` and not crash-looping — a worker that exits right after `migrate`/`minio-init` complete (its `depends_on` conditions) never gets the chance to heartbeat. Both workers must heartbeat before readiness succeeds; a database that's reachable but has zero recent heartbeats is a worker problem, not a database problem, even though the check is reported under `checks.worker` in the same JSON body as `checks.database`.
  </Accordion>

  <Accordion title="`provisioner-worker` never sends email or writes an upload — by design">
    The product stack runs two workers with deliberately disjoint job types (`infra/docker-compose.product.yml`, `CAUSELOOP_WORKER_JOB_TYPES`): `product-worker` handles `materialize_insights`, `send_email`, and `train_tenant_model`; `provisioner-worker` handles only `provision_tenant`. In local Compose the provisioner's environment simply omits Redis, email, and `SOURCE_CREDS_KEY` (credential-encryption) configuration — it isn't wired to touch any of those. Its object-store scope is a design/cloud-IAM property, not something the local compose file enforces: locally it's given the same MinIO root credentials as `product-worker`, so "read-only object access" describes the intended cloud permission boundary, not a restriction you can observe in this file. If you're debugging a job that seems stuck and it's an email or upload job, check `product-worker`'s logs, not the provisioner's; a job routed to the wrong worker will simply never be picked up.
  </Accordion>

  <Accordion title="Port collision with something else on your machine">
    Every product-stack port is loopback-only and overridable. If `13000`, `18000`, `55433`, `16379`, `19000`/`19001`, or `11025`/`18025` is already taken locally, override it before bringing the stack up:

    ```bash theme={null}
    CAUSELOOP_FRONTEND_PORT=13001 \
    CAUSELOOP_API_PORT=18001 \
    CAUSELOOP_POSTGRES_PORT=55434 \
    CAUSELOOP_REDIS_PORT=16380 \
    CAUSELOOP_MINIO_PORT=19002 \
    CAUSELOOP_MINIO_CONSOLE_PORT=19003 \
    CAUSELOOP_MAILPIT_SMTP_PORT=11026 \
    CAUSELOOP_MAILPIT_PORT=18026 \
    docker compose -f infra/docker-compose.product.yml up -d --build
    ```

    All of these default to the values in the port table on [Local product stack](/onboarding/local-product-stack); only override the ones that actually collide.
  </Accordion>

  <Accordion title="`npm run e2e` fails immediately with connection errors">
    `playwright.config.ts` starts (or reuses) the Next.js dev server on `http://localhost:3000` for you, but it does not start the backend. If the FastAPI product app isn't already listening on `http://127.0.0.1:18000`, every Playwright spec that hits a real page will fail on its first backend call. Start the product Compose stack first (`docker compose -f infra/docker-compose.product.yml up -d --build`), confirm `python3 scripts/smoke_product.py` passes, and only then run `npm run e2e`. If the authenticated specs (`auth.spec.ts`, `console.spec.ts`) are being skipped instead of failing, that's expected — they require `CAUSELOOP_E2E_WORKSPACE`, `CAUSELOOP_E2E_EMAIL`, and `CAUSELOOP_E2E_PASSWORD` for a tenant seeded ahead of time; see [Testing and verification](/onboarding/testing).
  </Accordion>

  <Accordion title="`docker compose down` looked like it deleted local product data">
    Plain `down` preserves the named `product_postgres`, `product_redis`, and `product_minio` volumes (`infra/docker-compose.product.yml`) — your tenants, uploads, and users survive a restart. Data is only actually gone if `-v` was passed (as CI's teardown step deliberately does) or the volumes were pruned some other way. Don't add `-v` to your own local `down` unless you mean to reset the stack from scratch.
  </Accordion>
</AccordionGroup>

## General approach

<Steps>
  <Step title="Identify the mode">
    Product stack, or one of `basic`/`mvp1`/`relational`? Commands, ports, and failure modes don't transfer between them — see [Research modes](/onboarding/research-modes).
  </Step>

  <Step title="Run that mode's doctor">
    ```bash theme={null}
    python3 scripts/check_local_setup.py --mode product   # or
    uv run python scripts/check_local_setup.py --mode basic|mvp1|relational
    ```

    Fix its first failing check before doing anything else.
  </Step>

  <Step title="Probe the specific service">
    Use the liveness/readiness endpoints (`/health`, `/health/live`, `/health/ready`, `/connectors`) or `docker compose ... logs <service>` to find which container is actually unhealthy, rather than guessing from symptoms in the UI.
  </Step>

  <Step title="Never work around a secret or auth check by exposing it">
    If a check fails because a key or password is missing, set it through your secret-aware local workflow. Never print, log, or commit `.env.local` or `infra/causeloop.local.env` to make a check pass.
  </Step>
</Steps>

## Related

* [Local product stack](/onboarding/local-product-stack)
* [Research modes](/onboarding/research-modes)
* [Testing and verification](/onboarding/testing)
* [Contributing](/onboarding/contributing)