> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Insights console

> The tenant console's read surface: the /insights/snapshot contract, how materialize_insights populates it, and which views render which collections.

The tenant console (`/(console)` routes in the frontend repo) never computes an insight on request. Every report a tenant user sees — loops, issues, severity, fishbone causes, LLM narratives — is precomputed by a background job into a Postgres cache table (`insight_collections`, tenant-schema) and the console only ever reads that cache. This page covers the read contract (`GET /insights/snapshot`), the job that fills it (`materialize_insights`), and how the four report-backed views (Home, Loops, Issues, Remediation) consume the result.

## The ten collections

`packages/causegraph/src/causegraph/onboarding/insights.py` defines two groups, and `services/api/insights_api.py` imports both:

```python theme={null}
MODEL_COLLECTIONS = ("summary", "themes", "issues", "severity", "lens")
LLM_COLLECTIONS = ("fishbone", "theme_summaries", "narratives", "recommendations", "cause_remediation")
```

**Model collections** are pure clustering-model output — always computable once a tenant has an active `model_checkpoints` row and at least one committed dataset, no LLM required:

| Collection | Payload shape                                                                                                                 |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `summary`  | `total_issues`, `cluster_count`, `noise_pct`, `silhouette`, `window.{first,last}`, `top_themes` (top 5 by size)               |
| `themes`   | `themes[]`: `theme_id`, `name`, `top_terms[]`, `size`, `share`                                                                |
| `issues`   | `issues[]`: `issue_id`, `text`, `created_at`, `theme_id`, `theme_name`                                                        |
| `severity` | `themes[]`: `theme_id`, `name`, `severity` (`0.6·share + 0.4·recency`), `level` (`high`/`medium`/`low`), `share`, `recency`   |
| `lens`     | `months[]` and `series[]` (`theme_id`, `name`, `counts[]` aligned to `months`) — a monthly issue-volume time series per theme |

**LLM collections** are generated per theme by the tenant's configured LLM provider (`llm_settings`) and are skipped, not failed, when no provider is configured:

| Collection          | Payload shape                                                                                                                                                                                                                             |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `fishbone`          | `themes[]`: `theme_id`, `name`, `analysis` (raw LLM JSON text), `categories[]` (`category`, `causes[]`) — a People/Process/Technology/External root-cause breakdown per theme                                                             |
| `theme_summaries`   | `themes[]`: `theme_id`, `name`, `summary` (2-3 sentence executive summary)                                                                                                                                                                |
| `recommendations`   | `themes[]`: `theme_id`, `name`, `recommendation` (single highest-impact fix + rationale)                                                                                                                                                  |
| `cause_remediation` | `causes[]`: one row per root-cause group with `bone`, `subcategory`, `cause`, `rationale`, `containment`, `corrective_action`, `preventive_action`, `owner_role`, `verification_evidence`, `success_metric`, `theme_id`, `cause_group_id` |
| `narratives`        | `narrative` — a one-paragraph, executive-audience portfolio summary across every theme                                                                                                                                                    |

<Note>
  `cause_remediation` is computed and stored like every other collection, but `insights_api.py` deliberately excludes it from `GET /insights/snapshot`'s response (`_EXPORT_ONLY_COLLECTIONS`) — it feeds the [branded XLSX export](/features/remediation) only. The console still reads its content indirectly: the export uses `load_visible_snapshot()` to include it, and the frontend's own `client-adapter.ts` on the browser side does read a `cause_remediation` key from the raw snapshot response for CAPA rationale/evidence — but the deployed contract (below) only ever hands the browser 9 collections, so that field is always absent on the wire today.
</Note>

## `GET /insights/snapshot`

Defined in `services/api/insights_api.py`, requiring an authenticated tenant session. This is the only insights endpoint in the deployed product contract — `services/api/product_app.py`'s `INSIGHT_OPERATIONS` allowlist admits exactly `("GET", "/insights/snapshot")`. The router also implements `GET /insights/{collection}` (single-collection read, 202 while generating) and `GET /insights` (per-collection ready/not-ready status used by the lock-screen decision), but neither is in the current allowlist and neither is called by the shipped console — the frontend's `fetchClientInsights()` (`src/lib/api/clientInsights.ts`) only ever calls `/insights/snapshot`.

```json theme={null}
{
  "dataset_version": 3,
  "model_version": 2,
  "collections": {
    "summary": { "status": "ready", "payload": { "...": "..." } },
    "themes": { "status": "ready", "payload": { "...": "..." } },
    "issues": { "status": "ready", "payload": { "...": "..." } },
    "severity": { "status": "ready", "payload": { "...": "..." } },
    "lens": { "status": "ready", "payload": { "...": "..." } },
    "fishbone": { "status": "generating" },
    "theme_summaries": { "status": "generating" },
    "narratives": { "status": "disabled" },
    "recommendations": { "status": "disabled" }
  },
  "model_ready": true,
  "all_ready": false,
  "share_onboarding_outputs": true
}
```

`status` per collection is one of:

* **`ready`** — a row exists in `insight_collections` for the chosen `(dataset_version, model_version)` pair; `payload` is populated.
* **`generating`** — no row exists yet for this collection at the chosen version pair. This covers any of the 5 `MODEL_COLLECTIONS` before the first `materialize_insights` run lands (a fresh tenant's `summary` reports `generating`, not absent), and any `LLM_COLLECTIONS` entry whose provider is configured but hasn't produced that collection yet (or is mid-run).
* **`disabled`** — an LLM collection with no `llm_settings` row, or an empty/`disabled` provider, for this tenant. `payload` is always `null` here; the console renders a fixed placeholder string instead ("Narrative enrichment disabled.", "No generated recommendation.") rather than an empty state.

`model_ready` is `true` once all 5 `MODEL_COLLECTIONS` have a row; `all_ready` is `true` once every requested collection is anything other than `generating` (i.e. `ready` or `disabled` both count as settled).

<Warning>
  The frontend's `useConsoleData()` hook (`src/lib/data/useConsoleData.ts`) polls every 4 seconds via `refetchInterval` while any collection in the raw `ClientInsightsSnapshot` reports `generating` — it does not consult `all_ready` at all. `clientInsights.ts`'s `fetchClientInsights()` maps every one of the 9 collections missing from the response to `{ status: "generating" }`, and `cause_remediation` (this page's own Note above: never sent on the wire) is always missing. In practice this means the 4-second poll never turns off, since `cause_remediation` reports `generating` forever on the client. This reads as a frontend bug rather than intended behavior.
</Warning>

### One coherent snapshot per read

`load_visible_snapshot()` — the function both `/insights/snapshot` and the [remediation export](/features/remediation) share — first picks a single `(dataset_version, model_version)` pair (the newest one with at least one `client_visible` row) and only then reads every collection for that exact pair. This is deliberate: an older per-collection "latest" read could combine a `themes` collection from one materialize run with a `fishbone` collection from a different, later run — silently showing loops that don't match the fishbone under them. Picking the version pair up front makes that impossible; a page load is always looking at output from one pipeline run.

### `client_visible` and `share_onboarding_outputs`

Every `insight_collections` row carries a `client_visible` boolean and a `source` (`"ingest"`, `"onboarding"`, or `"manual"`). The snapshot query filters to `client_visible = true`, and additionally excludes `source = "onboarding"` rows unless the tenant's `share_onboarding_outputs` flag is on:

* Rows from a **real post-go-live ingest** (`source = "ingest"`) are always `client_visible` — a live tenant always sees its own new data.
* Rows produced **during onboarding** (`source = "onboarding"`) are gated by `share_onboarding_outputs`, a per-tenant setting the staff onboarding flow controls (see [Client onboarding pipeline](/features/client-onboarding-pipeline)) — some tenants are deliberately onboarded with training output hidden until formal go-live.

## The `materialize_insights` job

`services/pipeline/materialize.py`'s `materialize_insights(tenant, dataset_id, dataset_version, source, client_visible)` is the only code path that ever writes `insight_collections`. It:

1. Loads the tenant's active `model_checkpoints` row (raises `NoActiveCheckpointError` if none exists).
2. Loads every `DatasetRecord` row for the given `dataset_id`.
3. Computes and upserts all 5 `MODEL_COLLECTIONS` unconditionally, via `compute_model_collection_from_data()` (the same storage-agnostic function the file-based research workbench uses).
4. Loads the tenant's `llm_settings`. If none is configured (or the model field is empty), every `LLM_COLLECTIONS` entry is recorded as **skipped** with a reason ("No LLM configured for this tenant.") and the function returns — the model collections are already written, so the portal is usable either way.
5. Otherwise, computes and upserts each of the 5 `LLM_COLLECTIONS` in turn via `compute_llm_collection_from_data()`, catching `LlmUnavailableError` per-collection so one unreachable provider call doesn't take down the other four.

Each write is an upsert on `(tenant_id, collection, model_version, dataset_version)` — re-materializing the same version pair overwrites in place rather than accumulating duplicate rows.

```mermaid theme={null}
flowchart TD
    A["Source sync / file upload<br/>(services/ingest/pipeline.py)"] --> B["enqueue_platform_job<br/>job_type=materialize_insights"]
    C["Staff activates a checkpoint<br/>(POST .../activate)"] --> B
    D["Onboarding ingest commit<br/>(POST .../ingest/commit)"] --> B
    B --> E["platform_jobs row<br/>(control schema, idempotency_key)"]
    E --> F["materialize_insights_job worker<br/>(services/pipeline/job_worker.py)"]
    F --> G["materialize_insights()<br/>writes insight_collections"]
    G --> H["GET /insights/snapshot<br/>reads the newest client_visible pair"]
```

### When a snapshot refreshes

`materialize_insights` is enqueued (never run inline) from three places, each with its own `idempotency_key` so a retried request can't double-enqueue the same logical job:

* **A source sync or file upload commits a new dataset version** (`services/ingest/pipeline.py`) — `client_visible` is set to `tenant.lifecycle_state == "live"`, `source="ingest"`.
* **Staff activates a model checkpoint** (`POST /admin/tenants/{tenant_id}/activate`, `services/api/onboarding_api.py`) — re-materializes the latest dataset against the newly active checkpoint, `source="onboarding"`, `client_visible` from the request body's `share_onboarding_outputs`.
* **The onboarding ingest-commit flow** (`POST /admin/tenants/{tenant_id}/ingest/commit`, `services/api/onboarding_api.py`) — committing a staged upload into a new dataset version, `source="onboarding"`, `client_visible` from the tenant's current `share_onboarding_outputs` value, `idempotency_key f"materialize:{tenant.id}:{dataset_id}:{next_version}"`. `POST /admin/tenants/{tenant_id}/train` only enqueues a separate `train_tenant_model` job (`services/pipeline/training_job.py`) and never touches `materialize_insights`.

<Warning>
  `services/api/onboarding_api.py` also exposes `POST /admin/tenants/{tenant_id}/materialize-now`, which calls `materialize_insights()` **synchronously in the request handler** rather than enqueuing it. Its own doc comment marks it `TEMPORARY`: the durable worker path above depends on a Render background-worker plan that isn't provisioned yet, so without this endpoint a freshly demoed tenant's collections would stay perpetually `generating`. Remove the reliance on it once the worker is actually deployed — it is not part of the durable job queue's retry/idempotency guarantees.
</Warning>

## Console views and their collections

All four report-backed views (`src/components/views/{Home,Loops,Issues,Remediation}View.tsx`) call the same `useConsoleData()` hook (`src/lib/data/useConsoleData.ts`), which wraps `fetchClientInsights()` and — once `modelReady` is true — runs `adaptClientInsights()` (`src/lib/data/client-adapter.ts`) to turn the 9 raw collections into one `ConsoleData` object every view shares. No view fetches a collection directly.

| View                             | Primarily reads                                                                                | Notes                                                                                                                                                                                                                                                                          |
| -------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Home** (`/`)                   | `summary`, `themes`, `severity`, `lens`, `narratives` (hero line), `recommendations`           | The KPI row, hazard wave, and "today's signal" cards are pure functions of the adapted `ConsoleData`; the hero line prefers the LLM `narratives.narrative` once generated, falling back to a deterministic sentence built from `severity`/`issues` if it's still `generating`. |
| **Loops** (`/loops`)             | `themes`, `severity`, `issues`, `fishbone`                                                     | The constellation map and aggregation panel are keyed on `severity`-derived loop stats; the "Ingestion" disclosure's 5C completeness bars read `issues`/`fishbone` readiness directly.                                                                                         |
| **Issues** (`/issues`)           | `issues`, `themes`, `severity`, `fishbone` (for the driver bone chip)                          | List/board rows are per-`issues` entry; loop badges resolve back to `themes`/`t1` for the theme name.                                                                                                                                                                          |
| **Remediation** (`/remediation`) | `cause_remediation` (via `causes`), `fishbone`, plus the [live CAP API](/features/remediation) | Suggested fix plans come from the adapted `causes` array (sourced from `cause_remediation`, browser-side, when present); managed plans come from `GET /remediation/caps`, not from any insight collection.                                                                     |

Every view is built to tolerate the 4 LLM collections still `generating` once the 5 model collections are ready: `adaptClientInsights()` returns a `pending: CollectionName[]` list, and dependent fields fall back to fixed placeholder strings (`"Summary generating…"`, `"Recommendation generating…"`) rather than blank space.

## Loading, locked, and error states

`useConsoleData()` returns one of four statuses, and every report-backed view switches on all of them before rendering its normal body:

```mermaid theme={null}
stateDiagram-v2
    [*] --> loading
    loading --> ready: model_ready and adaptable
    loading --> locked: !share_onboarding_outputs and nothing client_visible yet
    loading --> error: fetch/parse failure after retries
    ready --> loading: refetch (still generating collections)
    error --> loading: user clicks Retry
```

* **`loading`** — the query is pending, or the model collections aren't ready yet. Views render `ViewSkeleton` (`src/components/shell/Skeletons.tsx`), a layout-matching shimmer so the loading→ready transition doesn't jump the page.
* **`locked`** — a real tenant onboarded with `share_onboarding_outputs` off, with nothing `client_visible` yet. Renders `LockView` (`src/components/shell/LockView.tsx`): "Training outputs are hidden for this workspace," with a single CTA to `/sources` — deliberately no self-service "run analysis" action, since this tenant was never meant to trigger the pipeline itself.
* **`ready`** — `data`, optional `pending` collection names, and optional `narrative` are available; the view renders normally.
* **`error`** — a genuine (non-404) fetch or parse failure surviving TanStack Query's retries, kept deliberately distinct from "no report yet" so a real backend outage doesn't read as an empty workspace. Renders `ErrorState` (`src/components/shell/ErrorState.tsx`) with a "Retry" button wired to the hook's own `refetch`.

There is no separate `"empty"` status in the current type (`ConsoleDataResult`'s own comment notes the brief specified `loading | empty | ready`, but a fetch failure needed its own state, and `locked` was added for the real, non-demo case) — a tenant with genuinely zero data but `share_onboarding_outputs` on simply stays in `loading` until the first `materialize_insights` run lands.

## Theme toggle

Independent of any of the above: `src/lib/stores/theme.ts` is a small Zustand store (`useTheme`) holding `"light" | "dark"`, defaulting to `"light"` on every render (including SSR) for hydration safety. `ThemeToggle` (`src/components/shell/ThemeToggle.tsx`) calls `toggleTheme()`, which flips `document.documentElement.dataset.theme` and persists the choice to `localStorage` (`cl-theme`) — wrapped in a `try`/`catch` since private-mode or quota-restricted storage can throw. `ThemeInit` reads the persisted value client-side, after mount, and applies it once; there is no flash-of-wrong-theme guard beyond that.

## Related

* [Remediation](/features/remediation)
* [Sources](/features/sources)
* [Client onboarding pipeline](/features/client-onboarding-pipeline)
* [Workers and jobs](/architecture/workers-and-jobs)