Skip to main content
The tenant console (/(console) routes in the frontend repo) never computes an insight on request. Every report a tenant user sees — loops, issues, severity, fishbone causes, LLM narratives — is precomputed by a background job into a Postgres cache table (insight_collections, tenant-schema) and the console only ever reads that cache. This page covers the read contract (GET /insights/snapshot), the job that fills it (materialize_insights), and how the four report-backed views (Home, Loops, Issues, Remediation) consume the result.

​
The ten collections

packages/causegraph/src/causegraph/onboarding/insights.py defines two groups, and services/api/insights_api.py imports both:
Model collections are pure clustering-model output — always computable once a tenant has an active model_checkpoints row and at least one committed dataset, no LLM required: LLM collections are generated per theme by the tenant’s configured LLM provider (llm_settings) and are skipped, not failed, when no provider is configured:
cause_remediation is computed and stored like every other collection, but insights_api.py deliberately excludes it from GET /insights/snapshot’s response (_EXPORT_ONLY_COLLECTIONS) — it feeds the branded XLSX export only. The console still reads its content indirectly: the export uses load_visible_snapshot() to include it, and the frontend’s own client-adapter.ts on the browser side does read a cause_remediation key from the raw snapshot response for CAPA rationale/evidence — but the deployed contract (below) only ever hands the browser 9 collections, so that field is always absent on the wire today.

​
GET /insights/snapshot

Defined in services/api/insights_api.py, requiring an authenticated tenant session. This is the only insights endpoint in the deployed product contract — services/api/product_app.py’s INSIGHT_OPERATIONS allowlist admits exactly ("GET", "/insights/snapshot"). The router also implements GET /insights/{collection} (single-collection read, 202 while generating) and GET /insights (per-collection ready/not-ready status used by the lock-screen decision), but neither is in the current allowlist and neither is called by the shipped console — the frontend’s fetchClientInsights() (src/lib/api/clientInsights.ts) only ever calls /insights/snapshot.
status per collection is one of:
  • ready — a row exists in insight_collections for the chosen (dataset_version, model_version) pair; payload is populated.
  • generating — no row exists yet for this collection at the chosen version pair. This covers any of the 5 MODEL_COLLECTIONS before the first materialize_insights run lands (a fresh tenant’s summary reports generating, not absent), and any LLM_COLLECTIONS entry whose provider is configured but hasn’t produced that collection yet (or is mid-run).
  • disabled — an LLM collection with no llm_settings row, or an empty/disabled provider, for this tenant. payload is always null here; the console renders a fixed placeholder string instead (“Narrative enrichment disabled.”, “No generated recommendation.”) rather than an empty state.
model_ready is true once all 5 MODEL_COLLECTIONS have a row; all_ready is true once every requested collection is anything other than generating (i.e. ready or disabled both count as settled).
The frontend’s useConsoleData() hook (src/lib/data/useConsoleData.ts) polls every 4 seconds via refetchInterval while any collection in the raw ClientInsightsSnapshot reports generating — it does not consult all_ready at all. clientInsights.ts’s fetchClientInsights() maps every one of the 9 collections missing from the response to { status: "generating" }, and cause_remediation (this page’s own Note above: never sent on the wire) is always missing. In practice this means the 4-second poll never turns off, since cause_remediation reports generating forever on the client. This reads as a frontend bug rather than intended behavior.

​
One coherent snapshot per read

load_visible_snapshot() — the function both /insights/snapshot and the remediation export share — first picks a single (dataset_version, model_version) pair (the newest one with at least one client_visible row) and only then reads every collection for that exact pair. This is deliberate: an older per-collection “latest” read could combine a themes collection from one materialize run with a fishbone collection from a different, later run — silently showing loops that don’t match the fishbone under them. Picking the version pair up front makes that impossible; a page load is always looking at output from one pipeline run.

​
client_visible and share_onboarding_outputs

Every insight_collections row carries a client_visible boolean and a source ("ingest", "onboarding", or "manual"). The snapshot query filters to client_visible = true, and additionally excludes source = "onboarding" rows unless the tenant’s share_onboarding_outputs flag is on:
  • Rows from a real post-go-live ingest (source = "ingest") are always client_visible — a live tenant always sees its own new data.
  • Rows produced during onboarding (source = "onboarding") are gated by share_onboarding_outputs, a per-tenant setting the staff onboarding flow controls (see Client onboarding pipeline) — some tenants are deliberately onboarded with training output hidden until formal go-live.

​
The materialize_insights job

services/pipeline/materialize.py’s materialize_insights(tenant, dataset_id, dataset_version, source, client_visible) is the only code path that ever writes insight_collections. It:
  1. Loads the tenant’s active model_checkpoints row (raises NoActiveCheckpointError if none exists).
  2. Loads every DatasetRecord row for the given dataset_id.
  3. Computes and upserts all 5 MODEL_COLLECTIONS unconditionally, via compute_model_collection_from_data() (the same storage-agnostic function the file-based research workbench uses).
  4. Loads the tenant’s llm_settings. If none is configured (or the model field is empty), every LLM_COLLECTIONS entry is recorded as skipped with a reason (“No LLM configured for this tenant.”) and the function returns — the model collections are already written, so the portal is usable either way.
  5. Otherwise, computes and upserts each of the 5 LLM_COLLECTIONS in turn via compute_llm_collection_from_data(), catching LlmUnavailableError per-collection so one unreachable provider call doesn’t take down the other four.
Each write is an upsert on (tenant_id, collection, model_version, dataset_version) — re-materializing the same version pair overwrites in place rather than accumulating duplicate rows.

​
When a snapshot refreshes

materialize_insights is enqueued (never run inline) from three places, each with its own idempotency_key so a retried request can’t double-enqueue the same logical job:
  • A source sync or file upload commits a new dataset version (services/ingest/pipeline.py) — client_visible is set to tenant.lifecycle_state == "live", source="ingest".
  • Staff activates a model checkpoint (POST /admin/tenants/{tenant_id}/activate, services/api/onboarding_api.py) — re-materializes the latest dataset against the newly active checkpoint, source="onboarding", client_visible from the request body’s share_onboarding_outputs.
  • The onboarding ingest-commit flow (POST /admin/tenants/{tenant_id}/ingest/commit, services/api/onboarding_api.py) — committing a staged upload into a new dataset version, source="onboarding", client_visible from the tenant’s current share_onboarding_outputs value, idempotency_key f"materialize:{tenant.id}:{dataset_id}:{next_version}". POST /admin/tenants/{tenant_id}/train only enqueues a separate train_tenant_model job (services/pipeline/training_job.py) and never touches materialize_insights.
services/api/onboarding_api.py also exposes POST /admin/tenants/{tenant_id}/materialize-now, which calls materialize_insights() synchronously in the request handler rather than enqueuing it. Its own doc comment marks it TEMPORARY: the durable worker path above depends on a Render background-worker plan that isn’t provisioned yet, so without this endpoint a freshly demoed tenant’s collections would stay perpetually generating. Remove the reliance on it once the worker is actually deployed — it is not part of the durable job queue’s retry/idempotency guarantees.

​
Console views and their collections

All four report-backed views (src/components/views/{Home,Loops,Issues,Remediation}View.tsx) call the same useConsoleData() hook (src/lib/data/useConsoleData.ts), which wraps fetchClientInsights() and — once modelReady is true — runs adaptClientInsights() (src/lib/data/client-adapter.ts) to turn the 9 raw collections into one ConsoleData object every view shares. No view fetches a collection directly. Every view is built to tolerate the 4 LLM collections still generating once the 5 model collections are ready: adaptClientInsights() returns a pending: CollectionName[] list, and dependent fields fall back to fixed placeholder strings ("Summary generating…", "Recommendation generating…") rather than blank space.

​
Loading, locked, and error states

useConsoleData() returns one of four statuses, and every report-backed view switches on all of them before rendering its normal body:
  • loading — the query is pending, or the model collections aren’t ready yet. Views render ViewSkeleton (src/components/shell/Skeletons.tsx), a layout-matching shimmer so the loading→ready transition doesn’t jump the page.
  • locked — a real tenant onboarded with share_onboarding_outputs off, with nothing client_visible yet. Renders LockView (src/components/shell/LockView.tsx): “Training outputs are hidden for this workspace,” with a single CTA to /sources — deliberately no self-service “run analysis” action, since this tenant was never meant to trigger the pipeline itself.
  • ready — data, optional pending collection names, and optional narrative are available; the view renders normally.
  • error — a genuine (non-404) fetch or parse failure surviving TanStack Query’s retries, kept deliberately distinct from “no report yet” so a real backend outage doesn’t read as an empty workspace. Renders ErrorState (src/components/shell/ErrorState.tsx) with a “Retry” button wired to the hook’s own refetch.
There is no separate "empty" status in the current type (ConsoleDataResult’s own comment notes the brief specified loading | empty | ready, but a fetch failure needed its own state, and locked was added for the real, non-demo case) — a tenant with genuinely zero data but share_onboarding_outputs on simply stays in loading until the first materialize_insights run lands.

​
Theme toggle

Independent of any of the above: src/lib/stores/theme.ts is a small Zustand store (useTheme) holding "light" | "dark", defaulting to "light" on every render (including SSR) for hydration safety. ThemeToggle (src/components/shell/ThemeToggle.tsx) calls toggleTheme(), which flips document.documentElement.dataset.theme and persists the choice to localStorage (cl-theme) — wrapped in a try/catch since private-mode or quota-restricted storage can throw. ThemeInit reads the persisted value client-side, after mount, and applies it once; there is no flash-of-wrong-theme guard beyond that.