> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Onboarding to Inference: The Full Runbook

> End-to-end: onboard a client, ingest their issues, train a clustering model, pass the evaluation gates, promote it, and serve live inference from it.

<Info>
  This is the complete path from a brand-new client to a live inference endpoint. Every call
  here is real — no mock steps. If you only want the operator-facing portal walkthrough,
  see [Onboard a Client](/guides/onboard-a-client); this guide goes further and ends with a
  model actually serving traffic.
</Info>

## What you are building

```text theme={null}
client profile → data source → upload → validate → ingest issues
      → TRAIN  (clustering runs here)
      → EVALUATE (quality gates decide if the model is fit to serve)
      → PROMOTE (a passing model becomes the active artifact)
      → INFER  (POST /v1/inference/cluster assigns new issues to loops)
```

The clustering algorithm blends three signals — semantic embeddings, TF-IDF lexical
overlap, and an 8-factor severity matrix — into one weighted graph, cuts it at an edge
threshold, and runs greedy-modularity community detection to produce **Tier 1 themes** and
**Tier 2 loops** nested inside them. Training freezes those loops as centroids; inference
assigns new issues to the nearest loop, or honestly abstains.

## Before you start: two settings that silently break everything

<Warning>
  **`WM2_ARTIFACTS` must be `true`.** It defaults to `false`. With it off, training completes
  and *looks* successful, but never writes `centroids.json`, `config_lock.json`, or the
  `model_artifacts` row. Evaluation then 422s (`"training run has no wm@2 artifact"`),
  promotion has nothing to promote, and inference has nothing to load. This is the single
  most common way this runbook fails.
</Warning>

```bash theme={null}
export WM2_ARTIFACTS=true          # REQUIRED — writes the artifacts eval/inference need
export MODEL_STORE_URI=file:///abs/path/to/model-store   # where artifacts land
export EVAL_REQUIRED=true          # keep true: refuses to serve an unevaluated model
export ENVIRONMENT=development
export AUTH_PROVIDER=none          # dev only
export PROVISIONING_PORTAL_ENABLED=true
```

Mint a staff token (dev):

```bash theme={null}
python -m scripts.mint_dev_token ws_01 usr_01
export TOKEN=...        # staff role: ml_eng (training) / admin (overrides)
```

## 1. Create the client profile

```bash theme={null}
curl -sX POST $API/v1/provisioning/client-profiles \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"display_name":"Meridian Home Health","tenant_slug":"meridian",
       "industry":"healthcare_home_health","primary_format":"xlsx_csv"}'
```

Returns `{client: {id, workspace_id, status: "draft", ...}}`. Keep `id` and `workspace_id`.

## 2. Register a data source, upload, validate

```bash theme={null}
# register
curl -sX POST $API/v1/provisioning/client-profiles/$CLIENT/data-sources \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"kind":"upload"}'

# upload the client's issue export
curl -sX POST $API/v1/provisioning/client-profiles/$CLIENT/data-sources/$DS/upload \
  -H "Authorization: Bearer $TOKEN" -F "file=@issues.csv"

# validate: maps their columns onto our canonical fields, then INGESTS real issue rows
curl -sX POST $API/v1/provisioning/client-profiles/$CLIENT/data-sources/$DS/validate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"schema_map":{"Issue ID":"external_id","Issue Description":"text",
                     "Identification Date":"observed_at"}}'
```

Validate is the step that actually creates `issues` rows and embeds them. The client status
flips `draft → data_validated`. **You need enough issues for clustering to mean anything** —
a handful of rows will produce a degenerate partition that the gates will (correctly) reject.

<Note>
  The clustering text is built from the issue's `body` plus the canonical `concern` / `cause` /
  `consequence` fields, with `title` backfilling `concern` when it is absent. Issues with no
  text at all are rejected rather than silently clustered on an empty string.
</Note>

## 3. Train

```bash theme={null}
curl -sX POST $API/v1/provisioning/client-profiles/$CLIENT/training-runs \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"kind":"initial","params":{}}'
```

This launches a real engine run: `feature_forge → graph_weave → partition → dsu_crosscheck
→ loop_anchor → term_namer → …`. Clustering happens inside those stages, using the
**pinned** `cluster` config (not whatever is "currently active" — the run freezes its config
by content hash so the run is reproducible).

Poll it, or stream the console:

```bash theme={null}
curl -s $API/v1/provisioning/training-runs/$RUN -H "Authorization: Bearer $TOKEN"
curl -sN $API/v1/provisioning/training-runs/$RUN/events \
  -H "Authorization: Bearer $TOKEN" -H 'Accept: text/event-stream'
```

Wait for `status: "succeeded"`. The run registers a **`candidate`** model version — candidates
are never auto-activated.

## 4. Evaluate — the gates decide, not you

```bash theme={null}
curl -sX POST $API/v1/provisioning/training-runs/$RUN/evaluate \
  -H "Authorization: Bearer $TOKEN"

curl -s $API/v1/provisioning/training-runs/$RUN/evaluation \
  -H "Authorization: Bearer $TOKEN"    # the scorecard
```

The scorecard reports every gate. **Hard** gates block promotion; soft ones only inform:

| Gate      | Hard? | Asks                                                                 |
| --------- | ----- | -------------------------------------------------------------------- |
| `E-DET`   | hard  | Is the training run bit-reproducible?                                |
| `E-COV`   | hard  | Did enough issues actually get clustered?                            |
| `E-DEG`   | hard  | Is the partition non-degenerate (no one mega-loop, sane loop count)? |
| `E-SIL`   | hard  | Do the clusters actually *separate*? (silhouette ≥ 0.20)             |
| `E-HOLD`  | hard  | Do held-out issues land where they should?                           |
| `E-PROBE` | hard  | Is assignment stable under small perturbations?                      |
| `E-AGR`   | soft  | Does an independent union-find crosscheck agree?                     |

<Warning>
  **A failing hard gate is the system working.** `E-DEG` and `E-SIL` exist to catch exactly the
  failure where clustering collapses everything into two enormous blobs — a result that looks
  like success (a run "succeeded") but is useless. If E-SIL fails, do not lower the threshold.
  Fix the data or the config, and read the diagnosis in §8.
</Warning>

If the report is `passed`, or a human approves it:

```bash theme={null}
curl -sX POST $API/v1/provisioning/training-runs/$RUN/approve -H "Authorization: Bearer $TOKEN"
```

## 5. Promote

Promotion is phased — you do not have to flip straight to live:

```bash theme={null}
# optional: shadow (score in parallel, serve nothing) then canary (a % of traffic)
curl -sX POST $API/v1/provisioning/models/$WS/$VERSION/shadow  -H "Authorization: Bearer $TOKEN"
curl -sX POST $API/v1/provisioning/models/$WS/$VERSION/canary  -H "Authorization: Bearer $TOKEN" \
     -H 'Content-Type: application/json' -d '{"traffic_pct":10}'

# make it the active artifact
curl -sX POST $API/v1/provisioning/models/$WS/$VERSION/promote -H "Authorization: Bearer $TOKEN"

# and if it misbehaves
curl -sX POST $API/v1/provisioning/models/$WS/$VERSION/rollback -H "Authorization: Bearer $TOKEN"
```

Only a **promoted** version serves inference. With `EVAL_REQUIRED=true` (the default and the
right setting), an artifact that never passed its gates will **not** be loaded, even if you
point inference at it — the model simply reports itself unavailable.

## 6. Infer

Check what is serving:

```bash theme={null}
curl -s $API/v1/inference/model -H "Authorization: Bearer $USER_TOKEN"
```

```json theme={null}
{ "model": { "available": true, "version": 3, "content_hash": "9f2c…",
             "embed_manifest": "bootstrap@v0", "loop_count": 36 } }
```

`available: false` is an honest state, not an error — it means nothing has been trained, or
what was trained did not pass its gates and so was never promoted. The `reason` field says which.

Then score issues:

```bash theme={null}
curl -sX POST $API/v1/inference/cluster \
  -H "Authorization: Bearer $USER_TOKEN" -H 'Content-Type: application/json' \
  -d '{"issues":[{"id":"iss_123","text":"Nightly ACH settlement batch deadlocks and misses cutoff."}]}'
```

```json theme={null}
{
  "model": { "available": true, "version": 3, "loop_count": 36 },
  "assignments": [
    { "issue_id": "iss_123", "loop_id": "loop_7", "score": 0.81, "margin": 0.14,
      "abstained": false, "artifact_version": 3, "content_hash": "9f2c…" }
  ]
}
```

Key properties:

* **Tenancy comes from your token**, never the request body — you cannot score against another
  tenant's model.
* **It is read-only.** It scores and returns; it creates no patterns and appends no events, so
  it needs no `Idempotency-Key`. (The ingest path does the stamping.)
* **It abstains rather than guesses.** An issue whose best match is below the model's
  `tau_assign`, below that loop's local threshold, or too close to a runner-up comes back
  `abstained: true` with a `reason` (`below_tau_assign`, `margin_too_small`, `no_embedding`,
  `no_promoted_model`). An abstention is a *correct* answer, not a failure.
* Issues that already have a stored embedding reuse it; unseen text is embedded on the fly
  with the same provider the rest of the system uses.

## 7. Use it from the platform

The frontend talks to the same endpoint through the standard client:

```ts theme={null}
// apps/platform/lib/api/inference.ts
import { apiClient } from './client'

export async function getInferenceModel() {
  return apiClient.get('inference/model').json<{ model: ModelCard }>()
}

export async function clusterIssues(issues: { id: string; text: string }[]) {
  return apiClient.post('inference/cluster', { json: { issues } })
    .json<{ model: ModelCard; assignments: Assignment[] }>()
}
```

Auth headers are attached automatically. Add matching handlers to `lib/mock/router.ts` so
mock-mode dev keeps working. Render abstentions explicitly — a `loop_id: null` with a reason
is information the user should see, not an empty cell.

## 8. When the gates fail: reading the diagnosis

**`E-DEG` fails / `E-SIL` below 0.20 / everything lands in 2 huge clusters.**
This is a *signal balance* problem, and it has a known cause. The three clustering signals
live on very different scales:

| signal    | mean pairwise similarity | std — its actual discriminating power |
| --------- | ------------------------ | ------------------------------------- |
| embedding | 0.327                    | 0.111                                 |
| lexical   | 0.055                    | 0.050                                 |
| severity  | **0.841**                | **0.074**                             |

Severity has the *highest* mean and one of the *smallest* spreads — it is close to a constant
offset rather than a signal. If it is weighted too heavily, it alone pushes every pair over the
edge threshold: the graph becomes \~99.9% dense, the threshold stops doing anything, and
community detection collapses the corpus into two blobs.

There is a second, architectural reason the embedding must dominate: **inference assigns by
embedding cosine to loop centroids**. A partition that is not embedding-coherent has centroids
that mean nothing in the space where assignment actually happens — which is what `E-SIL` and
`E-PROBE` detect. The shipped defaults account for both (`w_embed 0.70 / w_tfidf 0.20 /
w_severity 0.10`, edge threshold `0.46`). If you override them, keep severity as a *tiebreaker*.

<Warning>
  Do not tune against `E-SIL` alone — it is gameable by fragmentation (a corpus split into
  singletons scores a perfect 1.0). `E-COV` and `E-DEG` are what stop that. Any retune has to clear
  all of them together.
</Warning>

You can tune this per workspace on the pinned `cluster` config:

```bash theme={null}
curl -sX POST $API/v1/configs/cluster/versions \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"body":{"w_embed":0.7,"w_tfidf":0.2,"w_severity":0.1,
               "edge_threshold_large":0.46,"edge_threshold_small":0.36}}'
```

Then retrain. Because the run pins its config by content hash, the new run is reproducible and
the old one is untouched.

**Everything abstains at inference.** The model's loops are tighter than your incoming issues.
Check `loop_count` — if it is very high relative to your corpus, the partition is over-split.

**Evaluation 422s with "no wm\@2 artifact".** `WM2_ARTIFACTS` is not `true`. See the warning at
the top.

## 9. What is deliberately *not* here

* There is no "just give me clusters without training" endpoint. Assignment requires a promoted
  artifact, on purpose: an assignment that cannot be traced to an evaluated, content-hashed
  model is not one you can defend to an auditor.
* Inference never invents a loop. `is_new_loop` is always false; genuinely novel issues abstain
  and are picked up by the next training run.