Onboarding to Inference: The Full Runbook

This is the complete path from a brand-new client to a live inference endpoint. Every call here is real — no mock steps. If you only want the operator-facing portal walkthrough, see Onboard a Client; this guide goes further and ends with a model actually serving traffic.

What you are building

client profile → data source → upload → validate → ingest issues
      → TRAIN  (clustering runs here)
      → EVALUATE (quality gates decide if the model is fit to serve)
      → PROMOTE (a passing model becomes the active artifact)
      → INFER  (POST /v1/inference/cluster assigns new issues to loops)

The clustering algorithm blends three signals — semantic embeddings, TF-IDF lexical overlap, and an 8-factor severity matrix — into one weighted graph, cuts it at an edge threshold, and runs greedy-modularity community detection to produce Tier 1 themes and Tier 2 loops nested inside them. Training freezes those loops as centroids; inference assigns new issues to the nearest loop, or honestly abstains.

Before you start: two settings that silently break everything

WM2_ARTIFACTS must be true. It defaults to false. With it off, training completes and looks successful, but never writes centroids.json, config_lock.json, or the model_artifacts row. Evaluation then 422s ("training run has no wm@2 artifact"), promotion has nothing to promote, and inference has nothing to load. This is the single most common way this runbook fails.

export WM2_ARTIFACTS=true          # REQUIRED — writes the artifacts eval/inference need
export MODEL_STORE_URI=file:///abs/path/to/model-store   # where artifacts land
export EVAL_REQUIRED=true          # keep true: refuses to serve an unevaluated model
export ENVIRONMENT=development
export AUTH_PROVIDER=none          # dev only
export PROVISIONING_PORTAL_ENABLED=true

Mint a staff token (dev):

python -m scripts.mint_dev_token ws_01 usr_01
export TOKEN=...        # staff role: ml_eng (training) / admin (overrides)

1. Create the client profile

curl -sX POST $API/v1/provisioning/client-profiles \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"display_name":"Meridian Home Health","tenant_slug":"meridian",
       "industry":"healthcare_home_health","primary_format":"xlsx_csv"}'

Returns {client: {id, workspace_id, status: "draft", ...}}. Keep id and workspace_id.

2. Register a data source, upload, validate

# register
curl -sX POST $API/v1/provisioning/client-profiles/$CLIENT/data-sources \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"kind":"upload"}'

# upload the client's issue export
curl -sX POST $API/v1/provisioning/client-profiles/$CLIENT/data-sources/$DS/upload \
  -H "Authorization: Bearer $TOKEN" -F "file=@issues.csv"

# validate: maps their columns onto our canonical fields, then INGESTS real issue rows
curl -sX POST $API/v1/provisioning/client-profiles/$CLIENT/data-sources/$DS/validate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"schema_map":{"Issue ID":"external_id","Issue Description":"text",
                     "Identification Date":"observed_at"}}'

Validate is the step that actually creates issues rows and embeds them. The client status flips draft → data_validated. You need enough issues for clustering to mean anything — a handful of rows will produce a degenerate partition that the gates will (correctly) reject.

The clustering text is built from the issue’s body plus the canonical concern / cause / consequence fields, with title backfilling concern when it is absent. Issues with no text at all are rejected rather than silently clustered on an empty string.

3. Train

curl -sX POST $API/v1/provisioning/client-profiles/$CLIENT/training-runs \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"kind":"initial","params":{}}'

This launches a real engine run: feature_forge → graph_weave → partition → dsu_crosscheck → loop_anchor → term_namer → …. Clustering happens inside those stages, using the pinned cluster config (not whatever is “currently active” — the run freezes its config by content hash so the run is reproducible). Poll it, or stream the console:

curl -s $API/v1/provisioning/training-runs/$RUN -H "Authorization: Bearer $TOKEN"
curl -sN $API/v1/provisioning/training-runs/$RUN/events \
  -H "Authorization: Bearer $TOKEN" -H 'Accept: text/event-stream'

Wait for status: "succeeded". The run registers a candidate model version — candidates are never auto-activated.

4. Evaluate — the gates decide, not you

curl -sX POST $API/v1/provisioning/training-runs/$RUN/evaluate \
  -H "Authorization: Bearer $TOKEN"

curl -s $API/v1/provisioning/training-runs/$RUN/evaluation \
  -H "Authorization: Bearer $TOKEN"    # the scorecard

The scorecard reports every gate. Hard gates block promotion; soft ones only inform:

Gate	Hard?	Asks
`E-DET`	hard	Is the training run bit-reproducible?
`E-COV`	hard	Did enough issues actually get clustered?
`E-DEG`	hard	Is the partition non-degenerate (no one mega-loop, sane loop count)?
`E-SIL`	hard	Do the clusters actually separate? (silhouette ≥ 0.20)
`E-HOLD`	hard	Do held-out issues land where they should?
`E-PROBE`	hard	Is assignment stable under small perturbations?
`E-AGR`	soft	Does an independent union-find crosscheck agree?

A failing hard gate is the system working. E-DEG and E-SIL exist to catch exactly the failure where clustering collapses everything into two enormous blobs — a result that looks like success (a run “succeeded”) but is useless. If E-SIL fails, do not lower the threshold. Fix the data or the config, and read the diagnosis in §8.

If the report is passed, or a human approves it:

curl -sX POST $API/v1/provisioning/training-runs/$RUN/approve -H "Authorization: Bearer $TOKEN"

5. Promote

Promotion is phased — you do not have to flip straight to live:

# optional: shadow (score in parallel, serve nothing) then canary (a % of traffic)
curl -sX POST $API/v1/provisioning/models/$WS/$VERSION/shadow  -H "Authorization: Bearer $TOKEN"
curl -sX POST $API/v1/provisioning/models/$WS/$VERSION/canary  -H "Authorization: Bearer $TOKEN" \
     -H 'Content-Type: application/json' -d '{"traffic_pct":10}'

# make it the active artifact
curl -sX POST $API/v1/provisioning/models/$WS/$VERSION/promote -H "Authorization: Bearer $TOKEN"

# and if it misbehaves
curl -sX POST $API/v1/provisioning/models/$WS/$VERSION/rollback -H "Authorization: Bearer $TOKEN"

Only a promoted version serves inference. With EVAL_REQUIRED=true (the default and the right setting), an artifact that never passed its gates will not be loaded, even if you point inference at it — the model simply reports itself unavailable.

6. Infer

Check what is serving:

curl -s $API/v1/inference/model -H "Authorization: Bearer $USER_TOKEN"

{ "model": { "available": true, "version": 3, "content_hash": "9f2c…",
             "embed_manifest": "bootstrap@v0", "loop_count": 36 } }

available: false is an honest state, not an error — it means nothing has been trained, or what was trained did not pass its gates and so was never promoted. The reason field says which. Then score issues:

curl -sX POST $API/v1/inference/cluster \
  -H "Authorization: Bearer $USER_TOKEN" -H 'Content-Type: application/json' \
  -d '{"issues":[{"id":"iss_123","text":"Nightly ACH settlement batch deadlocks and misses cutoff."}]}'

{
  "model": { "available": true, "version": 3, "loop_count": 36 },
  "assignments": [
    { "issue_id": "iss_123", "loop_id": "loop_7", "score": 0.81, "margin": 0.14,
      "abstained": false, "artifact_version": 3, "content_hash": "9f2c…" }
  ]
}

Key properties:

Tenancy comes from your token, never the request body — you cannot score against another tenant’s model.
It is read-only. It scores and returns; it creates no patterns and appends no events, so it needs no Idempotency-Key. (The ingest path does the stamping.)
It abstains rather than guesses. An issue whose best match is below the model’s tau_assign, below that loop’s local threshold, or too close to a runner-up comes back abstained: true with a reason (below_tau_assign, margin_too_small, no_embedding, no_promoted_model). An abstention is a correct answer, not a failure.
Issues that already have a stored embedding reuse it; unseen text is embedded on the fly with the same provider the rest of the system uses.

7. Use it from the platform

The frontend talks to the same endpoint through the standard client:

// apps/platform/lib/api/inference.ts
import { apiClient } from './client'

export async function getInferenceModel() {
  return apiClient.get('inference/model').json<{ model: ModelCard }>()
}

export async function clusterIssues(issues: { id: string; text: string }[]) {
  return apiClient.post('inference/cluster', { json: { issues } })
    .json<{ model: ModelCard; assignments: Assignment[] }>()
}

Auth headers are attached automatically. Add matching handlers to lib/mock/router.ts so mock-mode dev keeps working. Render abstentions explicitly — a loop_id: null with a reason is information the user should see, not an empty cell.

8. When the gates fail: reading the diagnosis

E-DEG fails / E-SIL below 0.20 / everything lands in 2 huge clusters. This is a signal balance problem, and it has a known cause. The three clustering signals live on very different scales:

signal	mean pairwise similarity	std — its actual discriminating power
embedding	0.327	0.111
lexical	0.055	0.050
severity	0.841	0.074

Severity has the highest mean and one of the smallest spreads — it is close to a constant offset rather than a signal. If it is weighted too heavily, it alone pushes every pair over the edge threshold: the graph becomes ~99.9% dense, the threshold stops doing anything, and community detection collapses the corpus into two blobs. There is a second, architectural reason the embedding must dominate: inference assigns by embedding cosine to loop centroids. A partition that is not embedding-coherent has centroids that mean nothing in the space where assignment actually happens — which is what E-SIL and E-PROBE detect. The shipped defaults account for both (w_embed 0.70 / w_tfidf 0.20 / w_severity 0.10, edge threshold 0.46). If you override them, keep severity as a tiebreaker.

Do not tune against E-SIL alone — it is gameable by fragmentation (a corpus split into singletons scores a perfect 1.0). E-COV and E-DEG are what stop that. Any retune has to clear all of them together.

You can tune this per workspace on the pinned cluster config:

curl -sX POST $API/v1/configs/cluster/versions \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"body":{"w_embed":0.7,"w_tfidf":0.2,"w_severity":0.1,
               "edge_threshold_large":0.46,"edge_threshold_small":0.36}}'

Then retrain. Because the run pins its config by content hash, the new run is reproducible and the old one is untouched. Everything abstains at inference. The model’s loops are tighter than your incoming issues. Check loop_count — if it is very high relative to your corpus, the partition is over-split. Evaluation 422s with “no wm@2 artifact”. WM2_ARTIFACTS is not true. See the warning at the top.

9. What is deliberately not here

There is no “just give me clusters without training” endpoint. Assignment requires a promoted artifact, on purpose: an assignment that cannot be traced to an evaluated, content-hashed model is not one you can defend to an auditor.
Inference never invents a loop. is_new_loop is always false; genuinely novel issues abstain and are picked up by the next training run.

​What you are building

​Before you start: two settings that silently break everything

​1. Create the client profile

​2. Register a data source, upload, validate

​3. Train

​4. Evaluate — the gates decide, not you

​5. Promote

​6. Infer

​7. Use it from the platform

​8. When the gates fail: reading the diagnosis

​9. What is deliberately not here

What you are building

Before you start: two settings that silently break everything

1. Create the client profile

2. Register a data source, upload, validate

3. Train

4. Evaluate — the gates decide, not you

5. Promote

6. Infer

7. Use it from the platform

8. When the gates fail: reading the diagnosis

9. What is deliberately not here