> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Lifecycle

> The real state machines behind a client's model — client profile, evaluation, registry status, and promotion phase — plus R-L1 (evaluation is mandatory) and R-L2 (overrides are audited forever).

<Warning>
  There is **no single unified `draft → ... → active` enum** anywhere in the code. A client's
  model moves through **four independent state machines** that compose together. Documenting
  them as one linear chain would misrepresent the system — this page documents the real,
  separate machines, verified against the schema and service code on this branch.
</Warning>

## The four state machines

### 1. `client_profiles.status` — the portal-visible stage

```sql theme={null}
CHECK (status IN ('draft','data_validated','training','trained','hosting','live','archived'))
```

(`db/migrations/0060_provisioning_domain.sql`)

Legal transitions, enforced by `assert_valid_status_transition` (`app/store/provisioning_domain.py`,
409 `invalid_transition` otherwise):

```
draft          -> data_validated
data_validated -> data_validated (re-validate)  | training
training       -> trained
trained        -> training | data_validated (D5 reject, initial-kind) | hosting
hosting        -> live | trained | training
live           -> trained | training
```

`hosting`/`live` re-entering `training` directly is deliberate: retraining an already-hosted
client is a normal, supported flow. `archived` is a declared CHECK value with **no code path
that ever sets it** — it exists in the schema but nothing in the current codebase transitions a
client there. Document it as defined-but-unused, not as a reachable end state.

### 2. `eval_runs.status` — the evaluation execution

```sql theme={null}
status text NOT NULL DEFAULT 'running' CHECK (status IN ('running','passed','failed','error'))
```

(`db/migrations/0063_lifecycle_eval.sql`) Exactly four values — "approved" / "overridden" /
"rejected" are **not** separate status values; they live one level down, in
`eval_runs.report.decision.kind` (a free-form jsonb field), set by the D3/D4/D5 endpoints.

### 3. `workspace_models.status` — the registry row

The column comment in `db/schema.sql` says `-- 'active' | 'superseded' | 'failed'`, but this is
**stale**: the actual code also writes `candidate`:

```python theme={null}
registry_row = {
    ...
    "status": "candidate" if portal_managed else "active",
    ...
}
```

(`app/services/model_lifecycle.py`) There is no DB `CHECK` constraint on this column, so
`candidate` is a real, uncontradicted value. A portal-managed training run (`trigger=
"model_train"`, the only trigger the portal's C2/F1 endpoints ever use) registers as
`candidate` and **never auto-activates** — only `promote()` (below) or hosting's `activate()`
can move a portal-managed version to `active`. This closes a real gap: before this behavior
existed, a never-evaluated candidate could start serving production traffic (via the in-process
inline inference path) the instant training finished.

### 4. `model_promotions.phase` — the traffic-routing pipeline

```sql theme={null}
phase text NOT NULL CHECK (phase IN ('shadow','canary','active','rolled_back'))
```

(`db/migrations/0063_lifecycle_eval.sql`), driven by `app/services/promotion.py`:

* **`start_shadow`** — legal only when no open (`shadow`/`canary`) or `active` promotion exists
  for that `(workspace, version)`. Writes **only** a `model_promotions` row —
  zero other side effects, zero traffic change. Shadow-phase assignment records land in the
  separate, throwaway `shadow_assignments` table (14-day TTL sweep), never touching the
  registry or live routing.
* **`start_canary`** — legal only from `shadow` or `canary` (covers both first entry and a
  percentage adjustment). Canary arm routing is deterministic:
  `sha256(content_hash) mod 100 < canary_pct` — never a salted hash, so the same issue always
  routes the same way for a given candidate.
* **`promote`** — legal only from `shadow` or `canary`. Atomically flips the
  `workspace_models` registry (`candidate → active`, prior active → `superseded`). This is
  **distinct from `activate`** (hosting's E5, service go-live): `promote` flips which model
  version is the workspace's registered active model; `activate` flips which *service* receives
  traffic. They are two different mutations on two different tables.
* **`rollback`** — legal only when the latest promotion for that version is `phase='active'`
  and has a recorded `incumbent_version`; re-activates the incumbent atomically and freezes the
  rolled-back version's final stats.
* **`compute_promotion_stats`** — zero matched (candidate, incumbent) sample pairs returns
  `sample_count: 0` and every rate field `null`, **never** a fabricated `0.0` or percentage.
  `latency_p95_ms` is always `null` — there is no timestamp-delta column and no real hosted
  round-trip measured against it; this is a documented gap, not a stub pretending to work.

## How the stages compose

Putting the four machines together (not a literal enum — a narrative across all four):

```
client_profiles.status:   draft → data_validated → training → trained ────────────────► hosting → live
                                                                    │
eval_runs.status:                                          (created) running → passed/failed/error
                                                                    │
eval_runs.report.decision.kind:                          approved / overridden / rejected
                                                                    │
workspace_models.status:                                    candidate ─────────► active (at promote) → superseded (later)
                                                                    │
model_promotions.phase:                                    shadow → canary → active / rolled_back
```

A client can sit at `client_profiles.status=trained` for the entire evaluate→approve→promote
window — that whole window has no dedicated client status of its own.

## R-L1 — evaluation is mandatory

There is no code path that lets a training run reach `approved` without a real evaluation
having genuinely passed:

```python theme={null}
def approve_eval_run(repo, training_run_id: str, *, actor_user_id: str) -> Dict:
    """D3 — staff:ml_eng confirms a PASSED evaluation. 409 unless the
    latest eval_run's status is 'passed'."""
    existing = repo.list_eval_runs_for_training_run(training_run_id)
    if not existing:
        raise ApiError(404, "not_found", "no evaluation exists...")
    latest = existing[0]
    if latest["status"] != "passed":
        raise ApiError(409, "invalid_request",
            f"cannot approve — latest evaluation status is {latest['status']!r}, not 'passed'")
```

(`app/services/model_eval.py`) `passed` can only be set by `run_and_persist_gates` — the real
gate-evaluation harness (see [Evaluation Gates](/deploy-security/evaluation-gates)). This is
enforced structurally, not just by convention.

Hosting itself carries a second, independent enforcement point — `EVAL_REQUIRED` (default
`True`):

```python theme={null}
def assert_eval_required_satisfied(repo, training_run_id: str) -> None:
    """Raises ApiError(409) unless the training run's evaluation genuinely
    PASSED or was explicitly OVERRIDDEN (D4)."""
```

(`app/services/model_eval.py`, called from `hosting.create_service`, E1) — a merely-`trained`,
never-evaluated, or failed-and-not-overridden run is refused with `409 eval_required`. Setting
`EVAL_REQUIRED=false` disables this specific check only; it does not disable R-L1's approval
gate above.

## R-L2 — override requires staff:admin, a reason, and the artifact is flagged forever

```http theme={null}
POST /v1/provisioning/training-runs/{run_id}/override
Authorization: Bearer <staff:admin JWT>
{"reason": "Corpus is a known-noisy fixture; SRE reviewed manually and signed off."}
```

Guarded by **both** `require_staff("admin")` and `require_feature("internal.eval_override")` —
`staff:admin` is the only role with that feature key. The reason is enforced twice — once by
the RBAC/feature gate implicitly restricting who can even call this, and once explicitly in the
service:

```python theme={null}
if not reason or not reason.strip():
    raise ApiError(422, "invalid_request", "override reason is required")
```

(`app/services/model_eval.py`) The override is not just a database flag — it **physically
rewrites the stored `manifest.json`** in the model store:

```python theme={null}
eval_block = dict(manifest.get("eval") or {})
eval_block.update({
    "overridden": True, "overridden_by": actor_user_id, "overridden_at": _now_iso(),
    "overridden_reason": reason,
})
manifest["eval"] = eval_block
store.put(workspace_id, candidate_version, "manifest.json", ...)
```

This is deliberate — "surfaced on the artifact forever," per the spec's own framing. The
`workspace_models.manifest` DB row (a denormalized copy the go-live summary reads) is
separately synced via `repo.update_workspace_model_manifest(...)` in the same call, so the
override shows up everywhere the artifact's manifest is read from — not just the physical
store. Every override is also audited (`provisioning.eval_override`, with before/after
snapshots) and emits a `model.lifecycle` event (`eval_failed → approved_via_override`).

<Note>
  `content_hash` and `determinism_hash` are frozen at **train time**, before any evaluation ever
  runs. An override never touches either hash — the manifest's integrity fields and its
  eval-decision fields are deliberately independent, so a future integrity check never mistakes
  an honest override for tampering.
</Note>

## Related pages

* [Evaluation Gates](/deploy-security/evaluation-gates) — every gate, its threshold, and the calibration gap
* [Provisioning Portal](/deploy-security/provisioning-portal) — the RBAC model behind these endpoints
* [Inference & Queue](/deploy-security/inference-and-queue) — the wm\@2 artifact this lifecycle produces