Skip to main content
There is no single unified draft → ... → active enum anywhere in the code. A client’s model moves through four independent state machines that compose together. Documenting them as one linear chain would misrepresent the system — this page documents the real, separate machines, verified against the schema and service code on this branch.

​
The four state machines

​
1. client_profiles.status — the portal-visible stage

CHECK (status IN ('draft','data_validated','training','trained','hosting','live','archived'))
(db/migrations/0060_provisioning_domain.sql) Legal transitions, enforced by assert_valid_status_transition (app/store/provisioning_domain.py, 409 invalid_transition otherwise): 
draft          -> data_validated
data_validated -> data_validated (re-validate)  | training
training       -> trained
trained        -> training | data_validated (D5 reject, initial-kind) | hosting
hosting        -> live | trained | training
live           -> trained | training
hosting/live re-entering training directly is deliberate: retraining an already-hosted client is a normal, supported flow. archived is a declared CHECK value with no code path that ever sets it — it exists in the schema but nothing in the current codebase transitions a client there. Document it as defined-but-unused, not as a reachable end state.

​
2. eval_runs.status — the evaluation execution

status text NOT NULL DEFAULT 'running' CHECK (status IN ('running','passed','failed','error'))
(db/migrations/0063_lifecycle_eval.sql) Exactly four values — “approved” / “overridden” / “rejected” are not separate status values; they live one level down, in eval_runs.report.decision.kind (a free-form jsonb field), set by the D3/D4/D5 endpoints.

​
3. workspace_models.status — the registry row

The column comment in db/schema.sql says -- 'active' | 'superseded' | 'failed', but this is stale: the actual code also writes candidate: 
registry_row = {
    ...
    "status": "candidate" if portal_managed else "active",
    ...
}
(app/services/model_lifecycle.py) There is no DB CHECK constraint on this column, so candidate is a real, uncontradicted value. A portal-managed training run (trigger= "model_train", the only trigger the portal’s C2/F1 endpoints ever use) registers as candidate and never auto-activates — only promote() (below) or hosting’s activate() can move a portal-managed version to active. This closes a real gap: before this behavior existed, a never-evaluated candidate could start serving production traffic (via the in-process inline inference path) the instant training finished.

​
4. model_promotions.phase — the traffic-routing pipeline

phase text NOT NULL CHECK (phase IN ('shadow','canary','active','rolled_back'))
(db/migrations/0063_lifecycle_eval.sql), driven by app/services/promotion.py:
  • start_shadow — legal only when no open (shadow/canary) or active promotion exists for that (workspace, version). Writes only a model_promotions row — zero other side effects, zero traffic change. Shadow-phase assignment records land in the separate, throwaway shadow_assignments table (14-day TTL sweep), never touching the registry or live routing.
  • start_canary — legal only from shadow or canary (covers both first entry and a percentage adjustment). Canary arm routing is deterministic: sha256(content_hash) mod 100 < canary_pct — never a salted hash, so the same issue always routes the same way for a given candidate.
  • promote — legal only from shadow or canary. Atomically flips the workspace_models registry (candidate → active, prior active → superseded). This is distinct from activate (hosting’s E5, service go-live): promote flips which model version is the workspace’s registered active model; activate flips which service receives traffic. They are two different mutations on two different tables.
  • rollback — legal only when the latest promotion for that version is phase='active' and has a recorded incumbent_version; re-activates the incumbent atomically and freezes the rolled-back version’s final stats.
  • compute_promotion_stats — zero matched (candidate, incumbent) sample pairs returns sample_count: 0 and every rate field null, never a fabricated 0.0 or percentage. latency_p95_ms is always null — there is no timestamp-delta column and no real hosted round-trip measured against it; this is a documented gap, not a stub pretending to work.

​
How the stages compose

Putting the four machines together (not a literal enum — a narrative across all four): 
client_profiles.status:   draft → data_validated → training → trained ────────────────► hosting → live
                                                                    │
eval_runs.status:                                          (created) running → passed/failed/error
                                                                    │
eval_runs.report.decision.kind:                          approved / overridden / rejected
                                                                    │
workspace_models.status:                                    candidate ─────────► active (at promote) → superseded (later)
                                                                    │
model_promotions.phase:                                    shadow → canary → active / rolled_back
A client can sit at client_profiles.status=trained for the entire evaluate→approve→promote window — that whole window has no dedicated client status of its own.

​
R-L1 — evaluation is mandatory

There is no code path that lets a training run reach approved without a real evaluation having genuinely passed: 
def approve_eval_run(repo, training_run_id: str, *, actor_user_id: str) -> Dict:
    """D3 — staff:ml_eng confirms a PASSED evaluation. 409 unless the
    latest eval_run's status is 'passed'."""
    existing = repo.list_eval_runs_for_training_run(training_run_id)
    if not existing:
        raise ApiError(404, "not_found", "no evaluation exists...")
    latest = existing[0]
    if latest["status"] != "passed":
        raise ApiError(409, "invalid_request",
            f"cannot approve — latest evaluation status is {latest['status']!r}, not 'passed'")
(app/services/model_eval.py) passed can only be set by run_and_persist_gates — the real gate-evaluation harness (see Evaluation Gates). This is enforced structurally, not just by convention. Hosting itself carries a second, independent enforcement point — EVAL_REQUIRED (default True): 
def assert_eval_required_satisfied(repo, training_run_id: str) -> None:
    """Raises ApiError(409) unless the training run's evaluation genuinely
    PASSED or was explicitly OVERRIDDEN (D4)."""
(app/services/model_eval.py, called from hosting.create_service, E1) — a merely-trained, never-evaluated, or failed-and-not-overridden run is refused with 409 eval_required. Setting EVAL_REQUIRED=false disables this specific check only; it does not disable R-L1’s approval gate above.

​
R-L2 — override requires staff:admin, a reason, and the artifact is flagged forever

POST /v1/provisioning/training-runs/{run_id}/override
Authorization: Bearer <staff:admin JWT>
{"reason": "Corpus is a known-noisy fixture; SRE reviewed manually and signed off."}
Guarded by both require_staff("admin") and require_feature("internal.eval_override") — staff:admin is the only role with that feature key. The reason is enforced twice — once by the RBAC/feature gate implicitly restricting who can even call this, and once explicitly in the service: 
if not reason or not reason.strip():
    raise ApiError(422, "invalid_request", "override reason is required")
(app/services/model_eval.py) The override is not just a database flag — it physically rewrites the stored manifest.json in the model store: 
eval_block = dict(manifest.get("eval") or {})
eval_block.update({
    "overridden": True, "overridden_by": actor_user_id, "overridden_at": _now_iso(),
    "overridden_reason": reason,
})
manifest["eval"] = eval_block
store.put(workspace_id, candidate_version, "manifest.json", ...)
This is deliberate — “surfaced on the artifact forever,” per the spec’s own framing. The workspace_models.manifest DB row (a denormalized copy the go-live summary reads) is separately synced via repo.update_workspace_model_manifest(...) in the same call, so the override shows up everywhere the artifact’s manifest is read from — not just the physical store. Every override is also audited (provisioning.eval_override, with before/after snapshots) and emits a model.lifecycle event (eval_failed → approved_via_override).
content_hash and determinism_hash are frozen at train time, before any evaluation ever runs. An override never touches either hash — the manifest’s integrity fields and its eval-decision fields are deliberately independent, so a future integrity check never mistakes an honest override for tampering.