app/services/model_eval.py, portal stage 05) is the mandatory quality
gate between a trained candidate and a client’s production traffic — see
Model Lifecycle for how approve/override/reject
compose with it. Every gate runs against frozen inputs — the candidate’s wm@2 manifest and
artifact files, plus pinned datasets — never live tables, so a completed evaluation is
reproducible and replayable.
Honesty rule: skipped is not passed
This is the load-bearing design decision on this page. Quoted directly from the code’s own
module docstring: “a gate that cannot be computed honestly is recorded SKIPPED —
measured.skipped=true, passed=False (never fabricated True) — and does NOT block.”passed: false with measured: {"skipped": true, "reason": "..."} — it is never silently counted as a pass, and (per _aggregate_status) it never blocks
the overall result either. Only a genuinely computed and failed hard gate fails the
evaluation. The most common skip reason across this table is WM2_ARTIFACTS having been off at
train time — most gates read wm@2-only files (loop_stats.json, centroids.json) that simply
don’t exist under the older wm@1 manifest shape.
Initial-training gates (always run)
| Gate | Hard? | Measures | Default threshold | Skipped when |
|---|---|---|---|---|
| E-DET | Hard (soft above 500k records) | Twin-train determinism_hash equality | exact match | No wm@2 determinism_hash on the candidate (WM2_ARTIFACTS was off) |
| E-COV | Hard | % of corpus covered by a loop ≥ min_cluster (3) members | e_cov = 0.85 | loop_stats.json unavailable, or no record_count |
| E-SIL | Hard | Centroid-approximated mean silhouette | e_sil = 0.20 (recalibrated 2026-07, was 0.45) | centroids.json/prior_map unavailable, or fewer than 2 loops |
| E-DEG | Hard | Degenerate partition: max loop share ≤ 0.40 AND loop count in a corpus-relative range | max_loop_share = 0.40 | loop_stats.json unavailable |
| E-AGR | Soft (demoted 2026-07 — was hard) | leiden_dsu_agreement, read straight off the candidate’s own recorded training stats | e_agr = 0.90 (advisory line, value unchanged) | No stability.leiden_dsu_agreement recorded on the candidate |
| E-HOLD | Hard | Held-out issue’s 5-nearest-neighbor majority vote matches its own assigned loop | e_hold = 0.55 (recalibrated 2026-07, was 0.80) | Prior_map unavailable, empty holdout, or no scoreable members |
| E-GOLD | Hard iff a golden set was uploaded, else skipped | ARI vs. the uploaded golden-labels CSV | e_gold = 0.70 | No golden set uploaded for the workspace (B7) |
| E-PROBE | Hard | 3 synthetic invariance probes (duplicate rows, field-order shuffle, whitespace/casing) must all be 100% invariant | boolean | Prior_map/centroids unavailable; whitespace_casing sub-probe alone skips under USE_MOCK_EMBEDDINGS (hash-based, not semantically meaningful) |
| E-ABST | Soft | Holdout abstention rate (centroid-margin proxy) | e_abst = 0.15 | Same unavailability conditions as E-HOLD |
| E-MARG | Soft | Median holdout margin (top1 − top2 cosine) | e_marg = 0.05 | Same as E-ABST, plus “every holdout member abstained” |
| E-LAT | Soft | Live assign-call latency | e_lat_ms = 50 | Always skipped, unconditionally, by design — quoted from the gate’s own docstring: “SOFT, always SKIPPED pre-E6 … LoopInference doesn’t exist yet — measure nothing, fabricate nothing.” |
| E-SIZE | Soft | Total wm@2 artifact size | e_size_mb = 500 | No wm@2 files on the candidate |
Retrain-only gates (E7 — additionally run when training_run.kind == 'retrain')
| Gate | Hard? | Measures | Default threshold | Skipped when |
|---|---|---|---|---|
| R-IDP | Hard | % of incumbent’s own loops preserved (majority-mapped to the same loop_id in the candidate) | r_idp = 0.90 | No incumbent loops meet min_cluster, or no incumbent/wm@2 artifact to compare |
| R-CHURN | Soft | Count of loops over the churn budget | r_churn_loops = 2 (a count, not a fraction) | Incumbent has no loop membership to compare |
| R-REG | Hard | Agreement with incumbent assignments over a pinned eval set | r_reg = 0.95 | Empty pinned eval set, or no pinned member resolves in the candidate’s corpus |
| R-GOLDΔ | Hard iff a golden set exists, else skipped | ARI drop vs. incumbent on the golden set | r_gold_delta = 0.02 (max allowed drop) | No golden set; candidate’s own E-GOLD was skipped; unparseable golden CSV |
| R-OUTΔ | Soft | Relative change in abstain rate, replayed over the last 7 days | r_out_delta = 0.50 (+50% relative) | No live inference activity in the replay window; candidate artifact couldn’t be loaded |
app/v2/config_registry.py’s EvalConfigBody Pydantic field defaults.
e_sil/e_hold are evidence-calibrated floors (2026-07, see The calibration
gap); everything else is a static SPEC-v1 value. The class docstring
says plainly: “do not drift these without a controller ruling” — the 2026-07 change to
e_sil/e_hold/E-AGR’s hard/soft classification is that controller ruling, documented
in-code with the measured evidence.
E-DET: why it compares determinism_hash, not content_hash
E-DET’s job is “did retraining this exact snapshot reproduce the same computation” — twin-train
the same corpus and compare. It deliberately compares manifest.determinism_hash, never
content_hash, for two independently real reasons:
1. AES-GCM’s random nonce. wm@2’s encrypted files (svd.npz, bm25.json.zst) are
envelope-encrypted before storage. app/crypto/envelope.py::encrypt draws a fresh random
12-byte AES-GCM nonce (os.urandom(12)) on every call — mandatory for semantic security,
not a bug — so two encrypt() calls over byte-identical plaintext produce different ciphertext,
hence different sha256, every single time. Hashing that into content_hash would make a twin
determinism check fail on every real encrypted run, unconditionally — “not a flake, a
structural certainty” (app/services/model_lifecycle.py).
2. Independent trains legitimately mint fresh identity. content_hash can never be
identical across two independent train_workspace_model() calls, even without encryption, by
construction: a fresh engine_run_id/snapshot_id/ledger_watermark.as_of_run_id is minted
on every execution, and the raw ledger snapshot bytes embed fresh row id/run_id/
created_at on every append. The codebase’s own claim (from the same module) — verified
empirically across 7 processes × 7 PYTHONHASHSEEDs, plaintext and real envelope encryption
alike — is that “ALL computational content is bit-stable across independent trains of the same
corpus; the divergence is EXCLUSIVELY those identity fields.”
determinism_hash’s basis therefore excludes exactly two files by name:
ledger_snapshot.jsonl.zst (raw bytes carry the identity residue above — a normalized
ledger, reduced to (loop_id, sorted member_ids), stands in for it) and eval_report.json
(written in by this very evaluation harness after training completes, so its content
legitimately varies per eval run and isn’t part of “did training reproduce the same result”).
Every other wm@2 file’s plaintext sha256, plus the SVD/BM25/inference/config_lock/theta blocks
and the normalized ledger, form the comparison basis. content_hash keeps its original meaning
unchanged — unique-per-train registry identity — this is purely an additional, narrower hash
for the one question E-DET actually needs answered.
The calibration gap
2026-07 evidence-based calibration.
e_sil and e_hold are now measured floors, not
SPEC-v1 placeholders — see the measured good/degenerate values below. e_agr is now soft for a
different reason: not “the threshold is wrong” but “the crosscheck algorithm it reads has no
true-positive regime on real data.” e_cov/max_loop_share/E-DET/E-PROBE were never touched —
they already passed real data with margin and failed degenerate fixtures with measured teeth.leiden_dsu_agreement = 0.2988 —
worse, not better, than a second corpus’s 0.5585. Root cause: the DSU single-linkage
crosscheck arm chains transitively through any edge above theta_low, absorbing 84.8% of
the real corpus into one dominant blob — a structural property of comparing Leiden against
single-linkage DSU on real, messy, semantically-overlapping text, not a Leiden partition-quality
defect (Leiden’s own partition passes E-DEG easily in both corpora). E-AGR has never had a
true-positive regime on real or realistic data in this codebase’s history and was
unconditionally blocking every onboarding — demoted to soft; still computed, recorded, and
checked against its 0.90 advisory line. Tracked debt: the DSU crosscheck’s chaining
behavior itself is not fixed — replacing/hardening it (or comparing against a
chaining-resistant baseline) remains open follow-up work.
E-SIL / E-HOLD (recalibrated, still hard). Measured on the SAME good real-qwen partition:
E-SIL=0.3315, E-HOLD=0.6842 (a second corpus, same 250 rows, bundled all-minilm vectors:
E-SIL=0.3898, E-HOLD=0.6579) — both metrics genuinely FAILED their SPEC-v1 defaults (0.45/0.80)
on a partition that is, by every other measure (E-DEG, E-COV, human review), good. The
degenerate end was independently measured on the same real qwen artifact: a random shuffle of
the good partition’s member→loop labels (fixed seed, same loop-id set/size distribution, only
the assignment scrambled) scored E-SIL=-0.4725, E-HOLD=0.0263 — a wide, clean gap. New floors:
e_sil=0.20 (≈0.13 margin below measured-good, >0.6 margin above measured-degenerate) and
e_hold=0.55 (≈0.11 margin below measured-good, >0.5 margin above measured-degenerate).
Override still exists for genuine edge cases. A workspace-specific corpus that fails even
the recalibrated floors, or fails E-AGR’s advisory line badly enough to warrant a manual
sign-off, still uses the override path — not silently loosening a config value:
staff:admin + internal.eval_override, a non-empty reason, and permanently flags
the artifact’s manifest (manifest.eval.overridden=true, with overridden_by/overridden_at/
overridden_reason) — see Model Lifecycle → R-L2
for the full mechanics. Every override is audited. The eval config kind remains per-workspace
overridable — per-workspace overrides may TIGHTEN these floors (a client wanting a stricter QA
bar), but should not loosen below the measured-degenerate ceiling without a fresh calibration
run.
Related pages
- Model Lifecycle — R-L1/R-L2 and the full state machine
- Provisioning Portal — the RBAC gating the override endpoint