> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluation Gates
> Every E-* and R-* gate the model evaluation harness runs: what it measures, its default threshold, hard vs. soft, when it's skipped — and the honest calibration gap in those defaults.
**2026-07 calibration.** `e_sil` and `e_hold` are now evidence-calibrated (measured against a
real 250-row corpus embedded with the production embedder, qwen3-embedding:0.6b) — see
[The calibration gap](#the-calibration-gap) below for the good-vs-degenerate measurements that
justify the new floors. **E-AGR is now a SOFT gate**, not hard: the DSU single-linkage
crosscheck it reads chains real, messy text into one dominant blob (measured 84.8% of a
250-row corpus in one component) and has no true-positive regime on real data — it never once
passed on non-trivial data in this codebase's history. E-AGR is still computed, recorded, and
checked against its `0.90` advisory line; it just no longer blocks. `e_cov`/`max_loop_share`/
E-DET/E-PROBE are unchanged — they already pass real data with margin and fail degenerate
fixtures with measured teeth.
The evaluation harness (`app/services/model_eval.py`, portal stage 05) is the mandatory quality
gate between a trained candidate and a client's production traffic — see
[Model Lifecycle](/deploy-security/model-lifecycle) for how `approve`/`override`/`reject`
compose with it. Every gate runs against **frozen** inputs — the candidate's wm\@2 manifest and
artifact files, plus pinned datasets — never live tables, so a completed evaluation is
reproducible and replayable.
## Honesty rule: skipped is not passed
This is the load-bearing design decision on this page. Quoted directly from the code's own
module docstring: *"a gate that cannot be computed honestly is recorded SKIPPED —
`measured.skipped=true`, `passed=False` (never fabricated True) — and does NOT block."*
```python theme={null}
def _aggregate_status(gates: List[Dict]) -> str:
"""'failed' iff at least one HARD, NON-SKIPPED gate failed. Soft gates
and skipped gates (hard or soft) never block."""
```
A skipped gate always reports `passed: false` with `measured: {"skipped": true, "reason":
"..."}` — it is never silently counted as a pass, and (per `_aggregate_status`) it never blocks
the overall result either. Only a genuinely **computed and failed hard gate** fails the
evaluation. The most common skip reason across this table is `WM2_ARTIFACTS` having been off at
train time — most gates read wm\@2-only files (`loop_stats.json`, `centroids.json`) that simply
don't exist under the older wm\@1 manifest shape.
## Initial-training gates (always run)
| Gate | Hard? | Measures | Default threshold | Skipped when |
| --------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **E-DET** | Hard (soft above 500k records) | Twin-train `determinism_hash` equality | exact match | No wm\@2 `determinism_hash` on the candidate (`WM2_ARTIFACTS` was off) |
| E-COV | Hard | % of corpus covered by a loop ≥ `min_cluster` (3) members | `e_cov = 0.85` | `loop_stats.json` unavailable, or no `record_count` |
| E-SIL | Hard | Centroid-approximated mean silhouette | `e_sil = 0.20` (recalibrated 2026-07, was `0.45`) | `centroids.json`/prior\_map unavailable, or fewer than 2 loops |
| E-DEG | Hard | Degenerate partition: max loop share ≤ 0.40 AND loop count in a corpus-relative range | `max_loop_share = 0.40` | `loop_stats.json` unavailable |
| E-AGR | **Soft** (demoted 2026-07 — was hard) | `leiden_dsu_agreement`, read straight off the candidate's own recorded training stats | `e_agr = 0.90` (advisory line, value unchanged) | No `stability.leiden_dsu_agreement` recorded on the candidate |
| E-HOLD | Hard | Held-out issue's 5-nearest-neighbor majority vote matches its own assigned loop | `e_hold = 0.55` (recalibrated 2026-07, was `0.80`) | Prior\_map unavailable, empty holdout, or no scoreable members |
| E-GOLD | Hard **iff** a golden set was uploaded, else skipped | ARI vs. the uploaded golden-labels CSV | `e_gold = 0.70` | No golden set uploaded for the workspace (B7) |
| E-PROBE | Hard | 3 synthetic invariance probes (duplicate rows, field-order shuffle, whitespace/casing) must all be 100% invariant | boolean | Prior\_map/centroids unavailable; `whitespace_casing` sub-probe alone skips under `USE_MOCK_EMBEDDINGS` (hash-based, not semantically meaningful) |
| E-ABST | Soft | Holdout abstention rate (centroid-margin proxy) | `e_abst = 0.15` | Same unavailability conditions as E-HOLD |
| E-MARG | Soft | Median holdout margin (top1 − top2 cosine) | `e_marg = 0.05` | Same as E-ABST, plus "every holdout member abstained" |
| E-LAT | Soft | Live assign-call latency | `e_lat_ms = 50` | **Always skipped, unconditionally, by design** — quoted from the gate's own docstring: *"SOFT, always SKIPPED pre-E6 ... LoopInference doesn't exist yet — measure nothing, fabricate nothing."* |
| E-SIZE | Soft | Total wm\@2 artifact size | `e_size_mb = 500` | No wm\@2 files on the candidate |
## Retrain-only gates (E7 — additionally run when `training_run.kind == 'retrain'`)
| Gate | Hard? | Measures | Default threshold | Skipped when |
| ------- | ---------------------------------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------- |
| R-IDP | Hard | % of incumbent's own loops preserved (majority-mapped to the same loop\_id in the candidate) | `r_idp = 0.90` | No incumbent loops meet `min_cluster`, or no incumbent/wm\@2 artifact to compare |
| R-CHURN | Soft | Count of loops over the churn budget | `r_churn_loops = 2` (a count, not a fraction) | Incumbent has no loop membership to compare |
| R-REG | Hard | Agreement with incumbent assignments over a pinned eval set | `r_reg = 0.95` | Empty pinned eval set, or no pinned member resolves in the candidate's corpus |
| R-GOLDΔ | Hard **iff** a golden set exists, else skipped | ARI drop vs. incumbent on the golden set | `r_gold_delta = 0.02` (max allowed drop) | No golden set; candidate's own E-GOLD was skipped; unparseable golden CSV |
| R-OUTΔ | Soft | Relative change in abstain rate, replayed over the last 7 days | `r_out_delta = 0.50` (+50% relative) | No live inference activity in the replay window; candidate artifact couldn't be loaded |
All defaults above are `app/v2/config_registry.py`'s `EvalConfigBody` Pydantic field defaults.
`e_sil`/`e_hold` are evidence-calibrated floors (2026-07, see [The calibration
gap](#the-calibration-gap)); everything else is a static SPEC-v1 value. The class docstring
says plainly: *"do not drift these without a controller ruling"* — the 2026-07 change to
`e_sil`/`e_hold`/E-AGR's hard/soft classification **is** that controller ruling, documented
in-code with the measured evidence.
## E-DET: why it compares `determinism_hash`, not `content_hash`
E-DET's job is "did retraining this exact snapshot reproduce the same computation" — twin-train
the same corpus and compare. It deliberately compares `manifest.determinism_hash`, **never**
`content_hash`, for two independently real reasons:
**1. AES-GCM's random nonce.** wm\@2's encrypted files (`svd.npz`, `bm25.json.zst`) are
envelope-encrypted before storage. `app/crypto/envelope.py::encrypt` draws a fresh random
12-byte AES-GCM nonce (`os.urandom(12)`) on **every** call — mandatory for semantic security,
not a bug — so two `encrypt()` calls over byte-identical plaintext produce different ciphertext,
hence different sha256, every single time. Hashing that into `content_hash` would make a twin
determinism check fail on every real encrypted run, unconditionally — "not a flake, a
structural certainty" (`app/services/model_lifecycle.py`).
**2. Independent trains legitimately mint fresh identity.** `content_hash` can never be
identical across two independent `train_workspace_model()` calls, even without encryption, by
construction: a fresh `engine_run_id`/`snapshot_id`/`ledger_watermark.as_of_run_id` is minted
on every execution, and the raw ledger snapshot bytes embed fresh row `id`/`run_id`/
`created_at` on every append. The codebase's own claim (from the same module) — verified
empirically across 7 processes × 7 `PYTHONHASHSEED`s, plaintext and real envelope encryption
alike — is that *"ALL computational content is bit-stable across independent trains of the same
corpus; the divergence is EXCLUSIVELY those identity fields."*
`determinism_hash`'s basis therefore excludes exactly two files by name:
`ledger_snapshot.jsonl.zst` (raw bytes carry the identity residue above — a **normalized**
ledger, reduced to `(loop_id, sorted member_ids)`, stands in for it) and `eval_report.json`
(written in by this very evaluation harness *after* training completes, so its content
legitimately varies per eval run and isn't part of "did training reproduce the same result").
Every other wm\@2 file's plaintext sha256, plus the SVD/BM25/inference/config\_lock/theta blocks
and the normalized ledger, form the comparison basis. `content_hash` keeps its original meaning
unchanged — unique-per-train registry identity — this is purely an additional, narrower hash
for the one question E-DET actually needs answered.
## The calibration gap
**2026-07 evidence-based calibration.** `e_sil` and `e_hold` are now measured floors, not
SPEC-v1 placeholders — see the measured good/degenerate values below. `e_agr` is now soft for a
different reason: not "the threshold is wrong" but "the crosscheck algorithm it reads has no
true-positive regime on real data." `e_cov`/`max_loop_share`/E-DET/E-PROBE were never touched —
they already passed real data with margin and failed degenerate fixtures with measured teeth.
**E-AGR (soft).** A full-stack drill against a real 250-row corpus, embedded with the production
embedder (qwen3-embedding:0.6b), on a GOOD, human-plausible Leiden partition (33 balanced loops,
max share 5.6% — E-DEG passes with a 7x margin) measured `leiden_dsu_agreement = 0.2988` —
*worse*, not better, than a second corpus's `0.5585`. Root cause: the DSU single-linkage
crosscheck arm chains transitively through any edge above `theta_low`, absorbing **84.8%** of
the real corpus into one dominant blob — a structural property of comparing Leiden against
single-linkage DSU on real, messy, semantically-overlapping text, not a Leiden partition-quality
defect (Leiden's own partition passes E-DEG easily in both corpora). E-AGR has never had a
true-positive regime on real or realistic data in this codebase's history and was
unconditionally blocking every onboarding — demoted to soft; still computed, recorded, and
checked against its `0.90` advisory line. **Tracked debt:** the DSU crosscheck's chaining
behavior itself is not fixed — replacing/hardening it (or comparing against a
chaining-resistant baseline) remains open follow-up work.
**E-SIL / E-HOLD (recalibrated, still hard).** Measured on the SAME good real-qwen partition:
E-SIL=0.3315, E-HOLD=0.6842 (a second corpus, same 250 rows, bundled all-minilm vectors:
E-SIL=0.3898, E-HOLD=0.6579) — both metrics genuinely FAILED their SPEC-v1 defaults (0.45/0.80)
on a partition that is, by every other measure (E-DEG, E-COV, human review), good. The
degenerate end was independently measured on the same real qwen artifact: a random shuffle of
the good partition's member→loop labels (fixed seed, same loop-id set/size distribution, only
the assignment scrambled) scored E-SIL=-0.4725, E-HOLD=0.0263 — a wide, clean gap. New floors:
`e_sil=0.20` (≈0.13 margin below measured-good, >0.6 margin above measured-degenerate) and
`e_hold=0.55` (≈0.11 margin below measured-good, >0.5 margin above measured-degenerate).
**Override still exists for genuine edge cases.** A workspace-specific corpus that fails even
the recalibrated floors, or fails E-AGR's advisory line badly enough to warrant a manual
sign-off, still uses the **override** path — not silently loosening a config value:
```http theme={null}
POST /v1/provisioning/training-runs/{run_id}/override
Authorization: Bearer
{"reason": "e_sil scored 0.18 on a genuinely sparse pilot corpus; SRE reviewed loop outputs manually and signed off."}
```
Requires `staff:admin` + `internal.eval_override`, a non-empty reason, and permanently flags
the artifact's manifest (`manifest.eval.overridden=true`, with `overridden_by`/`overridden_at`/
`overridden_reason`) — see [Model Lifecycle → R-L2](/deploy-security/model-lifecycle#r-l2--override-requires-staffadmin-a-reason-and-the-artifact-is-flagged-forever)
for the full mechanics. Every override is audited. The `eval` config kind remains per-workspace
overridable — per-workspace overrides may TIGHTEN these floors (a client wanting a stricter QA
bar), but should not loosen below the measured-degenerate ceiling without a fresh calibration
run.
## Related pages
* [Model Lifecycle](/deploy-security/model-lifecycle) — R-L1/R-L2 and the full state machine
* [Provisioning Portal](/deploy-security/provisioning-portal) — the RBAC gating the override endpoint