Skip to main content
These runbooks live in the backend repository at docs/runbooks/ (not in this docs site’s source tree) — this page indexes and summarizes them rather than duplicating their content. Open the linked file in the repo for the full diagnostic steps, exact commands, and code references.
Each runbook below corresponds to a row in docs/SPEC-model-lifecycle-and-streaming-inference.md §11’s failure-mode table, and documents the real symptom, cause, and remediation as implemented in this codebase — including a few places where the runbook itself corrects or narrows what the spec’s one-line table entry originally said.

​
Model lifecycle

A candidate model version has sat in phase='shadow' past the portal’s 7-day badge threshold. The incumbent keeps serving 100% of traffic with full side effects the entire time — shadow mode is record-only, so this is never customer-visible by itself, but it blocks that workspace’s next retrain/promotion cycle. No automatic expiry exists; a staff user must force a promote/reject decision.docs/runbooks/candidate-wedged-in-shadow.md
A canary serving real traffic was automatically flipped to traffic_pct=0 by promotion.maybe_auto_pause — a live check the inference consumer runs as a natural consequence of processing canary-arm messages, not a separate polling job. The incumbent is already back at 100%; this runbook is about deciding rollback vs. fix-forward, not about stopping an active regression (auto-pause already did that).docs/runbooks/canary-regression-auto-pause.md
Portal stage 05 (Evaluate) never resolves — eval_runs.status='error', distinct from passed/failed (legitimate gate outcomes with real eval_gate_results rows). This is an evaluation-harness failure, not a quality-gate failure; the training run’s artifact is untouched.docs/runbooks/eval-infra-error.md

​
Streaming inference

Issues ingest successfully (the outbox write happens alongside the issue write), but nothing reaches cl:issues:assigned:v1/cl:issues:outliers:v1, and the ingest stream itself isn’t growing — because app/services/outbox_relay.py::relay_once isn’t running. Restarting the relay resumes from unsent rows; nothing is lost.docs/runbooks/outbox-relay-down.md
dlq_depth > 100 or dlq_age_seconds > 3600 alert fires. Inspect with scripts/queue_replay.py --inspect; schema errors get a contract fix and a replay; poison messages get quarantined instead. See Inference & Queue → DLQ and replay for the replay mechanics.docs/runbooks/dlq-growth.md
After a code deploy, the consumer refuses messages whose tokenizer version doesn’t match the artifact’s — DLQ’d with the precise reason tokenizer_skew (the actual code label; more specific than the spec table’s generic schema_unsupported). Fix: ship an artifact retrained on the new tokenizer, or roll back the deploy.docs/runbooks/tokenizer-skew.md
The general at-least-once redelivery case is safe by design — a redelivered message that already has a consumer_ledger row is absorbed as a no-op. This runbook documents a narrower, real gap found in a later fix wave: a crash between claiming the ledger row and performing the actual side effects can permanently strand that claim, silently swallowing redelivery of that exact message. Read this before assuming “none needed” (the spec table’s original one-line verdict) is the whole story.docs/runbooks/consumer-ledger-ack-crash-window.md
Not a literal SPEC §11 row — covers “no active artifact” and “artifact load error/hash mismatch” failure modes plus the fallback_rate alert. A high fallback rate is often expected (a brand-new workspace with no trained model yet) rather than an incident; this runbook’s job is telling the two cases apart.docs/runbooks/cold-start-and-fallback.md