> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Operational Runbooks

> Index of the model-lifecycle and streaming-inference operational runbooks — symptom, cause, and fix for each SPEC §11 failure mode.

<Note>
  These runbooks live in the backend repository at `docs/runbooks/` (not in this docs site's
  source tree) — this page indexes and summarizes them rather than duplicating their content.
  Open the linked file in the repo for the full diagnostic steps, exact commands, and code
  references.
</Note>

Each runbook below corresponds to a row in `docs/SPEC-model-lifecycle-and-streaming-inference.md`
§11's failure-mode table, and documents the real symptom, cause, and remediation as implemented
in this codebase — including a few places where the runbook itself corrects or narrows what the
spec's one-line table entry originally said.

## Model lifecycle

<AccordionGroup>
  <Accordion title="Candidate wedged in shadow">
    A candidate model version has sat in `phase='shadow'` past the portal's 7-day badge
    threshold. The incumbent keeps serving 100% of traffic with full side effects the entire
    time — shadow mode is record-only, so this is never customer-visible by itself, but it
    blocks that workspace's next retrain/promotion cycle. No automatic expiry exists; a staff
    user must force a promote/reject decision.

    `docs/runbooks/candidate-wedged-in-shadow.md`
  </Accordion>

  <Accordion title="Canary regression (auto-pause)">
    A canary serving real traffic was automatically flipped to `traffic_pct=0` by
    `promotion.maybe_auto_pause` — a live check the inference consumer runs as a natural
    consequence of processing canary-arm messages, not a separate polling job. The incumbent is
    already back at 100%; this runbook is about deciding rollback vs. fix-forward, not about
    stopping an active regression (auto-pause already did that).

    `docs/runbooks/canary-regression-auto-pause.md`
  </Accordion>

  <Accordion title="Eval infra error (not a gate failure)">
    Portal stage 05 (Evaluate) never resolves — `eval_runs.status='error'`, distinct from
    `passed`/`failed` (legitimate gate outcomes with real `eval_gate_results` rows). This is an
    evaluation-harness failure, not a quality-gate failure; the training run's artifact is
    untouched.

    `docs/runbooks/eval-infra-error.md`
  </Accordion>
</AccordionGroup>

## Streaming inference

<AccordionGroup>
  <Accordion title="Outbox relay down">
    Issues ingest successfully (the outbox write happens alongside the issue write), but
    nothing reaches `cl:issues:assigned:v1`/`cl:issues:outliers:v1`, and the ingest stream
    itself isn't growing — because `app/services/outbox_relay.py::relay_once` isn't running.
    Restarting the relay resumes from unsent rows; nothing is lost.

    `docs/runbooks/outbox-relay-down.md`
  </Accordion>

  <Accordion title="DLQ growth">
    `dlq_depth > 100` or `dlq_age_seconds > 3600` alert fires. Inspect with
    `scripts/queue_replay.py --inspect`; schema errors get a contract fix and a replay; poison
    messages get quarantined instead. See
    [Inference & Queue → DLQ and replay](/deploy-security/inference-and-queue#dlq-and-replay)
    for the replay mechanics.

    `docs/runbooks/dlq-growth.md`
  </Accordion>

  <Accordion title="Tokenizer/artifact version skew after deploy">
    After a code deploy, the consumer refuses messages whose tokenizer version doesn't match
    the artifact's — DLQ'd with the precise reason `tokenizer_skew` (the actual code label;
    more specific than the spec table's generic `schema_unsupported`). Fix: ship an artifact
    retrained on the new tokenizer, or roll back the deploy.

    `docs/runbooks/tokenizer-skew.md`
  </Accordion>

  <Accordion title="Consumer-ledger vs. ack crash window">
    The general at-least-once redelivery case is safe by design — a redelivered message that
    already has a `consumer_ledger` row is absorbed as a no-op. This runbook documents a
    **narrower, real gap** found in a later fix wave: a crash between claiming the ledger row
    and performing the actual side effects can permanently strand that claim, silently
    swallowing redelivery of that exact message. Read this before assuming "none needed" (the
    spec table's original one-line verdict) is the whole story.

    `docs/runbooks/consumer-ledger-ack-crash-window.md`
  </Accordion>

  <Accordion title="Cold start / inference fallback">
    Not a literal SPEC §11 row — covers "no active artifact" and "artifact load error/hash
    mismatch" failure modes plus the `fallback_rate` alert. A high fallback rate is often
    *expected* (a brand-new workspace with no trained model yet) rather than an incident; this
    runbook's job is telling the two cases apart.

    `docs/runbooks/cold-start-and-fallback.md`
  </Accordion>
</AccordionGroup>

## Related pages

* [Evaluation Gates](/deploy-security/evaluation-gates) — the gate suite these lifecycle runbooks reference
* [Inference & Queue](/deploy-security/inference-and-queue) — the queue topology these streaming runbooks reference
* [Observability](/deploy-security/observability) — metrics and alerting these runbooks are triggered from