> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Inference & Queue
> The wm@2 artifact, in-process vs. hosted inference, FREE vs. PROD profiles, the Redis Streams topology, effectively-once semantics, the outbox, DLQ + replay, and the inline fallback.
**Supabase Storage, per-client Render hosting, and Upstash are configuration paths, not proven
capabilities.** None have been exercised against a live account in this environment. FREE
profile — local/shared, in-process inference — is what actually runs. See
[Config-only paths](#config-only-paths-never-exercised-against-a-live-account) below before
describing any of these as "wired up" to a client.
## The wm\@2 artifact
A trained workspace model (`app/services/model_lifecycle.py`) is a versioned manifest plus a
set of files, written through the model store. Behind `WM2_ARTIFACTS=false` (default), a
train produces the older `wm@1` manifest — 13 fixed keys, no separate files. Behind
`WM2_ARTIFACTS=true`, the manifest additionally carries:
| File | Encrypted | Contents |
| --------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| `manifest.json` | No (the manifest itself) | Every block below, plus `content_hash` and `determinism_hash` |
| `svd.npz` | Yes | Sign-pinned SVD basis, re-fit on the exact training corpus — audit/parity only; assignment stays in raw embedding space |
| `bm25.json.zst` | Yes | BM25 term-frequency model + min/max score normalization |
| `loop_stats.json` | No | Per-loop member count, radius p90, local tau |
| `centroids.json` | No | Per-loop centroid vectors in raw embedding space |
| `ledger_snapshot.jsonl.zst` | No | This run's loop-ledger rows + a derived `prior_map` (member\_id → canonical loop\_id) |
| `config_lock.json` | No | Every config kind's frozen body + content\_hash, plus the resolved theta chain, as literals |
| `ann/index.meta.json` | No | pgvector-HNSW index metadata |
Encryption is envelope encryption (`app/crypto/envelope.py`) keyed by `CAUSELOOP_MASTER_KEY`.
If the master key or `DATABASE_URL` is unavailable, encryption falls back to **plaintext with a
loud log warning** — never silent — matching the established pattern elsewhere in this
codebase (availability over confidentiality, but never a silent downgrade).
`content_hash` (unique per train, registry/lineage identity) and `determinism_hash`
(twin-train-comparable computation identity, excludes run-identity residue) are two
deliberately different hashes over deliberately different bases — see
[Evaluation Gates → E-DET](/deploy-security/evaluation-gates#e-det-why-it-compares-determinism_hash-not-content_hash)
for exactly why.
## In-process vs. hosted inference
Two independent axes control where inference runs:
**`LOOP_INFERENCE_ENABLED`** (default `false`) — when `true`, `forge_issue` (the single ingest
choke point) calls `loop_inference.assign_and_stamp` immediately after embedding, additionally
gated on `QUEUE_INGEST == "off"` (see below). This is the **in-process, synchronous** path.
**`INFERENCE_PROFILE`** (`"free"` default, or `"prod"`) — controls what a *hosted* inference
**service** (portal stage 06, `POST /provisioning/client-profiles/{id}/services`) actually
provisions:
* **`free`**: registers routing to `SHARED_INFERENCE_URL` (a shared, multi-tenant
cluster-inference service) if configured — **no Render API call at all**. If
`SHARED_INFERENCE_URL` is empty or unreachable, attach still succeeds; the service is marked
`degraded` and in-process inference keeps serving regardless (the circuit-breaker design:
attach means "register routing + a row," not "guarantee a healthy remote service").
* **`prod`**: runs the real Render create → deploy → poll → healthcheck flow
(`app/services/render_client.py`), per-client, using `RENDER_API_KEY`/`RENDER_OWNER_ID`.
## FREE vs. PROD — what's real and what's config-only
| | FREE (`INFERENCE_PROFILE=free`, default) | PROD (`INFERENCE_PROFILE=prod`) |
| -------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------ |
| Render API calls | None | Real create/deploy/poll flow — **but see the warning below** |
| Cost guard | plan=`starter`/`standard` only | plan=`pro` additionally requires `staff:admin` |
| Attach on missing shared service | Succeeds, marked `degraded`, in-process keeps serving | N/A |
| Test coverage | Exercised in this codebase's test suite | **respx-faked HTTP layer only** |
The PROD profile's Render integration is **dormant** in this environment. The code comment in
`app/services/render_client.py` says this directly: every code path is *"exercised only against
a faked HTTP layer, never a real Render API call"* — no live Render account exists here. Do not
describe per-client Render hosting as working; describe it as **implemented and tested against
a fake, never run against production infrastructure**.
## Redis Streams topology
Four streams, one real consumer group today:
| Stream | Purpose | Real consumer group |
| ------------------------ | ------------------------------------------------- | ----------------------------------------- |
| `cl:issues:ingest:v1` | New issue envelopes awaiting inference assignment | `inference` |
| `cl:issues:assigned:v1` | Assignment results | *none yet — documented future subscriber* |
| `cl:issues:outliers:v1` | Buffered/abstained assignments | *none yet — documented future subscriber* |
| `cl:models:lifecycle:v1` | Model promotion/activation events | *none yet — documented future subscriber* |
(`app/services/streaming_topology.py`) Only the ingest stream's `inference` consumer group is
implemented; the other three streams and their `platform`/`minibatch`/`drift`/`notify`
subscribers are documented as future work, not yet built.
### QUEUE\_INGEST — the two ingestion modes
```bash theme={null}
QUEUE_INGEST=off # default — inline synchronous inference, first-class and tested
QUEUE_INGEST=redis # outbox row -> relay -> stream -> consumer
```
**`off` (default):** the ingest API writes the issue directly; `forge_issue`'s inline hook does
inference synchronously in the same request, when `LOOP_INFERENCE_ENABLED=true`. Documented as
*"a first-class, tested fallback, not an afterthought."*
**`redis`:** the ingest API writes an `outbox` row instead of doing inference inline. A relay
job (`app/services/outbox_relay.py`, driven by `app/scheduler.py::register_outbox_relay_job`
every `OUTBOX_RELAY_INTERVAL_SECONDS` — default `3.0`) publishes unsent outbox rows to
`cl:issues:ingest:v1`. `app/services/inference_consumer.py` reads from there and does the
assignment.
### Effectively-once semantics — the consumer\_ledger
The outbox relay is **at-least-once**: if `relay_once` publishes successfully but then fails to
mark the row sent, the same row republishes on the next tick. This is explicitly safe by
design, because the consumer dedups before doing any work:
```python theme={null}
claim = repo.try_claim_consumer_ledger(stream, cgroup, content_hash)
if claim is None:
return "duplicate" # no-op ack
```
The dedup key is `(stream, consumer_group, content_hash)` — **not** the outbox row id or stream
message id — so a re-publish of the same logical message is silently absorbed. "At-least-once
publish + idempotent consumer = effectively-once," proven in
`tests/test_outbox_relay.py::test_crash_window_duplicate_publish_is_absorbed_downstream`.
One documented, un-closed gap: claiming the ledger row and performing the actual side effects
(embed + assign) are not one atomic unit. A crash between them permanently strands the ledger
row — redelivery of that exact `content_hash` is then silently swallowed with zero side
effects, an at-most-once window rather than effectively-once. Backstopped by two things: the
issue row itself is already written before this ever runs, and `auto_infer.py`'s debounced
pipeline re-run independently picks up anything streaming inference dropped.
### DLQ and replay
Messages dead-letter (published to `:dlq`) on: an unsupported/missing schema version, a
tokenizer-version mismatch between the artifact and the running code, or exceeding
`QUEUE_MAX_ATTEMPTS` (default `5`) via reclaim of stuck deliveries
(`QUEUE_RECLAIM_IDLE_MS`, default `60000`).
Replay is a **staff CLI**, not an automated background process:
```bash theme={null}
.venv/bin/python -m scripts.queue_replay --inspect
.venv/bin/python -m scripts.queue_replay --replay --id --yes
.venv/bin/python -m scripts.queue_replay --quarantine --id --yes
```
Dry-run by default (`--replay`/`--quarantine` without `--yes` print what would happen and
change nothing). A replay republishes to the original stream with a fresh `trace_id` but an
**unchanged** `content_hash` — the same dedup mechanism above makes "replaying an
already-processed message" a safe no-op. Every replay and quarantine writes an audit log row.
### Honest degradation — `GET /v1/provisioning/queues/health`
```python theme={null}
"""HONEST DEGRADATION ... 'a green dashboard on a dead queue is that bug in
its most dangerous form.' Returns {"enabled": false, "reason": ...} whenever
the queue plane is not actually live — QUEUE_INGEST != "redis", no
REDIS_URL configured, or a genuine Redis connection failure — and NEVER
zeros for streams/consumers in any of those cases."""
```
(`app/services/queue_health.py`) Reachability is proven with a real Redis `PING` (short
timeout), never assumed from settings alone. This endpoint is staff-gated (router-level
`require_staff()` only — a read, not a mutation) and returns per-stream `{name, length, lag,
pending, dlq_depth}` plus consumer liveness when the queue plane genuinely is up.
## Config-only paths, never exercised against a live account
Say this explicitly whenever describing storage/hosting options — do not imply any of the
following have been proven against a real account.
| Path | Status |
| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Supabase Storage** (`UPLOAD_STORE_URI=s3://...` → `SupabaseUploadStore`) | **Stub.** Every method (`save`/`open`/`delete`) raises `NotImplementedError("SupabaseUploadStore is config-only in this wave (untested-live)...")`. The default, tested upload backend is local disk (`file://./var/causeloop/uploads`). |
| **Per-client Render hosting** (`INFERENCE_PROFILE=prod`) | Real code, **respx-faked in tests only** — never run against a live Render account in this environment (see above). |
| **Upstash** | No Upstash-specific code exists anywhere in this repo. It is the presumptive hosted-Redis provider a real deployment would put behind `REDIS_URL` (a generic `redis://` connection string) — exercised only against local Redis in dev/test here. |
| **S3-compatible model store** (`MODEL_STORE_URI=s3://...` → `S3ModelStore`) | This one is genuinely real code — boto3 + `endpoint_url`, moto-tested — and *could* point at a live Supabase Storage or Cloudflare R2 S3-compatible endpoint. It has simply never been pointed at one in this environment. Don't conflate this with the upload-store stub above; they're different backends behind a shared `file://`/`s3://` convention. |
## Related pages
* [Model Lifecycle](/deploy-security/model-lifecycle) — what produces the wm\@2 artifact
* [Provisioning Portal](/deploy-security/provisioning-portal) — the hosting stage's RBAC gates
* [Runbooks](/deploy-security/runbooks) — DLQ growth, outbox-relay-down, and other operational playbooks