Supabase Storage, per-client Render hosting, and Upstash are configuration paths, not proven
capabilities. None have been exercised against a live account in this environment. FREE
profile — local/shared, in-process inference — is what actually runs. See
Config-only paths below before
describing any of these as “wired up” to a client. The wm@2 artifact
A trained workspace model (app/services/model_lifecycle.py) is a versioned manifest plus a
set of files, written through the model store. Behind WM2_ARTIFACTS=false (default), a
train produces the older wm@1 manifest — 13 fixed keys, no separate files. Behind
WM2_ARTIFACTS=true, the manifest additionally carries:
| File | Encrypted | Contents |
|---|
manifest.json | No (the manifest itself) | Every block below, plus content_hash and determinism_hash |
svd.npz | Yes | Sign-pinned SVD basis, re-fit on the exact training corpus — audit/parity only; assignment stays in raw embedding space |
bm25.json.zst | Yes | BM25 term-frequency model + min/max score normalization |
loop_stats.json | No | Per-loop member count, radius p90, local tau |
centroids.json | No | Per-loop centroid vectors in raw embedding space |
ledger_snapshot.jsonl.zst | No | This run’s loop-ledger rows + a derived prior_map (member_id → canonical loop_id) |
config_lock.json | No | Every config kind’s frozen body + content_hash, plus the resolved theta chain, as literals |
ann/index.meta.json | No | pgvector-HNSW index metadata |
app/crypto/envelope.py) keyed by CAUSELOOP_MASTER_KEY.
If the master key or DATABASE_URL is unavailable, encryption falls back to plaintext with a
loud log warning — never silent — matching the established pattern elsewhere in this
codebase (availability over confidentiality, but never a silent downgrade).
content_hash (unique per train, registry/lineage identity) and determinism_hash
(twin-train-comparable computation identity, excludes run-identity residue) are two
deliberately different hashes over deliberately different bases — see
Evaluation Gates → E-DET
for exactly why.
In-process vs. hosted inference
Two independent axes control where inference runs:
LOOP_INFERENCE_ENABLED (default false) — when true, forge_issue (the single ingest
choke point) calls loop_inference.assign_and_stamp immediately after embedding, additionally
gated on QUEUE_INGEST == "off" (see below). This is the in-process, synchronous path.
INFERENCE_PROFILE ("free" default, or "prod") — controls what a hosted inference
service (portal stage 06, POST /provisioning/client-profiles/{id}/services) actually
provisions:
free: registers routing to SHARED_INFERENCE_URL (a shared, multi-tenant
cluster-inference service) if configured — no Render API call at all. If
SHARED_INFERENCE_URL is empty or unreachable, attach still succeeds; the service is marked
degraded and in-process inference keeps serving regardless (the circuit-breaker design:
attach means “register routing + a row,” not “guarantee a healthy remote service”).prod: runs the real Render create → deploy → poll → healthcheck flow
(app/services/render_client.py), per-client, using RENDER_API_KEY/RENDER_OWNER_ID.
FREE vs. PROD — what’s real and what’s config-only
| FREE (INFERENCE_PROFILE=free, default) | PROD (INFERENCE_PROFILE=prod) |
|---|
| Render API calls | None | Real create/deploy/poll flow — but see the warning below |
| Cost guard | plan=starter/standard only | plan=pro additionally requires staff:admin |
| Attach on missing shared service | Succeeds, marked degraded, in-process keeps serving | N/A |
| Test coverage | Exercised in this codebase’s test suite | respx-faked HTTP layer only |
The PROD profile’s Render integration is dormant in this environment. The code comment in
app/services/render_client.py says this directly: every code path is “exercised only against
a faked HTTP layer, never a real Render API call” — no live Render account exists here. Do not
describe per-client Render hosting as working; describe it as implemented and tested against
a fake, never run against production infrastructure.
Redis Streams topology
Four streams, one real consumer group today:
| Stream | Purpose | Real consumer group |
|---|
cl:issues:ingest:v1 | New issue envelopes awaiting inference assignment | inference |
cl:issues:assigned:v1 | Assignment results | none yet — documented future subscriber |
cl:issues:outliers:v1 | Buffered/abstained assignments | none yet — documented future subscriber |
cl:models:lifecycle:v1 | Model promotion/activation events | none yet — documented future subscriber |
app/services/streaming_topology.py) Only the ingest stream’s inference consumer group is
implemented; the other three streams and their platform/minibatch/drift/notify
subscribers are documented as future work, not yet built.
QUEUE_INGEST — the two ingestion modes
QUEUE_INGEST=off # default — inline synchronous inference, first-class and tested
QUEUE_INGEST=redis # outbox row -> relay -> stream -> consumer
off (default): the ingest API writes the issue directly; forge_issue’s inline hook does
inference synchronously in the same request, when LOOP_INFERENCE_ENABLED=true. Documented as
“a first-class, tested fallback, not an afterthought.”
redis: the ingest API writes an outbox row instead of doing inference inline. A relay
job (app/services/outbox_relay.py, driven by app/scheduler.py::register_outbox_relay_job
every OUTBOX_RELAY_INTERVAL_SECONDS — default 3.0) publishes unsent outbox rows to
cl:issues:ingest:v1. app/services/inference_consumer.py reads from there and does the
assignment.
Effectively-once semantics — the consumer_ledger
The outbox relay is at-least-once: if relay_once publishes successfully but then fails to
mark the row sent, the same row republishes on the next tick. This is explicitly safe by
design, because the consumer dedups before doing any work:
claim = repo.try_claim_consumer_ledger(stream, cgroup, content_hash)
if claim is None:
return "duplicate" # no-op ack
(stream, consumer_group, content_hash) — not the outbox row id or stream
message id — so a re-publish of the same logical message is silently absorbed. “At-least-once
publish + idempotent consumer = effectively-once,” proven in
tests/test_outbox_relay.py::test_crash_window_duplicate_publish_is_absorbed_downstream.
One documented, un-closed gap: claiming the ledger row and performing the actual side effects
(embed + assign) are not one atomic unit. A crash between them permanently strands the ledger
row — redelivery of that exact content_hash is then silently swallowed with zero side
effects, an at-most-once window rather than effectively-once. Backstopped by two things: the
issue row itself is already written before this ever runs, and auto_infer.py’s debounced
pipeline re-run independently picks up anything streaming inference dropped.
DLQ and replay
Messages dead-letter (published to <stream>:dlq) on: an unsupported/missing schema version, a
tokenizer-version mismatch between the artifact and the running code, or exceeding
QUEUE_MAX_ATTEMPTS (default 5) via reclaim of stuck deliveries
(QUEUE_RECLAIM_IDLE_MS, default 60000).
Replay is a staff CLI, not an automated background process:
.venv/bin/python -m scripts.queue_replay --inspect
.venv/bin/python -m scripts.queue_replay --replay --id <entry-id> --yes
.venv/bin/python -m scripts.queue_replay --quarantine --id <entry-id> --yes
--replay/--quarantine without --yes print what would happen and
change nothing). A replay republishes to the original stream with a fresh trace_id but an
unchanged content_hash — the same dedup mechanism above makes “replaying an
already-processed message” a safe no-op. Every replay and quarantine writes an audit log row.
Honest degradation — GET /v1/provisioning/queues/health
"""HONEST DEGRADATION ... 'a green dashboard on a dead queue is that bug in
its most dangerous form.' Returns {"enabled": false, "reason": ...} whenever
the queue plane is not actually live — QUEUE_INGEST != "redis", no
REDIS_URL configured, or a genuine Redis connection failure — and NEVER
zeros for streams/consumers in any of those cases."""
app/services/queue_health.py) Reachability is proven with a real Redis PING (short
timeout), never assumed from settings alone. This endpoint is staff-gated (router-level
require_staff() only — a read, not a mutation) and returns per-stream {name, length, lag, pending, dlq_depth} plus consumer liveness when the queue plane genuinely is up.
Config-only paths, never exercised against a live account
Say this explicitly whenever describing storage/hosting options — do not imply any of the
following have been proven against a real account.
| Path | Status |
|---|
Supabase Storage (UPLOAD_STORE_URI=s3://... → SupabaseUploadStore) | Stub. Every method (save/open/delete) raises NotImplementedError("SupabaseUploadStore is config-only in this wave (untested-live)..."). The default, tested upload backend is local disk (file://./var/causeloop/uploads). |
Per-client Render hosting (INFERENCE_PROFILE=prod) | Real code, respx-faked in tests only — never run against a live Render account in this environment (see above). |
| Upstash | No Upstash-specific code exists anywhere in this repo. It is the presumptive hosted-Redis provider a real deployment would put behind REDIS_URL (a generic redis:// connection string) — exercised only against local Redis in dev/test here. |
S3-compatible model store (MODEL_STORE_URI=s3://... → S3ModelStore) | This one is genuinely real code — boto3 + endpoint_url, moto-tested — and could point at a live Supabase Storage or Cloudflare R2 S3-compatible endpoint. It has simply never been pointed at one in this environment. Don’t conflate this with the upload-store stub above; they’re different backends behind a shared file:///s3:// convention. |
Related pages
