> ## Documentation Index > Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt > Use this file to discover all available pages before exploring further. # Train and Infer Locally with Qwen > Run the full ingest → train → infer lifecycle end-to-end on your own machine, with zero hosted API calls — Ollama, Qwen3, and a real 250-issue dataset. Every command, response, and number on this page was captured from a real run against a local Ollama instance and the real Causeloop v2 API — nothing here is simulated. Token values are redacted; everything else is verbatim. This guide runs Causeloop's full lifecycle — ingest historical issues, calibrate the engine ("training"), then read live inference outputs (patterns, hazard forecasts, predictions, recommendations) — using **only local models** served by [Ollama](https://ollama.com). No Anthropic, OpenAI, or other hosted API key is used anywhere in this walkthrough. ## Concept: what "training" means in Causeloop Causeloop v2 deliberately has **no trainable model endpoint** — there is no `/engine/model/train`, no gradient descent, no model weights you fine-tune. If you're looking for that, it doesn't exist here on purpose (see [Concepts → Provenance](/get-started/concepts#provenance)). "Training" in Causeloop is the **calibration lifecycle**: a deterministic pipeline that turns raw historical issues into a calibrated, queryable model of your organization's recurring failures: 1. **Ingest** historical issues into the event log. 2. **Embed** the corpus (Qwen embeddings, in this guide). 3. **Run the engine** — theta calibration, Leiden graph partitioning, term naming, criticality scoring, and hazard (recurrence) fitting, all in one deterministic run. 4. **Review** any abstentions — low-confidence extractions the model declined to guess on, routed to a human via the review queue. 5. **Activate** — the resulting patterns, hazard fits, and predictions are live and queryable immediately; nothing further to "deploy." Every one of the five steps below is a real API call against this pipeline. Nothing is precomputed or faked for the walkthrough. ## Prerequisites ```bash theme={null} brew install ollama # macOS; see ollama.com for other platforms ollama serve # if not already running as a service ``` ```bash theme={null} ollama pull qwen3-embedding:0.6b # embeddings — 1024-dim ollama pull qwen3:4b # extraction, classification, narration ``` Verify both are present: ```bash theme={null} curl -s http://localhost:11434/api/tags | python3 -c \ "import json,sys; print([m['name'] for m in json.load(sys.stdin)['models']])" ``` ```bash theme={null} git clone https://github.com/your-org/causeloop-backend.git cd causeloop-backend python3 -m venv .venv .venv/bin/pip install -r requirements.txt ``` ### Hardware notes This guide was run on a single developer machine (no GPU cluster, no dedicated inference server) with Ollama serving both models. Two things to know before you start: * **qwen3:4b "thinks" by default — and Causeloop turns it off.** Qwen3's chat template enables a hidden `...` reasoning block before the visible answer, for every request. Measured on this machine: a 5-field JSON extraction burned **\~1,850 completion tokens / \~44 seconds** with thinking on, versus **103 tokens / 2.6 seconds** with thinking off and grammar-constrained JSON — a \~17x difference. Causeloop's Ollama provider disables thinking by default (`OLLAMA_THINK=false`); see [Production notes](#production-notes) for the full story and the quality tradeoff. * **Embeddings are cheap.** `qwen3-embedding:0.6b` returns a 1024-dim vector in well under a second per call — the LLM calls (extraction, classification, narration) dominate the cost, not the embedding calls. With thinking off, the full 250-issue dataset processes end-to-end on a laptop in tens of minutes — every number below is from that full-dataset run. ## Step 1 — Configure the local Qwen profile Start the backend with `DATABASE_URL` **unset** — this guide uses the in-memory store, which has no vector-dimension constraint. This matters: the repo's pgvector column is fixed at `vector(384)` (`db/migrations/0009_embeddings_pgvector.sql`), but `qwen3-embedding:0.6b` returns **1024-dim** vectors (verified below). Running against Postgres today would fail at the vector column; running in-memory sidesteps it entirely. See [Production notes](#production-notes) for the real options if you need Postgres persistence. ```bash theme={null} ANTHROPIC_API_KEY= OPENAI_API_KEY= DATABASE_URL= \ ENVIRONMENT=development \ AUTH_PROVIDER=none \ PORT=8001 \ EMBEDDINGS_PROVIDER=ollama \ EMBEDDINGS_MODEL=qwen3-embedding:0.6b \ EMBEDDINGS_BASE_URL=http://localhost:11434/v1 \ EMBEDDINGS_DIM=1024 \ USE_MOCK_EMBEDDINGS=false \ LLM_PROVIDER_ORDER=ollama \ OLLAMA_BASE_URL=http://localhost:11434 \ OLLAMA_MODEL=qwen3:4b \ USE_MOCK_LLM=false \ .venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8001 ``` The Ollama LLM provider also honors three tuning knobs, all defaulted for the local profile: `OLLAMA_THINK=false` (disables qwen3's hidden reasoning — the single biggest local-throughput lever, see [Production notes](#production-notes)), `OLLAMA_TEMPERATURE=0.0` and `OLLAMA_SEED=42` (deterministic decoding for a fixed prompt on a fixed model build). `LLM_PROVIDER_ORDER=ollama` and the `OLLAMA_*` settings require backend fixes made alongside this guide — see [Real bugs found and fixed](#real-bugs-found-and-fixed-while-writing-this-guide) below. On older builds this profile silently degrades every LLM call to the mock provider with no error. Mint a scoped dev token (no external identity provider needed with `AUTH_PROVIDER=none`): ```bash theme={null} TOKEN=$(.venv/bin/python -m scripts.mint_dev_token ws_01 usr_01) ``` Confirm the local providers actually resolved: ```bash theme={null} curl -s http://localhost:8001/v1/ai/providers -H "Authorization: Bearer $TOKEN" ``` ```json theme={null} { "providers": [ { "name": "ollama", "available": true, "models": { "reasoning": "qwen3:4b", "balanced": "qwen3:4b", "fast": "qwen3:4b" } }, { "name": "mock", "available": true, "models": { "reasoning": "mock-model", "balanced": "mock-model", "fast": "mock-model" } } ], "order": ["ollama", "mock"] } ``` `ollama` is first in `order` and `available: true` — every real LLM call in this run went to qwen3:4b, with `mock` only as the never-triggered safety net. Confirm the embedding dimension while you're at it — this is the number that has to match `EMBEDDINGS_DIM` and (in a Postgres deployment) the pgvector column: ```python theme={null} from app.ml.embeddings.ollama_provider import OllamaEmbeddingProvider p = OllamaEmbeddingProvider(base_url="http://localhost:11434/v1", model="qwen3-embedding:0.6b", dim=1024) vecs = await p.embed_batch(["hello world"]) len(vecs[0]) # -> 1024 ``` ## Step 2 — Convert and ingest the dataset The source dataset is `issues.xlsx` — a `Sheet1` with 250 rows and three columns: `Issue ID`, `Unstructured Text`, `issue_created`. Convert each row into the v2 ingest contract (`external_id`, `title`, `narrative`, `occurred_at`), following the same title convention as `scripts/convert_issue_set_2.py` — first sentence, truncated to 120 chars: ```python theme={null} def _title_from_text(text: str) -> str: c = (text or "").strip() first = c.split(". ")[0].strip() base = first if 0 < len(first) <= 120 else c return (base[:117] + "...") if len(base) > 120 else base record = { "external_id": str(row["Issue ID"]), "title": _title_from_text(row["Unstructured Text"]), "narrative": row["Unstructured Text"], "occurred_at": row["issue_created"].isoformat() + "Z", "source": "custom_api", } ``` Batch-ingest all 250 in five batches of 50, each with its own `Idempotency-Key`: ```python theme={null} for i in range(0, len(records), 50): batch = records[i:i + 50] requests.post(f"{BASE}/ingest/batch", json={"records": batch}, headers={"Authorization": f"Bearer {TOKEN}", "Idempotency-Key": str(uuid.uuid4())}) ``` **Real output** — all five batches, full 250-row file: ``` batch 0-50: accepted=50 rejected=0 dt=0.09s batch 50-100: accepted=50 rejected=0 dt=0.20s batch 100-150: accepted=50 rejected=0 dt=0.65s batch 150-200: accepted=50 rejected=0 dt=1.18s batch 200-250: accepted=50 rejected=0 dt=0.82s TOTAL accepted=250 rejected=0 wall=2.94s engine_status_summary: {"eligible": 250, "blocked_no_narrative": 0, "ineligible": 0} ``` `POST /ingest/batch` itself only validates and persists — it does not call any model synchronously. Every **engine-eligible** issue (one with a non-empty narrative) then automatically kicks off, in the background: 1. **Forge** — a 5C extraction call (`concern`/`context`/`consequence`/`cause`/`cap`) that produces the text the pattern-clustering embedding is built from, plus the embedding call itself. 2. **Conformal extraction** — two classification calls (`process_cadence`, `customer_harm`) that feed criticality/financials, each scoring every candidate label. That's up to **3 LLM calls per issue** — \~750 calls for the full file. With thinking disabled (the default), each call completes in a few seconds on this laptop; with qwen3's default thinking mode it would be \~44–50 seconds per call (\~9+ hours total — this is why `OLLAMA_THINK=false` is the default, see [Production notes](#production-notes)). ### Watching forge + extraction run Each accepted issue schedules its forge and extraction jobs as fire-and-forget background tasks — the ingest response returns immediately. Poll an issue to watch its `concern`/`cause`/`consequence` fields populate as forge completes: ```bash theme={null} curl -s "http://localhost:8001/v1/issues/$ISSUE_ID" -H "Authorization: Bearer $TOKEN" \ | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('concern'), d.get('cause'))" ``` **Real 5C extraction output** for one issue (`AI-AUDIT-001`, narrative: *"Just found out the dev team in London has been pasting snippets of our proprietary trading algorithms into public ChatGPT to debug them..."*): ```json theme={null} { "concern": "Exposure of proprietary trading algorithms to public via ChatGPT", "context": "Dev team in London has been pasting snippets of proprietary trading algorithms into public ChatGPT for debugging", "consequence": "Potential security breaches, loss of competitive advantage, and intellectual property compromise", "cause": "Internal debugging tools are perceived as too slow by the dev team", "cap": "Not specified in narrative" } ``` **Real timing, full 250-issue run on this laptop:** all 250 forge jobs (one 5C extraction + one embedding each) completed in **\~18 minutes**; the classification tail (two conformal scoring calls per issue) fully drained at **\~62 minutes** after ingest. That's \~750 qwen3:4b calls plus 250 embedding calls through a 4-deep client-side queue — roughly 5s per LLM call end-to-end — with **0 failed provider calls**. (With qwen3's default thinking mode this same workload measured out to \~9 hours; see [Production notes](#production-notes).) ## Step 3 — Run the engine ("train") Once forge has completed, trigger a full engine run: ```bash theme={null} curl -s -X POST http://localhost:8001/v1/engine/runs \ -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" \ -d '{"scope": {}, "trigger": "manual", "mode": "full"}' ``` ```json theme={null} {"run_id": "run_01KX05AZ9JVH7JDNMEKM2PP9ZH", "status": "queued", "scope_hash": "70c3a3b523ef2dc4a2a58929007de8181c4ad6a1c3a9f4e8ad780a0c88b7af31", "mode": "full", "mode_effective": "full", "mode_degraded": false} ``` Poll `GET /v1/engine/runs/{run_id}` until `status` is terminal. The run itself is fast — every LLM-heavy step already happened at ingest time, so the engine stages are pure deterministic computation over stored embeddings and events. **Real result** (250 issues, total stage time under one second): ``` status: succeeded counts: themes=9 loops=37 edges=870 partition_degenerate=false feature_forge succeeded 11ms graph_weave succeeded 608ms {edge_count: 870} leiden_partition succeeded 11ms {theme_count: 9, loop_count: 37} dsu_crosscheck succeeded 12ms {dsu_theme_count: 7, dsu_loop_count: 25, flagged_count: 0} loop_anchor succeeded 29ms {anchored: 37, inherited: 0, minted: 37} term_namer succeeded 37ms {versions_frozen: 37} taxonomy_label succeeded 0ms {labeled: 37, taxonomy_matched: 0} criticality_rollup succeeded 23ms {patterns_projected: 37, member_links: 250, rolled_up: 37} hazard_fit succeeded 32ms {patterns_fitted: 37, patterns_skipped: 0} alert_eval succeeded 1ms {rules_evaluated: 0} manifest_hash: sha256:e09bf1349bf30bbc2396f1cb75b7abe2fe6032c2f0391b0fa56db8175de07f9b ``` `partition_degenerate: false` is the signal that the Qwen embeddings produced real cluster structure (a broken embedding path shows up here as "100% singletons"). Note `criticality_rollup` projecting all 37 loops onto pattern rows and `hazard_fit` fitting all 37 — both were broken before the fixes shipped with this guide (see [bugs below](#real-bugs-found-and-fixed-while-writing-this-guide)). ## Step 4 — Inference: read the trained model The same run's outputs are queryable immediately — "inference" here just means reading patterns, forecasts, and recommendations back out of the run you just calibrated. ### Patterns ```bash theme={null} curl -s "http://localhost:8001/v1/patterns?limit=50" -H "Authorization: Bearer $TOKEN" ``` **37 patterns**, named from each cluster's BM25 top terms over real member narratives. A sample of what qwen3-embedding:0.6b's clusters actually look like on this dataset: ``` pat_01KX05AZZXS7E4VMQP6JVANTBX "compliance suspicious alerts monitoring operations" 11 issues pat_01KX05B000DV9H6RBFBNPDHA5C "database master certificate expired configuration" 10 issues pat_01KX05B00416PHBKXHAJAHECXC "tapes cleartext unencrypted backup party" 9 issues pat_01KX05B003H8Z7K1VF4JSCP5KP "model complex analysts defaults credit" 9 issues pat_01KX05B001DHDMGCG0XCHRSMX1 "chatbot salary candidates women terms" 6 issues pat_01KX05B004P5AKS3M50AS96N1J "military failed loan mortgage applicants" 8 issues ``` These are real semantic groupings — expired certificates, unencrypted backups, credit-model issues, AI-hiring-bias findings, fair-lending findings — surfaced with no labels, no taxonomy, no fine-tuning. ### Hazard forecast + counterfactual ```bash theme={null} curl -s "http://localhost:8001/v1/patterns/$PATTERN_ID/hazard" -H "Authorization: Bearer $TOKEN" ``` **Real fitted hazard** for the top pattern (11 member issues): ```json theme={null} { "model": { "family": "powerlaw_nhpp_hawkes_v1", "fitted_at_run": "run_01KX05AZ9JVH7JDNMEKM2PP9ZH" }, "fit_quality": { "expected_events_window": 3.34, "observed_events_window": 5, "window_days": 90.0 }, "forecast": [ { "horizon_days": 14, "lambda": 0.5085, "p_recurrence": 0.3986 }, { "horizon_days": 30, "lambda": 1.0864, "p_recurrence": 0.6626 }, { "horizon_days": 90, "lambda": 3.2262, "p_recurrence": 0.9603 }, { "horizon_days": 365, "lambda": 12.6335, "p_recurrence": 1.0 } ], "time_to_repeat": { "median_days": 19.1, "interval": { "level": 0.8, "lo_days": 2.9, "hi_days": 64.0 } }, "crest_alert": { "active": false, "threshold": 0.6, "consecutive_runs": 0 } } ``` The counterfactual is a pure function of the stored fit — deterministic, cached by request-body hash: ```bash theme={null} curl -s -X POST "http://localhost:8001/v1/patterns/$PATTERN_ID/hazard/counterfactual" \ -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" \ -d '{"covariates": {"ci": 0.0}, "horizons": [30, 365]}' ``` ```json theme={null} { "covariates": { "vp": 0.33, "ci": 0.0, "sh": 0.0 }, "forecast": [ { "horizon_days": 30, "lambda": 1.0864, "expected_events_pre": 1.0864, "expected_events_post": 1.0864, "events_avoided": 0.0 }, { "horizon_days": 365, "lambda": 12.6335, "expected_events_pre": 12.6335, "expected_events_post": 12.6335, "events_avoided": 0.0 } ] } ``` `events_avoided: 0.0` here is honest, not broken: the fitted criticality covariate (`ci`) is 0 for this pattern because criticality inputs are still sitting in the review queue (see Step 5) — a counterfactual that reduces a zero covariate changes nothing. Resolve the abstentions and re-run, and this number moves. ### Predictions ```bash theme={null} curl -s "http://localhost:8001/v1/predictions/summary" -H "Authorization: Bearer $TOKEN" ``` ```json theme={null} { "total": 37, "high_probability": 0, "above_alert_threshold": 0, "avg_probability": 0.2301, "crest_count": 0, "by_impact": { "critical": 0, "high": 0, "medium": 4, "low": 33 }, "by_pattern": [ { "name": "secure automated division service allow", "p_recurrence": 0.467, "median_days": 15.8 }, { "name": "network fire pay internet facility", "p_recurrence": 0.4276, "median_days": 17.4 }, { "name": "administrative database code hardcoded source", "p_recurrence": 0.4139, "median_days": 18.2 } ] } ``` `GET /v1/predictions` lists the per-pattern, per-horizon rows the run generated (37 patterns × horizons), each carrying `p_recurrence`, `horizon_days`, and full provenance back to the run. ### Backtest + accuracy ```bash theme={null} curl -s -X POST "http://localhost:8001/v1/predictions/backtests" \ -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" -d '{"horizon_days": 14}' ``` The rolling-origin backtest re-fits every pattern at every historical split point and scores the held-out next event — deterministic, no RNG. **Real result** (139 folds over this dataset): ```json theme={null} { "brier": 0.2098, "precision": 0.0, "recall": 0.0, "lead_time_days": 47.6177, "interval_coverage_observed": 0.7482, "sample_size": 139, "horizon_days": 14.0, "calibration_curve": [ { "bin_lo": 0.0, "bin_hi": 0.2, "predicted_mean": 0.1536, "observed_rate": 0.3111, "n": 45 }, { "bin_lo": 0.2, "bin_hi": 0.4, "predicted_mean": 0.2922, "observed_rate": 0.2, "n": 70 }, { "bin_lo": 0.4, "bin_hi": 0.6, "predicted_mean": 0.4691, "observed_rate": 0.0909, "n": 22 }, { "bin_lo": 0.6, "bin_hi": 0.8, "predicted_mean": 0.7283, "observed_rate": 0.0, "n": 2 } ] } ``` `GET /v1/predictions/accuracy` now reports `status: "ok"` with these figures instead of `no_backtest_yet` — accuracy is only ever computed from backtests, never fabricated. (The 0.0 precision/recall on a 14-day horizon over synthetic-history-free data is itself honest: the interval coverage of 0.75 and the calibration curve are the informative figures at this data volume.) ### Recommendation + business case ```bash theme={null} curl -s -X POST "http://localhost:8001/v1/recommendations" \ -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" \ -d '{"title": "Fix: compliance suspicious alerts monitoring operations", "description": "Address the root cause surfaced by this pattern.", "type": "process", "impact": "high", "effort": "medium", "fix_class": "structural", "pattern_id": "'$PATTERN_ID'"}' ``` `business_case` is computed **server-side only** from the pattern's hazard fit, its counterfactual, and the active rate-card — it is rejected with a 422 if a client tries to supply it. **Real response:** ```json theme={null} { "id": "rec_01KX05BDW6DKG9R6G8JJYGJFJG", "rank": 1, "status": "pending", "business_case": { "ale_cents": 283494, "remediation_cost_cents": 240000, "p_pre_365": 1.0, "p_post_365": 1.0, "expected_events_pre_365": 12.6335, "expected_events_post_365": 12.6335, "events_avoided_365": 0.0, "t_r_median_days": 19.1, "coupling": { "hazard_run": "run_01KX05AZ9JVH7JDNMEKM2PP9ZH", "counterfactual_hash": "sha256:0eb93878a219a501..." } } } ``` Same story as the counterfactual above: `events_avoided_365: 0.0` because the criticality covariate this fix-class would reduce is still zero pending review-queue resolution — the business case is coupled (by hash) to the exact hazard run and counterfactual that produced it, and it improves as the calibration data does. ## Step 5 — Review queue: what Qwen abstained on Causeloop never guesses on a low-confidence classification — a multi-label or empty conformal prediction set writes `value: null` and creates a review-queue item instead (`docs/v2/DESIGN.md` Design Rule 4: abstention is a valid answer). This is real, not simulated: qwen3:4b abstained on real issues in this run. **Real numbers from this run:** ```bash theme={null} curl -s "http://localhost:8001/v1/review-queue/stats" -H "Authorization: Bearer $TOKEN" ``` ```json theme={null} { "total_open": 494, "by_type": { "extraction": 494, "criticality": 0, "root_cause": 0, "ingest": 0 } } ``` **494 abstentions across 250 issues** — and this is *expected, correct* behavior for a brand-new workspace, not a failure. This is the conformal cold-start: with zero human-labeled calibration items, the abstention threshold is a fixed documented default (`tau = 0.50`), so any label scoring ≥ 0.5 enters the prediction set — and a set with two labels (genuine ambiguity) or zero labels (no signal) abstains to a human rather than guessing. 201 of 250 issues still got at least one attribute written directly (a confident singleton set); the rest of the attribute×issue pairs went to review. As humans resolve items, the calibration set grows and the threshold tightens to hit the target coverage (90%) empirically — abstention volume drops on real evidence, not on optimism. Here's a real item from this run — qwen3:4b scored a finding's `customer_harm` and found genuine ambiguity between `systemic` (0.85) and `material` (0.75): ```json theme={null} { "id": "rev_01KX0482PZ8H99A5Z9X9S31WZN", "item_type": "extraction", "payload": { "attr": "customer_harm", "prediction_set": ["systemic", "material"], "coverage_level": 0.9, "scores": { "systemic": 0.85, "mass_redress": 0.15, "vulnerable": 0.25, "material": 0.75, "none": 0.05 } } } ``` Both labels cleared the threshold, so the model — correctly — refused to pick one. A human resolves it: ```bash theme={null} curl -s -X POST "http://localhost:8001/v1/review-queue/rev_01KX0482PZ8H99A5Z9X9S31WZN/resolve" \ -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" \ -d '{"chosen_value": "material", "note": "Reviewed: single-customer material harm, not systemic"}' ``` Executed for real in this run: the item flipped to `resolved`, the value was written with `origin: "human"` (human always wins over the model), an `attr_corrected` event was appended, and the calibration set grew: ```bash theme={null} curl -s "http://localhost:8001/v1/ai/calibration/extract:customer_harm" -H "Authorization: Bearer $TOKEN" ``` ```json theme={null} { "task": "extract:customer_harm", "n_items": 1, "target_coverage": 0.9, "last_audit": null, "degraded": false } ``` The other large abstention class in this run is `financial_loss_cents` — that one involves **no LLM at all** (design rule: LLMs never produce dollars); a deterministic regex parses exactly one unambiguous monetary mention from the narrative, and zero or multiple mentions abstain to review. ## Step 6 — Replay: prove the run is deterministic Every engine run pins its inputs (config versions, embedding manifest, scope) into a manifest. Replaying a run re-executes from that exact manifest and diffs the result against the original — this is the determinism gate, and it's a real re-execution, not a cached response. ```bash theme={null} curl -s -X POST "http://localhost:8001/v1/engine/runs/$RUN_ID/replay" \ -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" ``` **Real result** — the replay re-executed all ten stages from the pinned manifest and matched the original exactly: ```json theme={null} { "status": "done", "shadow_run_id": "run_01KX05C3CWRPS7VDQ40DQBWN3F", "identical": true, "diff_summary": { "identical": true, "snapshot_hash_match": true, "theme_partition_match": true, "loop_partition_match": true, "counts_match": true, "original_snapshot_hash": "ad8891d4796c0eac224ab191d4e7275979cd4c3a4daaf19f31c300f21310f02c", "replay_snapshot_hash": "ad8891d4796c0eac224ab191d4e7275979cd4c3a4daaf19f31c300f21310f02c" } } ``` Same snapshot hash, same 9-theme/37-loop partition, same counts — on real Qwen embeddings. Note what this does and does not prove: the replay reads the original run's pinned inputs (embeddings are content-hash-cached; extraction outputs live in `llm_cache`), so it verifies the *engine* is deterministic given fixed inputs. Cross-run LLM determinism is a separate property — see [Production notes](#production-notes). ## Real bugs found and fixed while writing this guide This guide is only useful if the commands on this page actually work as written. Running the flow for real surfaced six genuine bugs in the local-Qwen path, all fixed in `causeloop-backend` (branch `feat/v2-pipeline`) alongside this guide: `app/ai/router.py`'s provider map only had `anthropic`, `openai`, and `mock` — there was no `ollama` entry, even though the capabilities layer (`app/capabilities/registration.py`) and its tests already assumed a provider literally named `"ollama"` existed. Setting `LLM_PROVIDER_ORDER=ollama` silently fell through to the mock safety net with no error. Fixed by adding `app/ai/providers/ollama_provider.py` — a native `/api/chat` client with `think: false`, grammar-constrained JSON output (`format: "json"` whenever the request carries a JSON schema), and seeded temperature-0 decoding — registered under `"ollama"` in the router, with new `OLLAMA_BASE_URL`/`OLLAMA_MODEL`/`OLLAMA_THINK`/`OLLAMA_TEMPERATURE`/`OLLAMA_SEED` settings. `app/capabilities/resolver.py` looked up a purpose's provider chain with `binding.bindings.get(str(purpose))` — but `Purpose` is a `(str, Enum)`, and under Python's Enum semantics `str(Purpose.EXTRACTION)` returns `"Purpose.EXTRACTION"`, not `"extraction"`. Every purpose lookup (extraction, naming, fishbone, recommendation, prediction) missed the YAML binding and raised `LookupError`, which every call site treats as "no provider resolvable" and silently abstains on. This affected **every** sovereign/hybrid-mode deployment, not just this guide's Ollama profile — confirmed by a 250-issue run that abstained on 100% of extractions before the fix. Fixed with a small `_binding_key()` helper that reads `.value` off an `Enum` member. Four call sites (`conformal_extract.py`, `extract.py`, `rootcause/classify.py`, `rootcause/narrate.py`) set `max_tokens` between 160 and 512 for short JSON/classification prompts — reasonable for a non-reasoning hosted model, but with thinking enabled qwen3:4b's `...` reasoning consumed the entire budget before ever emitting an answer, leaving `resp.text == ""` and indistinguishable from genuine model abstention. Measured directly: a 5-field extraction needed \~1,850 completion tokens; a 4-label classification needed more than 2,048. Budgets were raised to 3072/2048 as defense-in-depth for thinking-enabled configurations (the default `OLLAMA_THINK=false` path answers in \~100 tokens regardless). `POST /ingest/batch` intentionally skips synchronous per-record encoding for throughput (documented in the router as "deferred to Phase 2"), but nothing else ever picked it up — `app/services/engine_run.py`'s `real_feature_forge` only **reads** already-stored embedding vectors, it never computes any. Every batch-ingested workspace therefore reached the engine with zero embeddings. Confirmed against a full 250-issue batch: the engine run reported `"degenerate partition detected (100% singletons)"` and produced **0 patterns**, in \~16 seconds — far too fast for any real model calls to have happened. Fixed by firing the same forge job batch ingest already fires for extraction, mirroring the existing `_trigger_auto_extract` fire-and-forget pattern. Batch ingest fires forge + extraction jobs for every record concurrently, so \~250 first LLM calls hit the provider's single `httpx.AsyncClient` at once. httpx's default connection-pool acquire timeout is **5 seconds**; \~125 calls raised `PoolTimeout` in the opening seconds — and `PoolTimeout` stringifies to an **empty string**, so the log line read `provider call failed: ` with no reason. Worse: `conformal_extract` cached the resulting all-zero score dict into `llm_cache`, keyed by (model, prompt, narrative) — recording a transport failure as "the model confidently scored every label 0.0", permanently. Fixed with a client-side `asyncio.Semaphore(4)` (Ollama serializes across a handful of slots anyway; held requests wait in-process with no socket and no ticking timeout), the pool-acquire timer disabled, `%r` logging so empty-string exceptions stay identifiable, and — separately — failed scorings are no longer cached. Verified: the same 250-record burst re-run produced **0 provider failures**. `POST /engine/runs` is documented as the single entry point that "executes the whole pipeline", but the run DAG's `criticality_rollup` stage was a no-op deferring to "the wave that owns pattern projection" — which never landed. Only the Redis-gated pipeline orchestrator's S3 stage projected loops→patterns, and the auto-infer path is gated on a trained workspace model. Confirmed live: a run over 250 real issues succeeded with `loops=42` — and `hazard_fit` fitted **0** patterns (it fits per-pattern), `GET /patterns` returned `[]`, and predictions/recommendations/financials all stayed empty. Fixed by implementing `real_criticality_rollup`: find-or-create a pattern row per loop keyed by `fingerprint.loop_id` (the same semantics as the pipeline's S3 stage, so both entry points converge on the same rows), link member issues, name from taxonomy/top-terms, then roll up criticality per pattern. The run output above — 37 patterns projected, 37 fitted — is this fix working. ## Troubleshooting Symptoms → causes, distilled from the real debugging that produced this guide: | Symptom | Likely cause | Check / fix | | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Engine run succeeds but `GET /patterns` is `[]` | Running a backend without the pattern-projection fix (bug 6), or issues were never embedded (bug 4) | `criticality_rollup` counters in `stages[]` should show `patterns_projected > 0` | | Run reports `partition_degenerate: true` / "100% singletons" | Issues reached the engine with no (or all-identical) embeddings | Confirm forge ran: issue detail should show `concern`/`content_hash` or `5c_abstained:*` in `derived_flags` | | Every extraction abstains; log says `provider call failed: ` (blank reason) | Transport failure whose exception stringifies to empty — pool/read timeout under burst load | Blank-reason lines are timeouts, not model output; bound client concurrency (fixed in bug 5) | | Every LLM call answers instantly but output is canned/mock-looking | `LLM_PROVIDER_ORDER` didn't resolve a real provider and fell through to `mock` | `GET /v1/ai/providers` — the first entry in `order` must be `ollama` with `available: true` | | LLM calls return empty text with `finish_reason: length` | Thinking-mode tokens ate the whole `max_tokens` budget | Keep `OLLAMA_THINK=false`; if you enable it, budgets must be \~2,000+ tokens for short JSON tasks | | Calls take 40-50s each instead of 2-5s | qwen3 thinking mode is on (or you're on the OpenAI-compat endpoint, which can't turn it off) | `OLLAMA_THINK=false` via the native-API provider; verify with a single timed call | | Some issues never show `concern` after forge "completes" | Not stuck — the 5C extraction honestly abstained on that narrative | Look for `5c_abstained:concern` in the issue's `derived_flags`; the issue still embedded and clusters | | Hundreds of review-queue items right after first ingest | Conformal cold-start (no calibration items yet ⇒ documented default threshold) — expected | Resolve items; `GET /ai/calibration/{task}` shows the set growing and coverage tracking begin | | Server aborts at startup complaining about identity provider or memory store | A repo `.env` pinning `ENVIRONMENT=production` is shadowing your profile | Export `ENVIRONMENT=development AUTH_PROVIDER=none` in the launch env (process env beats `.env`) | ## Production notes **Postgres + pgvector dimension.** The repo's pgvector column is `vector(384)` (`db/migrations/0009_embeddings_pgvector.sql`), sized for the default TEI model (`BAAI/bge-small-en-v1.5`, 384-dim). `qwen3-embedding:0.6b` returns **1024-dim** vectors — that will not fit today. Two honest paths: (a) write an additive migration to a dimension-flexible embedding store if your schema can support it cleanly, or (b) run local-Qwen workflows against the in-memory store (as this guide does) and reserve Postgres persistence for a TEI-compatible 384-dim embedding model. Do not silently truncate or pad vectors to force a fit — that corrupts every downstream similarity computation. **TEI vs. Ollama for embeddings.** [HuggingFace TEI](https://github.com/huggingface/text-embeddings-inference) is the other supported embedding provider (`EMBEDDINGS_PROVIDER=tei`). TEI is purpose-built for batched embedding throughput and is the better choice for production ingestion volume; Ollama's embedding endpoint is convenient for local development but was not designed for high-throughput batch encoding. **Model size tradeoffs.** This guide uses `qwen3:4b` for every task tier (reasoning/balanced/fast) — simplest to reason about for a single-model local profile. On constrained hardware, `qwen3:1.7b` (set `OLLAMA_MODEL=qwen3:1.7b`) is meaningfully faster and lighter (\~1.4GB vs \~3.5GB resident) with a real quality tradeoff on harder extraction/classification tasks; `qwen3:0.6b` exists but is a stretch for structured extraction. Benchmark against your own data before committing — with thinking already disabled, model size is the second lever, not the first. **The thinking-mode tax — the single biggest local-throughput lever.** qwen3-family models emit a hidden `...` reasoning chain by default on *every* request, including 5-line JSON extractions. Measured on this machine (qwen3:4b, Ollama 0.23.2): with thinking on, one extraction = \~1,850 completion tokens / \~44s; with `think: false` **plus** `format: "json"` (grammar-constrained decoding), the same extraction = 103 tokens / **2.6s** — a \~17x speedup, and the difference between a \~9-hour and a \~25-minute full-dataset onboarding. Three operational facts worth knowing, all verified directly: * The `think` field only works on Ollama's **native** `/api/chat` endpoint — the OpenAI-compatible `/v1/chat/completions` layer silently ignores it, and qwen3:4b ignored the `/no_think` prompt-suffix soft switch entirely. This is why Causeloop's Ollama provider uses the native API. * `think: false` alone is not enough for structured tasks: qwen3:4b still narrates its chain-of-thought in plain content before the JSON. `format: "json"` constrains generation from the first token and eliminates it. * The tradeoff is real but small for closed-vocabulary tasks: thinking mainly buys accuracy on open-ended reasoning (root-cause synthesis, narration), not on "score these 4 labels" prompts. Causeloop defaults to `OLLAMA_THINK=false` globally; flip it to `true` if you observe quality problems on reasoning-tier tasks and can afford the latency. **Determinism and temperature.** The Ollama provider pins `temperature: 0` and `seed: 42` (`OLLAMA_TEMPERATURE`/`OLLAMA_SEED`), so decoding is deterministic for a fixed prompt on a fixed model build — this makes the `seed_supported=True` flag in `app/capabilities/registration.py` true in practice. Two honest caveats: (a) determinism holds per model build — pulling a new `qwen3:4b` blob or changing Ollama's numeric backend can change outputs for the same seed; (b) the [replay](#step-6-replay-prove-the-run-is-deterministic) determinism gate above does not depend on LLM decoding determinism at all — it re-executes from the run's pinned manifest, and extraction outputs are content-hash-cached in the `llm_cache` table (`app/engine/forge/conformal_extract.py`), so replays read the original run's frozen outputs rather than re-rolling the model. ## Related Provenance, abstention, hazard models, and configs — the vocabulary this guide assumes. The UI-driven onboarding wizard that wraps this same lifecycle for a new customer workspace. General local setup, without the Qwen-specific profile. Full endpoint reference for every call made in this guide.