Train and Infer Locally with Qwen

Every command, response, and number on this page was captured from a real run against a local Ollama instance and the real Causeloop v2 API — nothing here is simulated. Token values are redacted; everything else is verbatim.

This guide runs Causeloop’s full lifecycle — ingest historical issues, calibrate the engine (“training”), then read live inference outputs (patterns, hazard forecasts, predictions, recommendations) — using only local models served by Ollama. No Anthropic, OpenAI, or other hosted API key is used anywhere in this walkthrough.

Concept: what “training” means in Causeloop

Causeloop v2 deliberately has no trainable model endpoint — there is no /engine/model/train, no gradient descent, no model weights you fine-tune. If you’re looking for that, it doesn’t exist here on purpose (see Concepts → Provenance).

“Training” in Causeloop is the calibration lifecycle: a deterministic pipeline that turns raw historical issues into a calibrated, queryable model of your organization’s recurring failures:

Ingest historical issues into the event log.
Embed the corpus (Qwen embeddings, in this guide).
Run the engine — theta calibration, Leiden graph partitioning, term naming, criticality scoring, and hazard (recurrence) fitting, all in one deterministic run.
Review any abstentions — low-confidence extractions the model declined to guess on, routed to a human via the review queue.
Activate — the resulting patterns, hazard fits, and predictions are live and queryable immediately; nothing further to “deploy.”

Every one of the five steps below is a real API call against this pipeline. Nothing is precomputed or faked for the walkthrough.

Prerequisites

Install Ollama

brew install ollama   # macOS; see ollama.com for other platforms
ollama serve          # if not already running as a service

Pull the models used in this guide

ollama pull qwen3-embedding:0.6b   # embeddings — 1024-dim
ollama pull qwen3:4b               # extraction, classification, narration

Verify both are present:

curl -s http://localhost:11434/api/tags | python3 -c \
  "import json,sys; print([m['name'] for m in json.load(sys.stdin)['models']])"

Clone causeloop-backend and install dependencies

git clone https://github.com/your-org/causeloop-backend.git
cd causeloop-backend
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

Hardware notes

This guide was run on a single developer machine (no GPU cluster, no dedicated inference server) with Ollama serving both models. Two things to know before you start:

qwen3:4b “thinks” by default — and Causeloop turns it off. Qwen3’s chat template enables a hidden <think>...</think> reasoning block before the visible answer, for every request. Measured on this machine: a 5-field JSON extraction burned ~1,850 completion tokens / ~44 seconds with thinking on, versus 103 tokens / 2.6 seconds with thinking off and grammar-constrained JSON — a ~17x difference. Causeloop’s Ollama provider disables thinking by default (OLLAMA_THINK=false); see Production notes for the full story and the quality tradeoff.
Embeddings are cheap. qwen3-embedding:0.6b returns a 1024-dim vector in well under a second per call — the LLM calls (extraction, classification, narration) dominate the cost, not the embedding calls.

With thinking off, the full 250-issue dataset processes end-to-end on a laptop in tens of minutes — every number below is from that full-dataset run.

Step 1 — Configure the local Qwen profile

Start the backend with DATABASE_URL unset — this guide uses the in-memory store, which has no vector-dimension constraint. This matters: the repo’s pgvector column is fixed at vector(384) (db/migrations/0009_embeddings_pgvector.sql), but qwen3-embedding:0.6b returns 1024-dim vectors (verified below). Running against Postgres today would fail at the vector column; running in-memory sidesteps it entirely. See Production notes for the real options if you need Postgres persistence.

ANTHROPIC_API_KEY= OPENAI_API_KEY= DATABASE_URL= \
ENVIRONMENT=development \
AUTH_PROVIDER=none \
PORT=8001 \
EMBEDDINGS_PROVIDER=ollama \
EMBEDDINGS_MODEL=qwen3-embedding:0.6b \
EMBEDDINGS_BASE_URL=http://localhost:11434/v1 \
EMBEDDINGS_DIM=1024 \
USE_MOCK_EMBEDDINGS=false \
LLM_PROVIDER_ORDER=ollama \
OLLAMA_BASE_URL=http://localhost:11434 \
OLLAMA_MODEL=qwen3:4b \
USE_MOCK_LLM=false \
.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8001

The Ollama LLM provider also honors three tuning knobs, all defaulted for the local profile: OLLAMA_THINK=false (disables qwen3’s hidden reasoning — the single biggest local-throughput lever, see Production notes), OLLAMA_TEMPERATURE=0.0 and OLLAMA_SEED=42 (deterministic decoding for a fixed prompt on a fixed model build).

LLM_PROVIDER_ORDER=ollama and the OLLAMA_* settings require backend fixes made alongside this guide — see Real bugs found and fixed below. On older builds this profile silently degrades every LLM call to the mock provider with no error.

Mint a scoped dev token (no external identity provider needed with AUTH_PROVIDER=none):

TOKEN=$(.venv/bin/python -m scripts.mint_dev_token ws_01 usr_01)

Confirm the local providers actually resolved:

curl -s http://localhost:8001/v1/ai/providers -H "Authorization: Bearer $TOKEN"

{
  "providers": [
    { "name": "ollama", "available": true,
      "models": { "reasoning": "qwen3:4b", "balanced": "qwen3:4b", "fast": "qwen3:4b" } },
    { "name": "mock", "available": true,
      "models": { "reasoning": "mock-model", "balanced": "mock-model", "fast": "mock-model" } }
  ],
  "order": ["ollama", "mock"]
}

ollama is first in order and available: true — every real LLM call in this run went to qwen3:4b, with mock only as the never-triggered safety net. Confirm the embedding dimension while you’re at it — this is the number that has to match EMBEDDINGS_DIM and (in a Postgres deployment) the pgvector column:

from app.ml.embeddings.ollama_provider import OllamaEmbeddingProvider
p = OllamaEmbeddingProvider(base_url="http://localhost:11434/v1", model="qwen3-embedding:0.6b", dim=1024)
vecs = await p.embed_batch(["hello world"])
len(vecs[0])   # -> 1024

Step 2 — Convert and ingest the dataset

The source dataset is issues.xlsx — a Sheet1 with 250 rows and three columns: Issue ID, Unstructured Text, issue_created. Convert each row into the v2 ingest contract (external_id, title, narrative, occurred_at), following the same title convention as scripts/convert_issue_set_2.py — first sentence, truncated to 120 chars:

def _title_from_text(text: str) -> str:
    c = (text or "").strip()
    first = c.split(". ")[0].strip()
    base = first if 0 < len(first) <= 120 else c
    return (base[:117] + "...") if len(base) > 120 else base

record = {
    "external_id": str(row["Issue ID"]),
    "title": _title_from_text(row["Unstructured Text"]),
    "narrative": row["Unstructured Text"],
    "occurred_at": row["issue_created"].isoformat() + "Z",
    "source": "custom_api",
}

Batch-ingest all 250 in five batches of 50, each with its own Idempotency-Key:

for i in range(0, len(records), 50):
    batch = records[i:i + 50]
    requests.post(f"{BASE}/ingest/batch", json={"records": batch},
                   headers={"Authorization": f"Bearer {TOKEN}",
                            "Idempotency-Key": str(uuid.uuid4())})

Real output — all five batches, full 250-row file:

batch 0-50:   accepted=50 rejected=0 dt=0.09s
batch 50-100: accepted=50 rejected=0 dt=0.20s
batch 100-150: accepted=50 rejected=0 dt=0.65s
batch 150-200: accepted=50 rejected=0 dt=1.18s
batch 200-250: accepted=50 rejected=0 dt=0.82s

TOTAL accepted=250 rejected=0 wall=2.94s
engine_status_summary: {"eligible": 250, "blocked_no_narrative": 0, "ineligible": 0}

POST /ingest/batch itself only validates and persists — it does not call any model synchronously. Every engine-eligible issue (one with a non-empty narrative) then automatically kicks off, in the background:

Forge — a 5C extraction call (concern/context/consequence/cause/cap) that produces the text the pattern-clustering embedding is built from, plus the embedding call itself.
Conformal extraction — two classification calls (process_cadence, customer_harm) that feed criticality/financials, each scoring every candidate label.

That’s up to 3 LLM calls per issue — ~750 calls for the full file. With thinking disabled (the default), each call completes in a few seconds on this laptop; with qwen3’s default thinking mode it would be ~44–50 seconds per call (~9+ hours total — this is why OLLAMA_THINK=false is the default, see Production notes).

Watching forge + extraction run

Each accepted issue schedules its forge and extraction jobs as fire-and-forget background tasks — the ingest response returns immediately. Poll an issue to watch its concern/cause/consequence fields populate as forge completes:

curl -s "http://localhost:8001/v1/issues/$ISSUE_ID" -H "Authorization: Bearer $TOKEN" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('concern'), d.get('cause'))"

Real 5C extraction output for one issue (AI-AUDIT-001, narrative: “Just found out the dev team in London has been pasting snippets of our proprietary trading algorithms into public ChatGPT to debug them…”):

{
  "concern": "Exposure of proprietary trading algorithms to public via ChatGPT",
  "context": "Dev team in London has been pasting snippets of proprietary trading algorithms into public ChatGPT for debugging",
  "consequence": "Potential security breaches, loss of competitive advantage, and intellectual property compromise",
  "cause": "Internal debugging tools are perceived as too slow by the dev team",
  "cap": "Not specified in narrative"
}

Real timing, full 250-issue run on this laptop: all 250 forge jobs (one 5C extraction + one embedding each) completed in ~18 minutes; the classification tail (two conformal scoring calls per issue) fully drained at ~62 minutes after ingest. That’s ~750 qwen3:4b calls plus 250 embedding calls through a 4-deep client-side queue — roughly 5s per LLM call end-to-end — with 0 failed provider calls. (With qwen3’s default thinking mode this same workload measured out to ~9 hours; see Production notes.)

Step 3 — Run the engine (“train”)

Once forge has completed, trigger a full engine run:

curl -s -X POST http://localhost:8001/v1/engine/runs \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" \
  -d '{"scope": {}, "trigger": "manual", "mode": "full"}'

{"run_id": "run_01KX05AZ9JVH7JDNMEKM2PP9ZH", "status": "queued",
 "scope_hash": "70c3a3b523ef2dc4a2a58929007de8181c4ad6a1c3a9f4e8ad780a0c88b7af31",
 "mode": "full", "mode_effective": "full", "mode_degraded": false}

Poll GET /v1/engine/runs/{run_id} until status is terminal. The run itself is fast — every LLM-heavy step already happened at ingest time, so the engine stages are pure deterministic computation over stored embeddings and events. Real result (250 issues, total stage time under one second):

status: succeeded
counts: themes=9  loops=37  edges=870  partition_degenerate=false

feature_forge        succeeded   11ms
graph_weave          succeeded  608ms  {edge_count: 870}
leiden_partition     succeeded   11ms  {theme_count: 9, loop_count: 37}
dsu_crosscheck       succeeded   12ms  {dsu_theme_count: 7, dsu_loop_count: 25, flagged_count: 0}
loop_anchor          succeeded   29ms  {anchored: 37, inherited: 0, minted: 37}
term_namer           succeeded   37ms  {versions_frozen: 37}
taxonomy_label       succeeded    0ms  {labeled: 37, taxonomy_matched: 0}
criticality_rollup   succeeded   23ms  {patterns_projected: 37, member_links: 250, rolled_up: 37}
hazard_fit           succeeded   32ms  {patterns_fitted: 37, patterns_skipped: 0}
alert_eval           succeeded    1ms  {rules_evaluated: 0}

manifest_hash: sha256:e09bf1349bf30bbc2396f1cb75b7abe2fe6032c2f0391b0fa56db8175de07f9b

partition_degenerate: false is the signal that the Qwen embeddings produced real cluster structure (a broken embedding path shows up here as “100% singletons”). Note criticality_rollup projecting all 37 loops onto pattern rows and hazard_fit fitting all 37 — both were broken before the fixes shipped with this guide (see bugs below).

Step 4 — Inference: read the trained model

The same run’s outputs are queryable immediately — “inference” here just means reading patterns, forecasts, and recommendations back out of the run you just calibrated.

Patterns

curl -s "http://localhost:8001/v1/patterns?limit=50" -H "Authorization: Bearer $TOKEN"

37 patterns, named from each cluster’s BM25 top terms over real member narratives. A sample of what qwen3-embedding:0.6b’s clusters actually look like on this dataset:

pat_01KX05AZZXS7E4VMQP6JVANTBX  "compliance suspicious alerts monitoring operations"   11 issues
pat_01KX05B000DV9H6RBFBNPDHA5C  "database master certificate expired configuration"    10 issues
pat_01KX05B00416PHBKXHAJAHECXC  "tapes cleartext unencrypted backup party"              9 issues
pat_01KX05B003H8Z7K1VF4JSCP5KP  "model complex analysts defaults credit"                9 issues
pat_01KX05B001DHDMGCG0XCHRSMX1  "chatbot salary candidates women terms"                 6 issues
pat_01KX05B004P5AKS3M50AS96N1J  "military failed loan mortgage applicants"              8 issues

These are real semantic groupings — expired certificates, unencrypted backups, credit-model issues, AI-hiring-bias findings, fair-lending findings — surfaced with no labels, no taxonomy, no fine-tuning.

Hazard forecast + counterfactual

curl -s "http://localhost:8001/v1/patterns/$PATTERN_ID/hazard" -H "Authorization: Bearer $TOKEN"

Real fitted hazard for the top pattern (11 member issues):

{
  "model": { "family": "powerlaw_nhpp_hawkes_v1",
             "fitted_at_run": "run_01KX05AZ9JVH7JDNMEKM2PP9ZH" },
  "fit_quality": { "expected_events_window": 3.34, "observed_events_window": 5, "window_days": 90.0 },
  "forecast": [
    { "horizon_days": 14,  "lambda": 0.5085,  "p_recurrence": 0.3986 },
    { "horizon_days": 30,  "lambda": 1.0864,  "p_recurrence": 0.6626 },
    { "horizon_days": 90,  "lambda": 3.2262,  "p_recurrence": 0.9603 },
    { "horizon_days": 365, "lambda": 12.6335, "p_recurrence": 1.0 }
  ],
  "time_to_repeat": { "median_days": 19.1, "interval": { "level": 0.8, "lo_days": 2.9, "hi_days": 64.0 } },
  "crest_alert": { "active": false, "threshold": 0.6, "consecutive_runs": 0 }
}

The counterfactual is a pure function of the stored fit — deterministic, cached by request-body hash:

curl -s -X POST "http://localhost:8001/v1/patterns/$PATTERN_ID/hazard/counterfactual" \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" \
  -d '{"covariates": {"ci": 0.0}, "horizons": [30, 365]}'

{ "covariates": { "vp": 0.33, "ci": 0.0, "sh": 0.0 },
  "forecast": [
    { "horizon_days": 30,  "lambda": 1.0864,  "expected_events_pre": 1.0864,  "expected_events_post": 1.0864,  "events_avoided": 0.0 },
    { "horizon_days": 365, "lambda": 12.6335, "expected_events_pre": 12.6335, "expected_events_post": 12.6335, "events_avoided": 0.0 }
  ] }

events_avoided: 0.0 here is honest, not broken: the fitted criticality covariate (ci) is 0 for this pattern because criticality inputs are still sitting in the review queue (see Step 5) — a counterfactual that reduces a zero covariate changes nothing. Resolve the abstentions and re-run, and this number moves.

Predictions

curl -s "http://localhost:8001/v1/predictions/summary" -H "Authorization: Bearer $TOKEN"

{ "total": 37, "high_probability": 0, "above_alert_threshold": 0,
  "avg_probability": 0.2301, "crest_count": 0,
  "by_impact": { "critical": 0, "high": 0, "medium": 4, "low": 33 },
  "by_pattern": [
    { "name": "secure automated division service allow",      "p_recurrence": 0.467,  "median_days": 15.8 },
    { "name": "network fire pay internet facility",           "p_recurrence": 0.4276, "median_days": 17.4 },
    { "name": "administrative database code hardcoded source", "p_recurrence": 0.4139, "median_days": 18.2 }
  ] }

GET /v1/predictions lists the per-pattern, per-horizon rows the run generated (37 patterns × horizons), each carrying p_recurrence, horizon_days, and full provenance back to the run.

Backtest + accuracy

curl -s -X POST "http://localhost:8001/v1/predictions/backtests" \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" -d '{"horizon_days": 14}'

The rolling-origin backtest re-fits every pattern at every historical split point and scores the held-out next event — deterministic, no RNG. Real result (139 folds over this dataset):

{ "brier": 0.2098, "precision": 0.0, "recall": 0.0,
  "lead_time_days": 47.6177, "interval_coverage_observed": 0.7482,
  "sample_size": 139, "horizon_days": 14.0,
  "calibration_curve": [
    { "bin_lo": 0.0, "bin_hi": 0.2, "predicted_mean": 0.1536, "observed_rate": 0.3111, "n": 45 },
    { "bin_lo": 0.2, "bin_hi": 0.4, "predicted_mean": 0.2922, "observed_rate": 0.2,    "n": 70 },
    { "bin_lo": 0.4, "bin_hi": 0.6, "predicted_mean": 0.4691, "observed_rate": 0.0909, "n": 22 },
    { "bin_lo": 0.6, "bin_hi": 0.8, "predicted_mean": 0.7283, "observed_rate": 0.0,    "n": 2 }
  ] }

GET /v1/predictions/accuracy now reports status: "ok" with these figures instead of no_backtest_yet — accuracy is only ever computed from backtests, never fabricated. (The 0.0 precision/recall on a 14-day horizon over synthetic-history-free data is itself honest: the interval coverage of 0.75 and the calibration curve are the informative figures at this data volume.)

Recommendation + business case

curl -s -X POST "http://localhost:8001/v1/recommendations" \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" \
  -d '{"title": "Fix: compliance suspicious alerts monitoring operations",
       "description": "Address the root cause surfaced by this pattern.",
       "type": "process", "impact": "high", "effort": "medium",
       "fix_class": "structural", "pattern_id": "'$PATTERN_ID'"}'

business_case is computed server-side only from the pattern’s hazard fit, its counterfactual, and the active rate-card — it is rejected with a 422 if a client tries to supply it. Real response:

{ "id": "rec_01KX05BDW6DKG9R6G8JJYGJFJG", "rank": 1, "status": "pending",
  "business_case": {
    "ale_cents": 283494, "remediation_cost_cents": 240000,
    "p_pre_365": 1.0, "p_post_365": 1.0,
    "expected_events_pre_365": 12.6335, "expected_events_post_365": 12.6335,
    "events_avoided_365": 0.0, "t_r_median_days": 19.1,
    "coupling": { "hazard_run": "run_01KX05AZ9JVH7JDNMEKM2PP9ZH",
                  "counterfactual_hash": "sha256:0eb93878a219a501..." } } }

Same story as the counterfactual above: events_avoided_365: 0.0 because the criticality covariate this fix-class would reduce is still zero pending review-queue resolution — the business case is coupled (by hash) to the exact hazard run and counterfactual that produced it, and it improves as the calibration data does.

Step 5 — Review queue: what Qwen abstained on

Causeloop never guesses on a low-confidence classification — a multi-label or empty conformal prediction set writes value: null and creates a review-queue item instead (docs/v2/DESIGN.md Design Rule 4: abstention is a valid answer). This is real, not simulated: qwen3:4b abstained on real issues in this run. Real numbers from this run:

curl -s "http://localhost:8001/v1/review-queue/stats" -H "Authorization: Bearer $TOKEN"

{ "total_open": 494,
  "by_type": { "extraction": 494, "criticality": 0, "root_cause": 0, "ingest": 0 } }

494 abstentions across 250 issues — and this is expected, correct behavior for a brand-new workspace, not a failure. This is the conformal cold-start: with zero human-labeled calibration items, the abstention threshold is a fixed documented default (tau = 0.50), so any label scoring ≥ 0.5 enters the prediction set — and a set with two labels (genuine ambiguity) or zero labels (no signal) abstains to a human rather than guessing. 201 of 250 issues still got at least one attribute written directly (a confident singleton set); the rest of the attribute×issue pairs went to review. As humans resolve items, the calibration set grows and the threshold tightens to hit the target coverage (90%) empirically — abstention volume drops on real evidence, not on optimism. Here’s a real item from this run — qwen3:4b scored a finding’s customer_harm and found genuine ambiguity between systemic (0.85) and material (0.75):

{ "id": "rev_01KX0482PZ8H99A5Z9X9S31WZN",
  "item_type": "extraction",
  "payload": {
    "attr": "customer_harm",
    "prediction_set": ["systemic", "material"],
    "coverage_level": 0.9,
    "scores": { "systemic": 0.85, "mass_redress": 0.15, "vulnerable": 0.25,
                "material": 0.75, "none": 0.05 } } }

Both labels cleared the threshold, so the model — correctly — refused to pick one. A human resolves it:

curl -s -X POST "http://localhost:8001/v1/review-queue/rev_01KX0482PZ8H99A5Z9X9S31WZN/resolve" \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" \
  -d '{"chosen_value": "material", "note": "Reviewed: single-customer material harm, not systemic"}'

Executed for real in this run: the item flipped to resolved, the value was written with origin: "human" (human always wins over the model), an attr_corrected event was appended, and the calibration set grew:

curl -s "http://localhost:8001/v1/ai/calibration/extract:customer_harm" -H "Authorization: Bearer $TOKEN"

{ "task": "extract:customer_harm", "n_items": 1, "target_coverage": 0.9,
  "last_audit": null, "degraded": false }

The other large abstention class in this run is financial_loss_cents — that one involves no LLM at all (design rule: LLMs never produce dollars); a deterministic regex parses exactly one unambiguous monetary mention from the narrative, and zero or multiple mentions abstain to review.

Step 6 — Replay: prove the run is deterministic

Every engine run pins its inputs (config versions, embedding manifest, scope) into a manifest. Replaying a run re-executes from that exact manifest and diffs the result against the original — this is the determinism gate, and it’s a real re-execution, not a cached response.

curl -s -X POST "http://localhost:8001/v1/engine/runs/$RUN_ID/replay" \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)"

Real result — the replay re-executed all ten stages from the pinned manifest and matched the original exactly:

{ "status": "done",
  "shadow_run_id": "run_01KX05C3CWRPS7VDQ40DQBWN3F",
  "identical": true,
  "diff_summary": {
    "identical": true,
    "snapshot_hash_match": true,
    "theme_partition_match": true,
    "loop_partition_match": true,
    "counts_match": true,
    "original_snapshot_hash": "ad8891d4796c0eac224ab191d4e7275979cd4c3a4daaf19f31c300f21310f02c",
    "replay_snapshot_hash":   "ad8891d4796c0eac224ab191d4e7275979cd4c3a4daaf19f31c300f21310f02c" } }

Same snapshot hash, same 9-theme/37-loop partition, same counts — on real Qwen embeddings. Note what this does and does not prove: the replay reads the original run’s pinned inputs (embeddings are content-hash-cached; extraction outputs live in llm_cache), so it verifies the engine is deterministic given fixed inputs. Cross-run LLM determinism is a separate property — see Production notes.

Real bugs found and fixed while writing this guide

This guide is only useful if the commands on this page actually work as written. Running the flow for real surfaced six genuine bugs in the local-Qwen path, all fixed in causeloop-backend (branch feat/v2-pipeline) alongside this guide:

1. LLM_PROVIDER_ORDER=ollama silently resolved to nothing

app/ai/router.py’s provider map only had anthropic, openai, and mock — there was no ollama entry, even though the capabilities layer (app/capabilities/registration.py) and its tests already assumed a provider literally named "ollama" existed. Setting LLM_PROVIDER_ORDER=ollama silently fell through to the mock safety net with no error. Fixed by adding app/ai/providers/ollama_provider.py — a native /api/chat client with think: false, grammar-constrained JSON output (format: "json" whenever the request carries a JSON schema), and seeded temperature-0 decoding — registered under "ollama" in the router, with new OLLAMA_BASE_URL/OLLAMA_MODEL/OLLAMA_THINK/OLLAMA_TEMPERATURE/OLLAMA_SEED settings.

2. Sovereign-mode capability routing was completely broken

app/capabilities/resolver.py looked up a purpose’s provider chain with binding.bindings.get(str(purpose)) — but Purpose is a (str, Enum), and under Python’s Enum semantics str(Purpose.EXTRACTION) returns "Purpose.EXTRACTION", not "extraction". Every purpose lookup (extraction, naming, fishbone, recommendation, prediction) missed the YAML binding and raised LookupError, which every call site treats as “no provider resolvable” and silently abstains on. This affected every sovereign/hybrid-mode deployment, not just this guide’s Ollama profile — confirmed by a 250-issue run that abstained on 100% of extractions before the fix. Fixed with a small _binding_key() helper that reads .value off an Enum member.

3. Structured extraction prompts didn't budget for thinking-mode tokens

Four call sites (conformal_extract.py, extract.py, rootcause/classify.py, rootcause/narrate.py) set max_tokens between 160 and 512 for short JSON/classification prompts — reasonable for a non-reasoning hosted model, but with thinking enabled qwen3:4b’s <think>...</think> reasoning consumed the entire budget before ever emitting an answer, leaving resp.text == "" and indistinguishable from genuine model abstention. Measured directly: a 5-field extraction needed ~1,850 completion tokens; a 4-label classification needed more than 2,048. Budgets were raised to 3072/2048 as defense-in-depth for thinking-enabled configurations (the default OLLAMA_THINK=false path answers in ~100 tokens regardless).

4. Batch ingest never forged (embedded) issues at all

POST /ingest/batch intentionally skips synchronous per-record encoding for throughput (documented in the router as “deferred to Phase 2”), but nothing else ever picked it up — app/services/engine_run.py’s real_feature_forge only reads already-stored embedding vectors, it never computes any. Every batch-ingested workspace therefore reached the engine with zero embeddings. Confirmed against a full 250-issue batch: the engine run reported "degenerate partition detected (100% singletons)" and produced 0 patterns, in ~16 seconds — far too fast for any real model calls to have happened. Fixed by firing the same forge job batch ingest already fires for extraction, mirroring the existing _trigger_auto_extract fire-and-forget pattern.

5. A 250-record burst pool-timed-out ~125 LLM calls — and poisoned the cache with the failures

Batch ingest fires forge + extraction jobs for every record concurrently, so ~250 first LLM calls hit the provider’s single httpx.AsyncClient at once. httpx’s default connection-pool acquire timeout is 5 seconds; ~125 calls raised PoolTimeout in the opening seconds — and PoolTimeout stringifies to an empty string, so the log line read provider call failed: with no reason. Worse: conformal_extract cached the resulting all-zero score dict into llm_cache, keyed by (model, prompt, narrative) — recording a transport failure as “the model confidently scored every label 0.0”, permanently. Fixed with a client-side asyncio.Semaphore(4) (Ollama serializes across a handful of slots anyway; held requests wait in-process with no socket and no ticking timeout), the pool-acquire timer disabled, %r logging so empty-string exceptions stay identifiable, and — separately — failed scorings are no longer cached. Verified: the same 250-record burst re-run produced 0 provider failures.

6. Engine runs never projected patterns — the whole inference surface stayed empty

POST /engine/runs is documented as the single entry point that “executes the whole pipeline”, but the run DAG’s criticality_rollup stage was a no-op deferring to “the wave that owns pattern projection” — which never landed. Only the Redis-gated pipeline orchestrator’s S3 stage projected loops→patterns, and the auto-infer path is gated on a trained workspace model. Confirmed live: a run over 250 real issues succeeded with loops=42 — and hazard_fit fitted 0 patterns (it fits per-pattern), GET /patterns returned [], and predictions/recommendations/financials all stayed empty. Fixed by implementing real_criticality_rollup: find-or-create a pattern row per loop keyed by fingerprint.loop_id (the same semantics as the pipeline’s S3 stage, so both entry points converge on the same rows), link member issues, name from taxonomy/top-terms, then roll up criticality per pattern. The run output above — 37 patterns projected, 37 fitted — is this fix working.

Troubleshooting

Symptoms → causes, distilled from the real debugging that produced this guide:

Symptom	Likely cause	Check / fix
Engine run succeeds but `GET /patterns` is `[]`	Running a backend without the pattern-projection fix (bug 6), or issues were never embedded (bug 4)	`criticality_rollup` counters in `stages[]` should show `patterns_projected > 0`
Run reports `partition_degenerate: true` / “100% singletons”	Issues reached the engine with no (or all-identical) embeddings	Confirm forge ran: issue detail should show `concern`/`content_hash` or `5c_abstained:*` in `derived_flags`
Every extraction abstains; log says `provider call failed:` (blank reason)	Transport failure whose exception stringifies to empty — pool/read timeout under burst load	Blank-reason lines are timeouts, not model output; bound client concurrency (fixed in bug 5)
Every LLM call answers instantly but output is canned/mock-looking	`LLM_PROVIDER_ORDER` didn’t resolve a real provider and fell through to `mock`	`GET /v1/ai/providers` — the first entry in `order` must be `ollama` with `available: true`
LLM calls return empty text with `finish_reason: length`	Thinking-mode tokens ate the whole `max_tokens` budget	Keep `OLLAMA_THINK=false`; if you enable it, budgets must be ~2,000+ tokens for short JSON tasks
Calls take 40-50s each instead of 2-5s	qwen3 thinking mode is on (or you’re on the OpenAI-compat endpoint, which can’t turn it off)	`OLLAMA_THINK=false` via the native-API provider; verify with a single timed call
Some issues never show `concern` after forge “completes”	Not stuck — the 5C extraction honestly abstained on that narrative	Look for `5c_abstained:concern` in the issue’s `derived_flags`; the issue still embedded and clusters
Hundreds of review-queue items right after first ingest	Conformal cold-start (no calibration items yet ⇒ documented default threshold) — expected	Resolve items; `GET /ai/calibration/{task}` shows the set growing and coverage tracking begin
Server aborts at startup complaining about identity provider or memory store	A repo `.env` pinning `ENVIRONMENT=production` is shadowing your profile	Export `ENVIRONMENT=development AUTH_PROVIDER=none` in the launch env (process env beats `.env`)

Production notes

Postgres + pgvector dimension. The repo’s pgvector column is vector(384) (db/migrations/0009_embeddings_pgvector.sql), sized for the default TEI model (BAAI/bge-small-en-v1.5, 384-dim). qwen3-embedding:0.6b returns 1024-dim vectors — that will not fit today. Two honest paths: (a) write an additive migration to a dimension-flexible embedding store if your schema can support it cleanly, or (b) run local-Qwen workflows against the in-memory store (as this guide does) and reserve Postgres persistence for a TEI-compatible 384-dim embedding model. Do not silently truncate or pad vectors to force a fit — that corrupts every downstream similarity computation. TEI vs. Ollama for embeddings. HuggingFace TEI is the other supported embedding provider (EMBEDDINGS_PROVIDER=tei). TEI is purpose-built for batched embedding throughput and is the better choice for production ingestion volume; Ollama’s embedding endpoint is convenient for local development but was not designed for high-throughput batch encoding. Model size tradeoffs. This guide uses qwen3:4b for every task tier (reasoning/balanced/fast) — simplest to reason about for a single-model local profile. On constrained hardware, qwen3:1.7b (set OLLAMA_MODEL=qwen3:1.7b) is meaningfully faster and lighter (~1.4GB vs ~3.5GB resident) with a real quality tradeoff on harder extraction/classification tasks; qwen3:0.6b exists but is a stretch for structured extraction. Benchmark against your own data before committing — with thinking already disabled, model size is the second lever, not the first. The thinking-mode tax — the single biggest local-throughput lever. qwen3-family models emit a hidden <think>...</think> reasoning chain by default on every request, including 5-line JSON extractions. Measured on this machine (qwen3:4b, Ollama 0.23.2): with thinking on, one extraction = ~1,850 completion tokens / ~44s; with think: false plus format: "json" (grammar-constrained decoding), the same extraction = 103 tokens / 2.6s — a ~17x speedup, and the difference between a ~9-hour and a ~25-minute full-dataset onboarding. Three operational facts worth knowing, all verified directly:

The think field only works on Ollama’s native /api/chat endpoint — the OpenAI-compatible /v1/chat/completions layer silently ignores it, and qwen3:4b ignored the /no_think prompt-suffix soft switch entirely. This is why Causeloop’s Ollama provider uses the native API.
think: false alone is not enough for structured tasks: qwen3:4b still narrates its chain-of-thought in plain content before the JSON. format: "json" constrains generation from the first token and eliminates it.
The tradeoff is real but small for closed-vocabulary tasks: thinking mainly buys accuracy on open-ended reasoning (root-cause synthesis, narration), not on “score these 4 labels” prompts. Causeloop defaults to OLLAMA_THINK=false globally; flip it to true if you observe quality problems on reasoning-tier tasks and can afford the latency.

Determinism and temperature. The Ollama provider pins temperature: 0 and seed: 42 (OLLAMA_TEMPERATURE/OLLAMA_SEED), so decoding is deterministic for a fixed prompt on a fixed model build — this makes the seed_supported=True flag in app/capabilities/registration.py true in practice. Two honest caveats: (a) determinism holds per model build — pulling a new qwen3:4b blob or changing Ollama’s numeric backend can change outputs for the same seed; (b) the replay determinism gate above does not depend on LLM decoding determinism at all — it re-executes from the run’s pinned manifest, and extraction outputs are content-hash-cached in the llm_cache table (app/engine/forge/conformal_extract.py), so replays read the original run’s frozen outputs rather than re-rolling the model.

Concepts

Provenance, abstention, hazard models, and configs — the vocabulary this guide assumes.

Onboarding Tutorial

The UI-driven onboarding wizard that wraps this same lifecycle for a new customer workspace.

Local Development

General local setup, without the Qwen-specific profile.

API Reference

Full endpoint reference for every call made in this guide.

​Concept: what “training” means in Causeloop

​Prerequisites

​Hardware notes

​Step 1 — Configure the local Qwen profile

​Step 2 — Convert and ingest the dataset

​Watching forge + extraction run

​Step 3 — Run the engine (“train”)

​Step 4 — Inference: read the trained model

​Patterns

​Hazard forecast + counterfactual

​Predictions

​Backtest + accuracy

​Recommendation + business case

​Step 5 — Review queue: what Qwen abstained on

​Step 6 — Replay: prove the run is deterministic

​Real bugs found and fixed while writing this guide

​Troubleshooting

​Production notes

​Related

Concepts

Onboarding Tutorial

Local Development

API Reference

Concept: what “training” means in Causeloop

Prerequisites

Hardware notes

Step 1 — Configure the local Qwen profile

Step 2 — Convert and ingest the dataset

Watching forge + extraction run

Step 3 — Run the engine (“train”)

Step 4 — Inference: read the trained model

Patterns

Hazard forecast + counterfactual

Predictions

Backtest + accuracy

Recommendation + business case

Step 5 — Review queue: what Qwen abstained on

Step 6 — Replay: prove the run is deterministic

Real bugs found and fixed while writing this guide

Troubleshooting

Production notes

Related