Skip to main content
The MVP1 relational pipeline turns an immutable issue source into a governed, two-tier clustering workbook. It is the audited relational acceptance path for the 250-row banking and HomeCare acceptance datasets — the most heavily specified piece of the research surface, with its own migrations, its own content-addressed PostgreSQL runtime, and its own acceptance test suite. Source of truth: docs/MVP1_RELATIONAL_PIPELINE.md and workers/run_banking_relational_e2e.py.
This is a worker-driven batch pipeline, not an HTTP endpoint. It is not exposed by any /mvp1 route — the interactive /mvp1/pipeline/run and /mvp1/issue-intelligence/latest routes documented in API surface use a separate, simpler interactive operational service and do not promise this pipeline’s nine-sheet workbook or its issue-routed fishbone output. Treat the two as related but distinct: same underlying strict-analysis idea, different implementations, different output contracts.

​
Pipeline stages

Each issue moves through the following stages, from raw text to a validated, immutable report and workbook: The LLM generates all eight severity factors for each issue in one structured batch response. It does not generate issue type, product area, cost estimates, time estimates, or corrective actions for this strict profile — those enriched attributes exist elsewhere in the research surface but are never used for cluster membership here.

​
Data and transfer boundary

Only de-identified fields ever leave the process, and different stages see progressively less: Owner, customer/account identifiers, status, product area, and raw tags are excluded from every external call. The OPENAI_API_KEY used for these calls is loaded from the process environment (or an ignored local env file); it is never written to reports, workbooks, logs, or PostgreSQL payloads.

​
The explicit transfer approval mechanism

A live run requires two things together — a boolean flag is not sufficient on its own:
  1. --approve-openai-llm on the command line.
  2. --openai-authorization "<exact statement>" — a literal, dataset-specific string the runner compares byte-for-byte against a constant baked into the script. For the banking dataset, that constant is defined directly in workers/run_banking_relational_e2e.py:
_async_main in that same script asserts both conditions before doing anything else — args.approve_openai_llm must be truthy, and args.openai_authorization must equal BANKING_OPENAI_AUTHORIZATION exactly. Passing the flag without the exact string (or a paraphrase of it) fails fast before any network call, migration, or database write. The HomeCare entrypoint (workers/run_homecare_relational_e2e.py) enforces the same two-part mechanism with its own dataset-specific authorization sentence.
This is one of the explicit human-approval boundaries in AGENTS.md. A code agent must never construct or pass this authorization string on a user’s behalf — the approval has to come from the user’s own explicit, in-the-moment decision to incur OpenAI cost and transfer data, not be inferred from a task description.
The runner also asserts its own client configuration before running, so a silent drift in batch size or model can’t slip through unnoticed: the embedding provider must resolve to text-embedding-3-small; the severity client must use batch_size=8 and max_concurrency=8; the cluster-label and fishbone clients must both use max_concurrency=8.

​
Content-addressed reuse in PostgreSQL

Alembic migrations 20260711_0002 through 20260714_0007 (migrations/versions/) create a normalized relational runtime purpose-built for this pipeline:
  • Pipeline runs, steps, and append-only events
  • Leased LLM batches, attempts, and batch items
  • One severity result plus exactly eight factor rows per issue
  • pgvector metadata and vectors, keyed by exact input hash
  • Two-tier clusters, with one membership per issue per tier
  • Content-addressed OpenAI cluster-label results and their per-run assignments
  • Content-addressed issue-route results, canonical leaf allocations, and per-run assignments
  • Content-addressed path-synthesis results, cause groups, exact evidence partitions, and per-run/variant assignments
  • Immutable report and artifact publications, with hashes and sizes
Cache identity includes dataset, canonical issue or cluster identity, exact input hash, provider, model, prompt/schema contract, and required provenance. Each validated API response is committed immediately in its own small transaction, so a retry resumes only the work that’s actually missing — an interrupted run never has to redo completed OpenAI calls. This is why a rerun on unchanged input is cheap and deterministic: an unchanged rerun must report
The immutable report bytes are reproduced exactly, and a previously validated workbook is reused whenever the report hash and workbook hash are both unchanged — avoiding exporter ZIP-metadata churn and unnecessary artifact conflicts on a rerun that changed nothing. packages/causegraph/src/causegraph/storage/mvp1_postgres.py (Mvp1PostgresRepository) is the transactional repository behind all of this; packages/causegraph/src/causegraph/providers/mvp1_llm.py, mvp1_cluster_labels.py, and mvp1_fishbone.py are the batched OpenAI clients for severity, cluster labeling, and issue routing/path synthesis respectively.

​
Reviewed Excel workbook contract

The pipeline’s output artifact is a reviewed nine-sheet workbook, built by workers/artifacts/build_mvp1_clustering_workbook.mjs (a Node script that imports the bundled @oai/artifact-tool package — see Research modes for the local Node-path setup this requires):
  1. Overview
  2. Tier 1 Clusters
  3. Tier 2 Clusters
  4. Issue Assignments
  5. Fishbone Allocations
  6. Fishbone Paths
  7. Cause & Remediation
  8. Severity Matrix
  9. LLM Audit
The builder validates the strict analysis contract before it will export: exact issue/factor coverage, one membership per issue per tier, one canonical route per issue, all three allocation variants (hard-hard, soft-hard, soft-soft), 100% per-issue allocation weights, weighted-issue reconciliation, exact cause-evidence coverage, provider provenance, formulas, sheet ranges, and post-export importability. It also renders every sheet for visual QA and includes a formula-backed stacked bar chart of weighted issue equivalents by variant and top-level bone. For a 250-row dataset, a complete acceptance run reconciles at least: 250 exact-input severity issues, 2,000 severity factor rows (8 per issue), 250 valid OpenAI embedding vectors, 250 Tier 1 memberships and 250 Tier 2 memberships, one label assignment and one unique response ID per cluster, 250 canonical issue routes and exactly 250 hard-hard allocations, 100% allocation weight for every issue under every variant, one path analysis per populated variant/Tier-1/exact-leaf path, exact cause-group evidence coverage for every path membership, and one published report plus the workbook and all nine sheet previews.

​
Dataset entrypoints

Both entrypoints share the same underlying runner logic (workers/run_homecare_relational.py’s RelationalHomeCareRunner, despite the module’s name — it’s the dataset-neutral relational execution engine both scripts drive) and apply Alembic before executing.
The banking source defaults to the tracked synthetic snapshot at docs/issue-set-2-records.json — 250 records, no external file required:
The same loader also accepts the original workbook via --source, but a fresh clone does not need it.
The 250-record file’s own internal dataset field is cause_loop_issue_set_2 — the same identifier the interactive /mvp1 product routes use as their dataset_id default (see docs/MVP1_API_CONTRACT.md). The relational banking runner, however, uses its own Postgres content-addressing identifier, DATASET_ID = "banking_issue_set_250", and its own tenant scope (TENANT_ID = "demo"). These are two different identifiers for two different consumers of the same 250 rows — don’t assume one implies the other when cross-referencing the interactive service and this relational pipeline.

​
Output layout

Output defaults to the repository-relative outputs/mvp1/ directory. It is untracked by convention — nothing in the codebase creates or commits files there — but it is not covered by .gitignore, so nothing structurally prevents a run’s output from being accidentally git added:
Every path — report, workbook, preview directory, workbook builder script, and Node executable — is overridable with the corresponding CLI flag, so a custom local layout never requires editing the script itself.

​
Acceptance checks

Focused regression tests cover the runner, the acceptance contract, checkpoint migration, cluster labeling, fishbone routing, LLM batching, Postgres persistence, and the strict pipeline end to end:

​
Key implementation files