> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Research datasets

> What data the research workbench actually ships with versus what has to be downloaded or generated: the tracked synthetic issue set, the helpdesk/BPIC13 event-log adapters, and the public-dataset acquisition program.

The research workbench's dataset story has three distinct tiers: a small set of files committed directly to the repository, a set of real-world event-log and public-incident sources that a fresh clone does **not** have and must fetch or convert, and a documented plan for adapters that don't exist as runnable code yet. Source of truth: `docs/DATASETS.md`, verified here against the actual contents of `data/` and the adapter code in `packages/causegraph/src/causegraph/datasets.py` and `workers/ingest/acquire_public_datasets.py`.

<Note>
  Don't confuse this page's ingestion machinery with the product's. `services/ingest/connectors/` (`FileUploadConnector`, `S3Connector`, `DatabaseConnector`, `KafkaConnector`) is real, deployed tenant-ingest code used by `services/api/sources_api.py` — a different system entirely from everything on this page, which is research-only: `workers/ingest/acquire_public_datasets.py` (public dataset acquisition) and `causegraph.onboarding.connectors` (the legacy, file-based `/clients` connector registry documented in [API surface](/research/api-surface#file-based-clients-onboarding-api)).
</Note>

## What ships in the repository

Only one dataset ships as data files tracked in Git, under `docs/`:

| File                                                                                                                       | Contents                                                                                                                                                                                           |
| -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `docs/issue-set-2-records.json`                                                                                            | 250 synthetic banking/enterprise issue records — the same 250-issue set the [MVP1 pipeline](/research/mvp1-pipeline) analyzes. Each record's internal `dataset` field is `cause_loop_issue_set_2`. |
| `docs/issue-set-2-embeddings.json`, `docs/issue-set-2-embedding-clustering.json`, `docs/issue-set-2-local-clustering.json` | Precomputed embeddings and clustering results from earlier exploration passes over that same 250-record set                                                                                        |
| `docs/issue-set-2-llm-theme-summaries.json` / `.md`                                                                        | LLM-generated theme summaries from an earlier run                                                                                                                                                  |
| `docs/issue-set-2-cluster-records.xlsx` / `.inspect.ndjson`                                                                | An earlier clustering workbook and its flattened inspection dump                                                                                                                                   |
| `docs/issue-set-2-clustering-report.md`                                                                                    | A prose write-up of an earlier clustering pass                                                                                                                                                     |

`data/raw/` contains only a `.gitkeep` placeholder (no real event-log or public-incident files are committed), and `data/clients/` holds whatever local, non-representative manifests a developer happens to have created while exercising the file-based `/clients` API locally. `data/artifacts/state/` is the one exception: it ships tracked, non-empty research-artifact bookkeeping JSON (`manifest.json`, `artifact_manifests.json`, `runs.json`, `task_run_artifacts.json`, `product_control.json`) — pipeline-run and artifact-registry state, not dataset records.

<Warning>
  `outputs/mvp1/` (the MVP1 relational pipeline's report/workbook output directory) is likewise untracked by convention, though not covered by `.gitignore` — see [MVP1 pipeline](/research/mvp1-pipeline#output-layout).
</Warning>

## Event-log adapters: Helpdesk and BPIC13

The process-discovery and conformance task family (see [API surface](/research/api-surface#process-discovery-and-conformance)) was built against two real event logs, chosen specifically because they carry case IDs, activities, timestamps, status transitions, and handoffs — the shape process mining needs:

* **Helpdesk** — expected at `data/raw/helpdesk/finale.csv`. `parse_helpdesk_csv` (`causegraph/datasets.py`) reads it with a tolerant column-name matcher (`case`/`Case ID`/`CaseID`/`ticket`, `activity`/`concept:name`/`task`, and similar aliases for resource and timestamp), so it accepts several common event-log CSV export shapes. `docs/DATASETS.md` records this file at 21,348 event rows importing into 4,580 cases.
* **BPIC13** — expected at `data/raw/bpic13/BPI_Challenge_2013_incidents.xes.gz`. `parse_xes` reads gzip-compressed XES directly (standard event-log XML), walking `trace`/`event` elements and pulling `concept:name`, `org:resource`, and `time:timestamp` attributes.

Neither file is present in a fresh clone; `GET /datasets/status` reports `exists: false` for both under `local_files` until you supply them. `POST /datasets/import` accepts a `dataset` of `helpdesk` or `bpic13` plus a `file_path`, or — with `allow_demo_data: true` — falls back to a small built-in synthetic trace (`demo_records` in `causegraph/datasets.py`: 12 Helpdesk-style ticket events, 13 BPIC13-style incident events) so the workbench's UI and API paths are still exercisable without either real file. `dataset_status()` reports these fallback sizes directly under `dev_only.demo_fallback_records`.

`workers/ingest/activate_bpic13.py` is the scripted path for activating the *real* BPIC13 dataset once you have the file: it imports it via `/datasets/import`, asserts the expected record counts (`EXPECTED_BPIC13_RECORDS = 65_533`, `EXPECTED_HELPDESK_RECORDS = 21_348`), rebuilds the BM25 index, and proceeds through embedding and clustering — useful as a template for onboarding any other event log the same way.

## Public dataset acquisition

Beyond the two event logs, `workers/ingest/acquire_public_datasets.py` defines a catalog of 19 public safety/cyber/reliability datasets (`PUBLIC_DATASET_SOURCES`) that feed the narrative-normalization pipeline (`Narrative`, `EvidenceChunk`, `ExtractedObservation`) rather than the process-mining task family. Each source records a download URL, an expected raw filename, a converter name (or `None` if unimplemented), and a `catalog_status` field that in code takes exactly one of two values: `adapter_ready` (10 of the 19 sources — a real converter exists: OSHA, MAUDE, NHTSA, VCDB, CISA KEV, FIRST EPSS, NVD Recent, NASA C-MAPSS, an operational-risk proxy, and TAWOS Jira) or `catalog_only` (the remaining 9 — Public Jira, GitBugs, BugHub, GitHub-issue HF, NTSB CAROL, NTSB Safety Recommendations, FDA Warning Letters, FDA Close-Out Letters, and FDA 483 observations — a planning contract with no converter or local dump wired yet). Two more per-source flags matter alongside that field: `download_requires_manual_file` (`True` only for TAWOS Jira, whose SQL dump can't be fetched by URL alone) and `proxy_only` (`True` for 9 sources — the ones DATASETS.md is explicit are remediation/outcome-language proxies, not full CAPA records).

`docs/DATASETS.md` layers a finer, four-step readiness ladder — `loaded`, `dataset_gated`, `manual_file_required`, `catalog_only` — on top of that binary code field, by combining `catalog_status` with how far a given workspace has actually gotten (has it imported, BM25-indexed, and embedded every record for that family yet?) and whether `download_requires_manual_file` applies. `GET /datasets/public/sources` (see [API surface](/research/api-surface#datasets-and-records)) is what actually reports the workspace-progress half of that ladder: per source, whether the raw file is downloaded, canonicalized, imported (with a record count), indexed, embedded, and included in the active theme, plus aggregate counts by `catalog_status`, `source_family`, and `lifecycle_role`. As of `docs/DATASETS.md` (dated 2026-06-14), the workspace that document describes had completed import and BM25 indexing for 3,524,305 total records across Helpdesk, BPIC13, and the ten `adapter_ready` public families, with 1,311,215 embedded — NHTSA intentionally left mostly unembedded (2,048 of 2,215,138 vectorized) pending a separate decision to expand embedding scope. Treat those specific counts as a point-in-time snapshot from that document, not a live guarantee about the state of any given local environment — run `GET /datasets/public/sources` yourself to see current state.

To (re)run acquisition for one or more sources:

```bash theme={null}
UV_CACHE_DIR=/tmp/uv-cache uv run python workers/ingest/acquire_public_datasets.py \
  --sources all \
  --download \
  --convert \
  --import-data \
  --index \
  --embed
```

`--sources` also accepts a comma-separated subset of catalog keys (`osha`, `maude`, `nhtsa`, `vcdb`, `cisa_kev`, `epss`, `nvd_recent`, `nasa_cmapss`, `operational_risk`, `tawos_jira`, and others) instead of `all`.

<Tip>
  Large public downloads (NHTSA, TAWOS, MAUDE) can be slow or blocked by network policy in some environments. `docs/DATASETS.md` notes a 2026-06-07 admin run where only CISA KEV and NVD Recent completed end to end for that reason — that's an operational constraint of the environment the acquisition ran in, not a defect in the worker.
</Tip>

## What public data does not provide

None of the public sources above give Causeloop a complete enterprise remediation lifecycle end to end:

```text theme={null}
issue -> root cause -> CAPA -> evidence requirement -> closure package -> effectiveness check -> outcome
```

Public incident/safety/cyber datasets are strong for the *front half* of that chain (incident narrative, root-cause language, severity signal) but weak or absent on the *back half* (a real CAPA record, its evidence trail, and a verified effectiveness outcome). `docs/DATASETS.md` is explicit that several catalog entries — NTSB Safety Recommendations, FDA Warning/Close-Out letters, FDA 483 observations, openFDA MAUDE, NHTSA safety data, OSHA Severe Injury Reports — are **remediation and outcome proxies only** and must not be presented in product UI as full CAPA/effectiveness datasets. Closing that gap requires either real customer data or a synthetic CAPA/control/evidence overlay — which is exactly what the tracked `docs/issue-set-2-records.json` set and the MVP1 pipeline's CAP suggestions are a synthetic stand-in for today.

## Dataset-to-task coverage

A useful mental model when picking a dataset for a piece of research work — which public or event-log source is actually relevant to which task family:

| Dataset family                                                                                             | Relevant task coverage                                                                                                                      |
| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Helpdesk, BPIC13 incidents                                                                                 | Process discovery, conformance/deviation, predictive monitoring baselines, anomaly detection, process-stub stitching, related-issue linking |
| TAWOS Jira / Public Jira / GitBugs / BugHub / GitHub-issue HF datasets (catalog-only or manual-file today) | True issue text, labels, comments, duplicate detection, related/blocking links, workflow/history                                            |
| CISA KEV, NVD Recent                                                                                       | Cyber vulnerability prioritization, narrative extraction, exploitability/triage baselines                                                   |
| OSHA, MAUDE, NHTSA, VCDB (and catalog-only NTSB/FAA/FDA families)                                          | Narrative failure extraction, safety/root-cause classification, evidence/control extraction, regulatory/control language                    |
| NASA C-MAPSS                                                                                               | Maintenance-failure and remaining-useful-life provider                                                                                      |

`docs/issue-set-2-records.json` (the tracked synthetic set) sits outside this table — it's a purpose-built issue-intelligence dataset for the MVP1 severity/clustering/fishbone pipeline, not an event log or public-incident narrative source.

## Planned adapters and implementation order

`docs/DATASETS.md` also records a prioritized queue for the next true issue/proxy adapters, reasoning from which sources most directly extend the issue-and-remediation task family rather than just adding more incident narratives:

1. **TAWOS Jira** — the first practical true-issue-text adapter target: issue text, comments, links, changelog/history, components, versions, and sprints. Its converter is implemented (`catalog_status: adapter_ready`), but it needs a manually supplied SQL dump (`download_requires_manual_file`) before conversion can run.
2. **The Public Jira Dataset** — a larger-scale Jira follow-on, planned once TAWOS is stable and resumable.
3. **GitBugs, BugHub, and GitHub-issue Hugging Face datasets** — duplicate/related-bug/triage benchmarks, planned once the issue-record contract from TAWOS is proven.
4. **NTSB CAROL and NTSB Safety Recommendations** — remediation/outcome proxy adapters (incident → probable cause → recommendation → action status/outcome).
5. **FDA Warning Letters, FDA Close-Out Letters, FDA 483 observations, openFDA MAUDE, NHTSA safety families, and OSHA Severe Injury Reports** — additional proxy families, all explicitly remediation/control-language proxies rather than full CAPA records.

Every family in this queue is required to keep raw/canonical/imported/indexed/embedded counts honestly at zero (`catalog_status: catalog_only`, or `adapter_ready` with `download_requires_manual_file` still pending) until a real converter, a local source file, and a completed import/index/embed pass all exist for that family in the same run or a resumable follow-up — `docs/DATASETS.md` is explicit that catalog-only rows must never imply runnable availability in the normal product UI.

## Related

* [MVP1 pipeline](/research/mvp1-pipeline)
* [API surface](/research/api-surface)
* [Research workbench overview](/research/overview)
* [Sources](/features/sources)