Skip to main content
The research workbench’s dataset story has three distinct tiers: a small set of files committed directly to the repository, a set of real-world event-log and public-incident sources that a fresh clone does not have and must fetch or convert, and a documented plan for adapters that don’t exist as runnable code yet. Source of truth: docs/DATASETS.md, verified here against the actual contents of data/ and the adapter code in packages/causegraph/src/causegraph/datasets.py and workers/ingest/acquire_public_datasets.py.
Don’t confuse this page’s ingestion machinery with the product’s. services/ingest/connectors/ (FileUploadConnector, S3Connector, DatabaseConnector, KafkaConnector) is real, deployed tenant-ingest code used by services/api/sources_api.py — a different system entirely from everything on this page, which is research-only: workers/ingest/acquire_public_datasets.py (public dataset acquisition) and causegraph.onboarding.connectors (the legacy, file-based /clients connector registry documented in API surface).

​
What ships in the repository

Only one dataset ships as data files tracked in Git, under docs/: data/raw/ contains only a .gitkeep placeholder (no real event-log or public-incident files are committed), and data/clients/ holds whatever local, non-representative manifests a developer happens to have created while exercising the file-based /clients API locally. data/artifacts/state/ is the one exception: it ships tracked, non-empty research-artifact bookkeeping JSON (manifest.json, artifact_manifests.json, runs.json, task_run_artifacts.json, product_control.json) — pipeline-run and artifact-registry state, not dataset records.
outputs/mvp1/ (the MVP1 relational pipeline’s report/workbook output directory) is likewise untracked by convention, though not covered by .gitignore — see MVP1 pipeline.

​
Event-log adapters: Helpdesk and BPIC13

The process-discovery and conformance task family (see API surface) was built against two real event logs, chosen specifically because they carry case IDs, activities, timestamps, status transitions, and handoffs — the shape process mining needs:
  • Helpdesk — expected at data/raw/helpdesk/finale.csv. parse_helpdesk_csv (causegraph/datasets.py) reads it with a tolerant column-name matcher (case/Case ID/CaseID/ticket, activity/concept:name/task, and similar aliases for resource and timestamp), so it accepts several common event-log CSV export shapes. docs/DATASETS.md records this file at 21,348 event rows importing into 4,580 cases.
  • BPIC13 — expected at data/raw/bpic13/BPI_Challenge_2013_incidents.xes.gz. parse_xes reads gzip-compressed XES directly (standard event-log XML), walking trace/event elements and pulling concept:name, org:resource, and time:timestamp attributes.
Neither file is present in a fresh clone; GET /datasets/status reports exists: false for both under local_files until you supply them. POST /datasets/import accepts a dataset of helpdesk or bpic13 plus a file_path, or — with allow_demo_data: true — falls back to a small built-in synthetic trace (demo_records in causegraph/datasets.py: 12 Helpdesk-style ticket events, 13 BPIC13-style incident events) so the workbench’s UI and API paths are still exercisable without either real file. dataset_status() reports these fallback sizes directly under dev_only.demo_fallback_records. workers/ingest/activate_bpic13.py is the scripted path for activating the real BPIC13 dataset once you have the file: it imports it via /datasets/import, asserts the expected record counts (EXPECTED_BPIC13_RECORDS = 65_533, EXPECTED_HELPDESK_RECORDS = 21_348), rebuilds the BM25 index, and proceeds through embedding and clustering — useful as a template for onboarding any other event log the same way.

​
Public dataset acquisition

Beyond the two event logs, workers/ingest/acquire_public_datasets.py defines a catalog of 19 public safety/cyber/reliability datasets (PUBLIC_DATASET_SOURCES) that feed the narrative-normalization pipeline (Narrative, EvidenceChunk, ExtractedObservation) rather than the process-mining task family. Each source records a download URL, an expected raw filename, a converter name (or None if unimplemented), and a catalog_status field that in code takes exactly one of two values: adapter_ready (10 of the 19 sources — a real converter exists: OSHA, MAUDE, NHTSA, VCDB, CISA KEV, FIRST EPSS, NVD Recent, NASA C-MAPSS, an operational-risk proxy, and TAWOS Jira) or catalog_only (the remaining 9 — Public Jira, GitBugs, BugHub, GitHub-issue HF, NTSB CAROL, NTSB Safety Recommendations, FDA Warning Letters, FDA Close-Out Letters, and FDA 483 observations — a planning contract with no converter or local dump wired yet). Two more per-source flags matter alongside that field: download_requires_manual_file (True only for TAWOS Jira, whose SQL dump can’t be fetched by URL alone) and proxy_only (True for 9 sources — the ones DATASETS.md is explicit are remediation/outcome-language proxies, not full CAPA records). docs/DATASETS.md layers a finer, four-step readiness ladder — loaded, dataset_gated, manual_file_required, catalog_only — on top of that binary code field, by combining catalog_status with how far a given workspace has actually gotten (has it imported, BM25-indexed, and embedded every record for that family yet?) and whether download_requires_manual_file applies. GET /datasets/public/sources (see API surface) is what actually reports the workspace-progress half of that ladder: per source, whether the raw file is downloaded, canonicalized, imported (with a record count), indexed, embedded, and included in the active theme, plus aggregate counts by catalog_status, source_family, and lifecycle_role. As of docs/DATASETS.md (dated 2026-06-14), the workspace that document describes had completed import and BM25 indexing for 3,524,305 total records across Helpdesk, BPIC13, and the ten adapter_ready public families, with 1,311,215 embedded — NHTSA intentionally left mostly unembedded (2,048 of 2,215,138 vectorized) pending a separate decision to expand embedding scope. Treat those specific counts as a point-in-time snapshot from that document, not a live guarantee about the state of any given local environment — run GET /datasets/public/sources yourself to see current state. To (re)run acquisition for one or more sources:
--sources also accepts a comma-separated subset of catalog keys (osha, maude, nhtsa, vcdb, cisa_kev, epss, nvd_recent, nasa_cmapss, operational_risk, tawos_jira, and others) instead of all.
Large public downloads (NHTSA, TAWOS, MAUDE) can be slow or blocked by network policy in some environments. docs/DATASETS.md notes a 2026-06-07 admin run where only CISA KEV and NVD Recent completed end to end for that reason — that’s an operational constraint of the environment the acquisition ran in, not a defect in the worker.

​
What public data does not provide

None of the public sources above give Causeloop a complete enterprise remediation lifecycle end to end:
Public incident/safety/cyber datasets are strong for the front half of that chain (incident narrative, root-cause language, severity signal) but weak or absent on the back half (a real CAPA record, its evidence trail, and a verified effectiveness outcome). docs/DATASETS.md is explicit that several catalog entries — NTSB Safety Recommendations, FDA Warning/Close-Out letters, FDA 483 observations, openFDA MAUDE, NHTSA safety data, OSHA Severe Injury Reports — are remediation and outcome proxies only and must not be presented in product UI as full CAPA/effectiveness datasets. Closing that gap requires either real customer data or a synthetic CAPA/control/evidence overlay — which is exactly what the tracked docs/issue-set-2-records.json set and the MVP1 pipeline’s CAP suggestions are a synthetic stand-in for today.

​
Dataset-to-task coverage

A useful mental model when picking a dataset for a piece of research work — which public or event-log source is actually relevant to which task family: docs/issue-set-2-records.json (the tracked synthetic set) sits outside this table — it’s a purpose-built issue-intelligence dataset for the MVP1 severity/clustering/fishbone pipeline, not an event log or public-incident narrative source.

​
Planned adapters and implementation order

docs/DATASETS.md also records a prioritized queue for the next true issue/proxy adapters, reasoning from which sources most directly extend the issue-and-remediation task family rather than just adding more incident narratives:
  1. TAWOS Jira — the first practical true-issue-text adapter target: issue text, comments, links, changelog/history, components, versions, and sprints. Its converter is implemented (catalog_status: adapter_ready), but it needs a manually supplied SQL dump (download_requires_manual_file) before conversion can run.
  2. The Public Jira Dataset — a larger-scale Jira follow-on, planned once TAWOS is stable and resumable.
  3. GitBugs, BugHub, and GitHub-issue Hugging Face datasets — duplicate/related-bug/triage benchmarks, planned once the issue-record contract from TAWOS is proven.
  4. NTSB CAROL and NTSB Safety Recommendations — remediation/outcome proxy adapters (incident → probable cause → recommendation → action status/outcome).
  5. FDA Warning Letters, FDA Close-Out Letters, FDA 483 observations, openFDA MAUDE, NHTSA safety families, and OSHA Severe Injury Reports — additional proxy families, all explicitly remediation/control-language proxies rather than full CAPA records.
Every family in this queue is required to keep raw/canonical/imported/indexed/embedded counts honestly at zero (catalog_status: catalog_only, or adapter_ready with download_requires_manual_file still pending) until a real converter, a local source file, and a completed import/index/embed pass all exist for that family in the same run or a resumable follow-up — docs/DATASETS.md is explicit that catalog-only rows must never imply runnable availability in the normal product UI.