Skip to main content
A tenant’s model is a model_checkpoints row: a versioned snapshot of an embed-then-cluster run over one dataset version, computed by causegraph.onboarding.training.compute_training_checkpoint() and persisted through services/pipeline/training_job.py. Exactly one checkpoint per tenant is ever is_active at a time, and only an active checkpoint’s clusters back the tenant console’s insight collections (see Client onboarding pipeline for where activation sits in the onboarding sequence).

​
Training configuration

Call: PUT /admin/tenants/{tenant_id}/training-config, requires onboarding_admin. The body (TrainingConfigRequest) covers two independent concerns in one payload: The route splits this into two tenant-schema rows in one transaction: the embedding/clustering half merges into settings.key_values["training_config"] (the same JSONB-merge pattern data policy uses), and the LLM half upserts one llm_settings row per tenant (UniqueConstraint on tenant_id — there is exactly one LLM configuration per tenant, not one per use case).

​
Server-side key handling

llm_api_key is write-only: it is never returned by any endpoint, and its absence on a PUT means “keep the existing key” rather than “clear it” — the route only touches encrypted_api_key/api_key_hint when a new key is actually supplied (if body.llm_api_key: ...). When a key is supplied, encrypt_secret() (services/api/secrets.py) encrypts it — Fernet locally, AWS KMS or Azure Key Vault in production, versioned per-envelope so a key rotation never breaks decryption of rows written under the old provider — and secret_hint() computes a last-4-style display hint ("*" * (len-4) + plaintext[-4:], or all-asterisks if 4 characters or fewer). The route’s response includes llm_api_key_hint so the UI can confirm a key is configured (e.g. "**********kj42") without ever re-exposing the key itself. At read time, materialize_insights()’s _load_llm_settings() (services/pipeline/materialize.py) is the only code path that calls decrypt_secret() on this column, and only inside the worker process computing narrative collections — never on any HTTP response.
Choosing embedding_provider="hash" (the wizard’s fixed default) makes embedding fully deterministic and offline — no network call, no external dependency, identical output for identical input text every time, which is why it’s the only option exposed as an editable field in the staff UI today (the ollama/openai/local_api providers are reachable through the API but not surfaced as select options in either onboarding UI). Switching clusterer changes the training job’s idempotency key (via the config hash), so toggling between hdbscan and mixed_graph_greedy_modularity and training again always produces a genuinely new checkpoint version rather than replaying a cached one.

​
The train job

Call: POST /admin/tenants/{tenant_id}/train, 202. Requires a committed dataset (409 otherwise). The job’s idempotency key includes a hash of the current training config (sha256(json.dumps(config, sort_keys=True))[:16]), so re-training after changing the clustering algorithm produces a distinct job rather than deduplicating against a stale run; retry_terminal=True lets a previously-failed training job be retried from the same enqueue call. run_training_job() (services/pipeline/training_job.py), executed by the product-worker:
  1. Loads every DatasetRecord.payload for the tenant’s latest Dataset version, and the tenant’s TrainingConfig from settings.key_values.
  2. Calls compute_training_checkpoint() (causegraph/onboarding/training.py) — the same storage-agnostic embed→cluster core the legacy file-based /clients research path uses, so the algorithm has exactly one implementation:
    • Resolves an embedding provider (resolve_embedding_provider()) and embeds every record’s text.
    • Builds a BM25 lexical index over the same texts; the graph clusterer (mixed_graph_greedy_modularity) additionally computes per-record BM25 neighbor lists to score graph edges.
    • Clusters (HdbscanClusterer / KMeansClusterer / MixedGraphClusterer), producing named themes with member record IDs and a centroid (mean embedding vector) per theme.
    • Computes noise_pct (percentage of records assigned to no theme) and, when at least two themes exist and more records than themes are labeled, a silhouette score (sklearn.metrics.silhouette_score); otherwise silhouette is null.
  3. lock_tenant_model_versions() takes an advisory lock so two concurrent training runs for the same tenant can’t race on the next version number; the new ModelCheckpoint row gets version = max(existing) + 1, algorithm and embedding_provider copied from the checkpoint dict, metrics holding the entire checkpoint payload (themes, assignments, and the metrics sub-object together — the checkpoints-listing endpoint later re-extracts just the metrics sub-object for display), and is_active=false.
A freshly-trained checkpoint is never automatically active — activation is a separate, explicit step.

​
Checkpoint listing

Call: GET /admin/tenants/{tenant_id}/checkpoints — every checkpoint, newest version first:

​
Activation semantics

Call: POST /admin/tenants/{tenant_id}/activate with {checkpoint_version, share_onboarding_outputs}. An unknown version is 404. In one tenant transaction: every checkpoint’s is_active clears, then exactly the named version is set true — activation is a hard swap, never additive. If a dataset exists, a materialize_insights job enqueues for that checkpoint against the tenant’s latest dataset version (not necessarily the version the checkpoint was trained on), and the matching insight_collections rows for that (model_version, dataset_version) pair get their client_visible/source fields updated to the operator’s share_onboarding_outputs choice. On the control-plane side, lifecycle_state moves to review (see Client onboarding pipeline) and a checkpoint_activated audit row records the version and sharing decision. materialize_insights() itself computes all ten insight collections for that checkpoint/dataset pair — five deterministic, model-backed collections (summary, themes, issues, severity, lens) that always attempt, and five optional LLM-backed narrative collections (fishbone, theme_summaries, narratives, recommendations, cause_remediation) that are skipped, not failed, with a recorded reason ("No LLM configured for this tenant." or the provider’s own unavailability message) whenever no LLM is configured or it’s unreachable — so a tenant with llm_provider="disabled" still gets a fully usable portal on the five model-backed collections alone.

​
Retrain recommendation

Call: GET /admin/tenants/{tenant_id}/retrain-recommendation — read-only, computed on every request from two already-stored columns, not a new metric: the active checkpoint’s training-time dataset_id → row_count, and the tenant’s current latest dataset.row_count.
The two thresholds are named constants in onboarding_api.py: _FULL_RETRAIN_GROWTH_THRESHOLD = 0.30, _INCREMENTAL_GROWTH_THRESHOLD = 0.10. Because every ingest/train cycle writes an independent, immutable dataset version rather than mutating one in place, comparing row_count across two dataset versions is a direct, auditable read on how much underlying data has changed since the model was last trained — not an estimate. The tenant detail page renders this as a DriftBanner directly under the Train step (see Staff portal).

​
Model info in the tenant directory

The tenant directory’s Model column (GET /admin/tenants and GET /admin/tenants/{tenant_id}) is a best-effort join computed by _load_model_info(): it opens a tenant-schema transaction, reads llm_settings.provider/.model (formatted as "{provider}/{model}", or null if no model is configured) and the most recent model_checkpoints.created_at, and returns both. Any failure here — most commonly a draft tenant with no schema yet — is swallowed and reported as (null, null) rather than a 500, the same try/except-and-swallow pattern every other per-tenant best-effort read in this router uses, so one tenant’s read error never breaks the whole registry list.