> ## Documentation Index > Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt > Use this file to discover all available pages before exploring further. # Training and checkpoints > Per-tenant training configuration and LLM key handling, the train job's embed→cluster algorithm, checkpoint activation semantics, and the retrain-recommendation heuristic. A tenant's model is a `model_checkpoints` row: a versioned snapshot of an embed-then-cluster run over one dataset version, computed by `causegraph.onboarding.training.compute_training_checkpoint()` and persisted through `services/pipeline/training_job.py`. Exactly one checkpoint per tenant is ever `is_active` at a time, and only an active checkpoint's clusters back the tenant console's insight collections (see [Client onboarding pipeline](/features/client-onboarding-pipeline) for where activation sits in the onboarding sequence). ## Training configuration **Call:** `PUT /admin/tenants/{tenant_id}/training-config`, requires `onboarding_admin`. The body (`TrainingConfigRequest`) covers two independent concerns in one payload: | Field | Purpose | | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `embedding_provider` (default `"hash"`) | `hash` \| `ollama` \| `ollama_lan` \| any provider `create_embedding_provider()` supports (e.g. `openai`, `local_api`) | | `embedding_dimension` (default `64`) | Vector width for the deterministic hash provider | | `embedding_base_url` / `embedding_model` | Ollama-only; `None` falls back to `http://localhost:11434` and the provider default model | | `clusterer` (default `"hdbscan"`) | `hdbscan` \| `kmeans` \| `mixed_graph_greedy_modularity` | | `clusterer_config` | Free-form dict passed straight to the clusterer (e.g. `bm25_neighbor_limit` for the graph clusterer) | | `llm_provider`, `llm_model`, `llm_base_url` | Optional narrative-enrichment LLM, disabled by default (`llm_provider="disabled"`) | | `llm_api_key` | Write-only — see below | | `llm_temperature`, `llm_max_concurrency`, `llm_monthly_token_budget` | LLM call tuning; budget is optional (`None` = unbounded) | The route splits this into two tenant-schema rows in one transaction: the embedding/clustering half merges into `settings.key_values["training_config"]` (the same JSONB-merge pattern [data policy](/features/data-policy) uses), and the LLM half upserts one `llm_settings` row per tenant (`UniqueConstraint` on `tenant_id` — there is exactly one LLM configuration per tenant, not one per use case). ### Server-side key handling `llm_api_key` is **write-only**: it is never returned by any endpoint, and its absence on a `PUT` means *"keep the existing key"* rather than *"clear it"* — the route only touches `encrypted_api_key`/`api_key_hint` when a new key is actually supplied (`if body.llm_api_key: ...`). When a key is supplied, `encrypt_secret()` (`services/api/secrets.py`) encrypts it — Fernet locally, AWS KMS or Azure Key Vault in production, versioned per-envelope so a key rotation never breaks decryption of rows written under the old provider — and `secret_hint()` computes a last-4-style display hint (`"*" * (len-4) + plaintext[-4:]`, or all-asterisks if 4 characters or fewer). The route's response includes `llm_api_key_hint` so the UI can confirm *a* key is configured (e.g. `"**********kj42"`) without ever re-exposing the key itself. At read time, `materialize_insights()`'s `_load_llm_settings()` (`services/pipeline/materialize.py`) is the only code path that calls `decrypt_secret()` on this column, and only inside the worker process computing narrative collections — never on any HTTP response. ```bash theme={null} curl -sX PUT http://localhost:18000/admin/tenants/$TENANT_ID/training-config \ -H 'content-type: application/json' -H "x-csrf-token: $CSRF" \ --cookie "cl_admin_session=$SESSION; cl_csrf=$CSRF" \ -d '{ "embedding_provider": "hash", "embedding_dimension": 64, "clusterer": "hdbscan", "llm_provider": "openai_compatible", "llm_model": "gpt-4o-mini", "llm_api_key": "sk-...", "llm_temperature": 0.2, "llm_max_concurrency": 2 }' # -> {"status": "ok", "llm_api_key_hint": "**********ab12"} ``` Choosing `embedding_provider="hash"` (the wizard's fixed default) makes embedding fully deterministic and offline — no network call, no external dependency, identical output for identical input text every time, which is why it's the only option exposed as an editable field in the staff UI today (the `ollama`/`openai`/`local_api` providers are reachable through the API but not surfaced as select options in either onboarding UI). Switching `clusterer` changes the training job's idempotency key (via the config hash), so toggling between `hdbscan` and `mixed_graph_greedy_modularity` and training again always produces a genuinely new checkpoint version rather than replaying a cached one. ## The train job **Call:** `POST /admin/tenants/{tenant_id}/train`, `202`. Requires a committed dataset (`409` otherwise). The job's idempotency key includes a hash of the current training config (`sha256(json.dumps(config, sort_keys=True))[:16]`), so re-training after changing the clustering algorithm produces a distinct job rather than deduplicating against a stale run; `retry_terminal=True` lets a previously-failed training job be retried from the same enqueue call. `run_training_job()` (`services/pipeline/training_job.py`), executed by the `product-worker`: 1. Loads every `DatasetRecord.payload` for the tenant's latest `Dataset` version, and the tenant's `TrainingConfig` from `settings.key_values`. 2. Calls `compute_training_checkpoint()` (`causegraph/onboarding/training.py`) — the same storage-agnostic embed→cluster core the legacy file-based `/clients` research path uses, so the algorithm has exactly one implementation: * Resolves an embedding provider (`resolve_embedding_provider()`) and embeds every record's `text`. * Builds a BM25 lexical index over the same texts; the graph clusterer (`mixed_graph_greedy_modularity`) additionally computes per-record BM25 neighbor lists to score graph edges. * Clusters (`HdbscanClusterer` / `KMeansClusterer` / `MixedGraphClusterer`), producing named themes with member record IDs and a centroid (mean embedding vector) per theme. * Computes `noise_pct` (percentage of records assigned to no theme) and, when at least two themes exist and more records than themes are labeled, a `silhouette` score (`sklearn.metrics.silhouette_score`); otherwise `silhouette` is `null`. 3. `lock_tenant_model_versions()` takes an advisory lock so two concurrent training runs for the same tenant can't race on the next version number; the new `ModelCheckpoint` row gets `version = max(existing) + 1`, `algorithm` and `embedding_provider` copied from the checkpoint dict, `metrics` holding the **entire** checkpoint payload (themes, assignments, and the `metrics` sub-object together — the checkpoints-listing endpoint later re-extracts just the `metrics` sub-object for display), and `is_active=false`. A freshly-trained checkpoint is never automatically active — activation is a separate, explicit step. ## Checkpoint listing **Call:** `GET /admin/tenants/{tenant_id}/checkpoints` — every checkpoint, newest version first: ```json theme={null} [ { "version": 2, "algorithm": "hdbscan", "embedding_provider": "hash", "metrics": { "cluster_count": 14, "noise_pct": 6.5, "silhouette": 0.41 }, "is_active": true, "created_at": "2026-07-30T18:04:12Z" } ] ``` ## Activation semantics **Call:** `POST /admin/tenants/{tenant_id}/activate` with `{checkpoint_version, share_onboarding_outputs}`. An unknown version is `404`. In one tenant transaction: every checkpoint's `is_active` clears, then exactly the named version is set `true` — activation is a hard swap, never additive. If a dataset exists, a `materialize_insights` job enqueues for that checkpoint against the tenant's *latest* dataset version (not necessarily the version the checkpoint was trained on), and the matching `insight_collections` rows for that `(model_version, dataset_version)` pair get their `client_visible`/`source` fields updated to the operator's `share_onboarding_outputs` choice. On the control-plane side, `lifecycle_state` moves to `review` (see [Client onboarding pipeline](/features/client-onboarding-pipeline)) and a `checkpoint_activated` audit row records the version and sharing decision. `materialize_insights()` itself computes all ten insight collections for that checkpoint/dataset pair — five deterministic, model-backed collections (`summary`, `themes`, `issues`, `severity`, `lens`) that always attempt, and five optional LLM-backed narrative collections (`fishbone`, `theme_summaries`, `narratives`, `recommendations`, `cause_remediation`) that are **skipped, not failed**, with a recorded reason (`"No LLM configured for this tenant."` or the provider's own unavailability message) whenever no LLM is configured or it's unreachable — so a tenant with `llm_provider="disabled"` still gets a fully usable portal on the five model-backed collections alone. ## Retrain recommendation **Call:** `GET /admin/tenants/{tenant_id}/retrain-recommendation` — read-only, computed on every request from two already-stored columns, not a new metric: the active checkpoint's training-time `dataset_id → row_count`, and the tenant's current latest `dataset.row_count`. ```json theme={null} { "recommendation": "incremental", "checkpoint_dataset_version": 2, "checkpoint_row_count": 1000, "latest_dataset_version": 3, "latest_row_count": 1150, "growth_pct": 15.0 } ``` | `recommendation` | Trigger | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `no_checkpoint` | No active checkpoint exists yet | | `full_retrain` | The active checkpoint's dataset was itself deleted (`dataset_id` is `NULL` via `ON DELETE SET NULL`), **or** row-count growth since that checkpoint's dataset is ≥ 30% | | `incremental` | Row-count growth is ≥ 10% but \< 30% | | `up_to_date` | The checkpoint's dataset version *is* the latest version (0% growth by definition), or growth is under 10% | The two thresholds are named constants in `onboarding_api.py`: `_FULL_RETRAIN_GROWTH_THRESHOLD = 0.30`, `_INCREMENTAL_GROWTH_THRESHOLD = 0.10`. Because every ingest/train cycle writes an independent, immutable dataset version rather than mutating one in place, comparing `row_count` across two dataset versions is a direct, auditable read on how much underlying data has changed since the model was last trained — not an estimate. The tenant detail page renders this as a `DriftBanner` directly under the Train step (see [Staff portal](/features/staff-portal)). ## Model info in the tenant directory The tenant directory's **Model** column (`GET /admin/tenants` and `GET /admin/tenants/{tenant_id}`) is a best-effort join computed by `_load_model_info()`: it opens a tenant-schema transaction, reads `llm_settings.provider`/`.model` (formatted as `"{provider}/{model}"`, or `null` if no model is configured) and the most recent `model_checkpoints.created_at`, and returns both. Any failure here — most commonly a `draft` tenant with no schema yet — is swallowed and reported as `(null, null)` rather than a `500`, the same try/except-and-swallow pattern every other per-tenant best-effort read in this router uses, so one tenant's read error never breaks the whole registry list. ## Related * [Client onboarding pipeline](/features/client-onboarding-pipeline) * [Staff portal](/features/staff-portal) * [Workers and jobs](/architecture/workers-and-jobs) * [Security model](/architecture/security)