> ## Documentation Index
> Fetch the complete documentation index at: https://docs.causeloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Client onboarding runbook
> The operations playbook for onboarding a real paying client against a live production deployment: roles, prerequisites, data-file expectations, production email delivery, and the common failure playbook.
[Client onboarding pipeline](/features/client-onboarding-pipeline) documents every API call, validation, and worker job in the onboarding sequence at the code level. This page is its operational companion: the same eight steps, but written for the person actually running them against a **production** deployment — who is allowed to do what, what a real client's data file needs to look like before you upload it, what "training" actually costs in wall-clock time, how invitation email really gets delivered once Mailpit is no longer in the loop, and what to do when a step fails. If you are onboarding a client into a local Compose stack for a demo or development, see [Local onboarding walkthrough](/onboarding/local-onboarding-walkthrough) instead — the API calls are identical, but none of the production caveats below apply there.
## Roles
Onboarding a client is entirely a **staff-side** operation. There is no client self-service signup anywhere in the product.
| Role | What they can do |
| -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Employee, `onboarding_admin` control role | Every mutating step below: create, provision, ingest, configure training, train, activate, invite, go live. `services/api/onboarding_api.py` aliases this as `_MUTATE` (`Depends(require_employee_role("onboarding_admin"))`) on every write route. |
| Employee, `support_readonly` control role | Can load every `GET` in the onboarding router (tenant detail, provision/train status, checkpoints, invitations) but every mutation `403`s. Useful for a support engineer verifying progress without being able to change anything. |
| The client's own tenant users (`org_admin`, `risk_admin`, `analyst`, `viewer` role profiles) | Cannot sign in at all until `go-live` flips `lifecycle_state` to `live` — `login_tenant_user()` checks that gate directly. Before then, the only way to see the tenant's console as it will appear to them is staff **impersonation** (`POST /admin/tenants/{tenant_id}/impersonate`), which mints a real tenant session and writes an unconditional `control.audit_log` row every time — see [Security model](/architecture/security). |
No route in the deployed product surface lets a client provision themselves, upload their own historical data before go-live, or invite their own colleagues before an `onboarding_admin` has completed the pipeline. Every one of the steps below is run by your own staff, not the client.
The role you invite the client's first user into (step 6 below, `role_profile_key`) determines what they can do from day one — `services/api/tenancy/provisioner.py`'s `DEFAULT_ROLE_PROFILES`, seeded during provisioning:
| `role_profile_key` | Permissions |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `org_admin` | Everything: `insights.read`, `sources.manage`, `ingest.run`, `training.configure`, `reviews.write`, `caps.write`, `exports.create`, `users.manage`, `settings.llm`, `settings.share_outputs`, `tokens.manage`, `pii.unmasked` |
| `risk_admin` | `insights.read`, `sources.manage`, `ingest.run`, `training.configure`, `reviews.write`, `caps.write`, `exports.create` — everything but user management and unmasked PII |
| `analyst` | `insights.read`, `reviews.write`, `caps.write`, `exports.create` |
| `viewer` | `insights.read` only |
The default on the invite route is `org_admin` — confirm with the client which of their people should hold that role before sending the first invitation, since `users.manage` and `pii.unmasked` are broad grants.
## Before onboarding the first real client
Do this once per environment, not once per client:
1. The environment itself must already satisfy `GET /health/ready` — see the release order in [AWS deployment](/deployment/aws) or [Azure deployment](/deployment/azure). A client onboarded against a not-yet-ready environment will hit `503`s at the ingest step (object storage) or silently queued-forever jobs (missing worker heartbeat).
2. Production email delivery must be configured and proven with **one real invitation delivered to a controlled inbox** before the first client invitation goes out — see [Production email delivery](#production-email-delivery-vs-mailpit) below. `docs/CLOUD_OPERATIONS.md` is explicit that readiness does not send a message; a `ready` deployment can still be misconfigured for the recipient's actual mailbox provider (SES sandbox, an unverified sender domain) in a way `/health/ready` cannot see.
3. Credential encryption must be a real KMS/Key Vault provider, not local Fernet — the per-tenant LLM API key set in step 4 below is encrypted at rest with whatever `secret_encryption_readiness()` reports (`services/api/secrets.py`).
## The pipeline, with production verification checkpoints
Each step matches [Client onboarding pipeline](/features/client-onboarding-pipeline) exactly; only the verification checkpoint and production-specific caveats are new here.
`POST /admin/tenants` with a slug matching `^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$`. **Verify:** `GET /admin/tenants/{tenant_id}` returns `lifecycle_state: "draft"` and a `control.audit_log` row with `action="tenant_created"` exists (visible in the tenant detail response's `audit_trail`).
`POST /admin/tenants/{tenant_id}/provision`, `202`. This runs on the **provisioner-worker**, a separately privileged process from the product worker. **Verify:** poll `GET /admin/tenants/{tenant_id}/provision/status` until `status: "succeeded"` and `steps_completed` lists all six steps (`create_schema`, `run_tenant_migrations`, `grant_app_rw`, `seed_role_profiles`, `verify_object_storage`, `activate_onboarding`). A provision stuck on `verify_object_storage` almost always means the object-storage credentials the provisioner identity was granted don't actually resolve the bucket/container — check [Monitoring and incidents](/deployment/monitoring-and-incidents) for the `object_storage` readiness check.
See [Data-file expectations](#data-file-expectations-for-client-uploads) below before requesting the file from the client. **Verify:** the upload response's `row_count` and `detected_pii_columns` match what you expect from the file's actual shape; after commit, `GET /admin/tenants/{tenant_id}` should reflect a dataset exists (checked indirectly via the go-live prerequisite check later, or directly against the tenant schema if you have operator access).
`PUT /admin/tenants/{tenant_id}/training-config`. If the client wants a non-`disabled` LLM provider for narrative generation, the API key is submitted once (`llm_api_key`), encrypted server-side, and every subsequent response returns only `llm_api_key_hint` — never the plaintext again. **Verify:** the response's `llm_api_key_hint` is non-null if you submitted a key.
`POST /admin/tenants/{tenant_id}/train`, `202`. See [Training duration expectations](#training-duration-expectations) below. **Verify:** poll `GET /admin/tenants/{tenant_id}/train/status` until `succeeded`, then `GET /admin/tenants/{tenant_id}/checkpoints` shows a new version with populated `metrics`.
`POST /admin/tenants/{tenant_id}/activate` with the checkpoint version and whether onboarding-era insight collections should be client-visible (`share_onboarding_outputs`). **Verify:** `lifecycle_state` becomes `review`.
`POST /admin/tenants/{tenant_id}/invite`. See [Production email delivery](#production-email-delivery-vs-mailpit) below. **Verify:** `GET /admin/tenants/{tenant_id}/invitations` shows the invitation with `delivery_status`. Do not proceed to go-live until `delivery_status` is `succeeded`, not just `queued`.
`POST /admin/tenants/{tenant_id}/go-live`. Fails with a single `409` naming every unmet prerequisite (dataset ingested, checkpoint active, invitation pending/accepted) rather than the first one found. **Verify:** `lifecycle_state` is `live`; the invited user can now actually sign in at `/login`, resolving their workspace by slug.
## Data-file expectations for client uploads
The only ingestion path reachable through the deployed product surface is **file upload** — either `POST /admin/tenants/{tenant_id}/ingest/upload` (staff-portal onboarding, used above) or the tenant-facing `POST /sources/{source_id}/upload` for a `file_upload`-kind source. `services/ingest/connectors/registry.py` also registers `s3`, `database`, and `kafka` connector kinds with real pull logic, but the routes that test, sync, or poll those connector kinds are not in `product_app.py`'s allowlist — they exist for the tenant to register (`POST /sources` validates the config schema) but cannot currently be exercised end-to-end through the deployed API. Plan a client's first ingestion around a file, not a live feed.
`causegraph.onboarding.ingestion.parse_upload()` accepts exactly these file types, detected from the filename extension:
| Extension | Parser | Notes |
| --------- | ---------------------------------------------------- | ----------------------------------------------------------------------- |
| `.xlsx` | `openpyxl`, first worksheet only, `read_only=True` | Header row becomes column names; blank trailing rows are skipped. |
| `.csv` | Standard library `csv`, decoded as `utf-8-sig` | A leading byte-order mark is tolerated. |
| `.json` | A top-level list of objects, or `{"records": [...]}` | Every item must be an object; a bare object or scalar list is rejected. |
| `.jsonl` | One JSON object per non-blank line | Same object-shape requirement as `.json`. |
Regardless of format, the parsed table must resolve to **at least three columns** (an issue id, a text/description field, and a created-date field) or the upload is rejected with `422` before anything is written. `detect_mapping()` guesses which column plays which role using substring hints (`id`/`key`/`ref` for the identifier; `text`/`description`/`summary`/`narrative`/`details` for the text field; `creat`/`date`/`time`/`opened`/`reported` for the date) and falls back to positional columns when no hint matches — always show the client the `suggested_mapping` from the upload response and let them correct it before committing, rather than trusting the heuristic blindly on an unfamiliar schema.
The upload endpoint also runs `_detect_pii_columns()` against the *column names themselves* (`email`, `phone`, `ssn`/`social_security`, `name`, `address`, `dob`/`date_of_birth`, `account_number`, `card_number`), suggesting a `redact` or `partial` masking strategy per match — these become `pii_field_policies` rows on commit. This is a name-based heuristic, not content inspection; a column literally named `notes` that happens to contain free-text PII is not caught, and should be reviewed with the client before commit.
`max_upload_bytes()` defaults to 25 MB (`CAUSELOOP_MAX_UPLOAD_BYTES`, `services/ingest/uploads.py`). A larger export needs to be pre-split by the client or the environment variable raised — a `413` at upload time means the file, not the request, is the problem.
## Training duration expectations
There is no published SLA for how long a `train_tenant_model` job takes to complete — it runs synchronously inside a single `product-worker` job claim, and its wall-clock time scales with row count and the configured `embedding_provider`. The `hash` embedding provider (the training-config default) computes a deterministic hash-based embedding locally with no external call, so it is the fastest option and the one to default to for a client's first onboarding pass unless they have specifically asked for a semantic embedding model. Any API-backed embedding or LLM provider (`embedding_base_url`, `llm_provider` other than `disabled`) adds per-row network latency on top of that baseline, and a monthly token budget (`llm_monthly_token_budget`) that can throttle a large run. Set client expectations qualitatively — "minutes, scaling with row count and embedding provider choice" — rather than promising a fixed number; if a training job needs to be re-run with a different config, the `train` route enqueues a genuinely new job (its idempotency key hashes the current config), so nothing is lost by adjusting `training-config` and re-training.
## Production email delivery vs. Mailpit
Locally, `CAUSELOOP_EMAIL_PROVIDER` defaults to `console`/`local-smtp` and invitations land in Mailpit's captured-mail UI. `services/api/auth/mailbox.py`'s `email_delivery_readiness()` **forbids both local providers outright in production** (`CAUSELOOP_ENVIRONMENT=production`). The supported production providers are:
| `CAUSELOOP_EMAIL_PROVIDER` | Required configuration | Cloud target |
| -------------------------- | ------------------------------------------------------------ | ---------------------------------- |
| `aws-ses` | `CAUSELOOP_EMAIL_FROM`; `AWS_REGION` or `AWS_DEFAULT_REGION` | Amazon SES |
| `azure-communication` | `CAUSELOOP_EMAIL_FROM`; `AZURE_COMMUNICATION_EMAIL_ENDPOINT` | Azure Communication Services Email |
`CAUSELOOP_PUBLIC_APP_URL` must also resolve to an absolute HTTPS origin in production — invitation links are built from it (`public_app_link()`), and readiness fails closed if it's missing or not HTTPS.
Two things to know operationally, both already noted in `docs/CLOUD_OPERATIONS.md`:
* **Readiness passing does not mean a message was actually delivered.** `/health/ready`'s `email_delivery` check only proves the provider contract is structurally complete (sender address, region/endpoint present) — it does not send a probe email. Before onboarding the first real client, send one real invitation to a controlled inbox you own and confirm it arrives; on AWS this also means confirming the SES identity is verified and the account is out of the SES sandbox, or arbitrary client recipients stay blocked even though every container looks healthy.
* **Delivery is at-least-once, not exactly-once.** The invitation row and its encrypted `send_email` job commit in one transaction (7-day token expiry, `INVITATION_LIFETIME` in `services/api/auth/tenant_auth.py`), so the invitation itself can never be silently lost — but Azure reuses the job UUID as its provider operation id (naturally idempotent), while Amazon SES has no such token, so a worker crash after SES accepts a message but before the database records success can duplicate the email. It cannot duplicate or lose the underlying invitation state.
`GET /admin/tenants/{tenant_id}/invitations` reports `delivery_status` and `delivery_attempts` per invitation by joining to the underlying `send_email` platform job — use it as your source of truth for "did the client actually get the email," not just the `201` from the invite call.
## Common failure playbook
| Symptom | Likely cause | Fix |
| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST /admin/tenants` returns `409` | Slug already exists in `control.tenants` | Pick a different slug, or confirm you're not re-onboarding an existing client under a new slug by mistake. |
| `POST .../provision` returns `409` | Tenant `lifecycle_state` is not `draft`/`provisioning`/`onboarding` (e.g. already `live`) | You cannot re-provision a live tenant through this route; if the client's schema genuinely needs rework, that is an operator-level action outside this pipeline. |
| Provision status stuck on `running`, `steps_completed` short of all six | Provisioner worker not heartbeating, or `verify_object_storage` failing | Check `control.runtime_heartbeats` for `provisioner-worker` (see [Monitoring and incidents](/deployment/monitoring-and-incidents)); confirm the provisioner identity's object-storage credentials actually resolve the configured bucket. |
| Upload returns `413` | File exceeds `CAUSELOOP_MAX_UPLOAD_BYTES` (25 MB default) | Have the client split the export, or raise the limit for this environment. |
| Upload or commit returns `422` | Fewer than three columns, unmapped required role, or malformed JSON/CSV | Inspect the `suggested_mapping` and `columns` in the upload response with the client before re-uploading; this never partially commits a dataset. |
| Upload returns `503` | Object storage unreachable | This is the same dependency `/health/ready`'s `object_storage` check reports — the environment itself is not ready; do not proceed with onboarding until it clears. |
| `train` returns `409` "No ingested dataset" | Ingest step was never committed, or was committed then a fresh tenant swap happened | Confirm `ingest/commit` actually returned a `dataset_id` before calling `train`. |
| Invitation `delivery_status` stuck on `queued` | Product-worker not advertising `send_email` (fails startup validation if email/encryption readiness isn't clean) or worker not heartbeating at all | Check `/health/ready`'s `worker` check and `email_delivery` check together — `validate_worker_configuration()` refuses to let a worker claim `send_email` jobs if the provider or encryption contract is broken, so a real config problem shows up here as jobs never leaving `queued`, not as an explicit error. |
| `go-live` returns `409` listing missing prerequisites | One of dataset/checkpoint/invitation genuinely isn't in place yet | The error names every missing item explicitly — complete exactly those steps, then retry; `go-live` is safely re-callable. |
## Ingesting more data after go-live
The pipeline above describes the client's first dataset, but ingest/commit is not a one-time gate — a live client can have new historical data uploaded and committed the same way at any point, producing a new `Dataset` row at `version = max(existing) + 1` in the same tenant schema. Re-committing the exact same file is safe: `content_hash`-based deduplication (`ON CONFLICT (tenant_id, dataset_id, content_hash) DO NOTHING`) means a byte-identical row landing in the same dataset version twice is a no-op, not a duplicate. If the new data should actually retrain the model rather than just sit alongside the existing checkpoint, re-run the `train` → `activate` steps above — training re-run against a live tenant follows the exact same job and idempotency-key mechanics as the first pass, and `GET /admin/tenants/{tenant_id}/retrain-recommendation` gives a full/incremental/current recommendation based on how much the dataset has grown since the active checkpoint was trained (see [Training and checkpoints](/features/training-and-checkpoints)).
## QA the client's view before go-live
Because tenant users can't sign in until `go-live`, the only way to see the console exactly as the client will see it — with their real ingested data, their activated checkpoint, and their `share_onboarding_outputs` choice applied — is staff **impersonation**: `POST /admin/tenants/{tenant_id}/impersonate`. This mints a real tenant session for the tenant's earliest-created active user and is worth doing as a deliberate QA step between "activate" and "go-live," not just as a support tool after the client is already live. Every use writes an unconditional `control.audit_log` row (`action="employee_impersonation"`), so there is no way to do this QA pass without it being auditable — treat it as a real, logged access grant into the client's data, not a read-only preview mode.
## Related
* [Client onboarding pipeline](/features/client-onboarding-pipeline)
* [Monitoring and incidents](/deployment/monitoring-and-incidents)
* [Security model](/architecture/security)
* [AWS deployment](/deployment/aws)