Client onboarding runbook

Client onboarding pipeline documents every API call, validation, and worker job in the onboarding sequence at the code level. This page is its operational companion: the same eight steps, but written for the person actually running them against a production deployment — who is allowed to do what, what a real client’s data file needs to look like before you upload it, what “training” actually costs in wall-clock time, how invitation email really gets delivered once Mailpit is no longer in the loop, and what to do when a step fails. If you are onboarding a client into a local Compose stack for a demo or development, see Local onboarding walkthrough instead — the API calls are identical, but none of the production caveats below apply there.

​

Roles

Onboarding a client is entirely a staff-side operation. There is no client self-service signup anywhere in the product. No route in the deployed product surface lets a client provision themselves, upload their own historical data before go-live, or invite their own colleagues before an onboarding_admin has completed the pipeline. Every one of the steps below is run by your own staff, not the client. The role you invite the client’s first user into (step 6 below, role_profile_key) determines what they can do from day one — services/api/tenancy/provisioner.py’s DEFAULT_ROLE_PROFILES, seeded during provisioning: The default on the invite route is org_admin — confirm with the client which of their people should hold that role before sending the first invitation, since users.manage and pii.unmasked are broad grants.

​

Before onboarding the first real client

Do this once per environment, not once per client:

1. The environment itself must already satisfy GET /health/ready — see the release order in AWS deployment or Azure deployment. A client onboarded against a not-yet-ready environment will hit 503s at the ingest step (object storage) or silently queued-forever jobs (missing worker heartbeat).
2. Production email delivery must be configured and proven with one real invitation delivered to a controlled inbox before the first client invitation goes out — see Production email delivery below. docs/CLOUD_OPERATIONS.md is explicit that readiness does not send a message; a ready deployment can still be misconfigured for the recipient’s actual mailbox provider (SES sandbox, an unverified sender domain) in a way /health/ready cannot see.
3. Credential encryption must be a real KMS/Key Vault provider, not local Fernet — the per-tenant LLM API key set in step 4 below is encrypted at rest with whatever secret_encryption_readiness() reports (services/api/secrets.py).

​

The pipeline, with production verification checkpoints

Each step matches Client onboarding pipeline exactly; only the verification checkpoint and production-specific caveats are new here.

​

Data-file expectations for client uploads

The only ingestion path reachable through the deployed product surface is file upload — either POST /admin/tenants/{tenant_id}/ingest/upload (staff-portal onboarding, used above) or the tenant-facing POST /sources/{source_id}/upload for a file_upload-kind source. services/ingest/connectors/registry.py also registers s3, database, and kafka connector kinds with real pull logic, but the routes that test, sync, or poll those connector kinds are not in product_app.py’s allowlist — they exist for the tenant to register (POST /sources validates the config schema) but cannot currently be exercised end-to-end through the deployed API. Plan a client’s first ingestion around a file, not a live feed. causegraph.onboarding.ingestion.parse_upload() accepts exactly these file types, detected from the filename extension: Regardless of format, the parsed table must resolve to at least three columns (an issue id, a text/description field, and a created-date field) or the upload is rejected with 422 before anything is written. detect_mapping() guesses which column plays which role using substring hints (id/key/ref for the identifier; text/description/summary/narrative/details for the text field; creat/date/time/opened/reported for the date) and falls back to positional columns when no hint matches — always show the client the suggested_mapping from the upload response and let them correct it before committing, rather than trusting the heuristic blindly on an unfamiliar schema. The upload endpoint also runs _detect_pii_columns() against the column names themselves (email, phone, ssn/social_security, name, address, dob/date_of_birth, account_number, card_number), suggesting a redact or partial masking strategy per match — these become pii_field_policies rows on commit. This is a name-based heuristic, not content inspection; a column literally named notes that happens to contain free-text PII is not caught, and should be reviewed with the client before commit.

​

Training duration expectations

There is no published SLA for how long a train_tenant_model job takes to complete — it runs synchronously inside a single product-worker job claim, and its wall-clock time scales with row count and the configured embedding_provider. The hash embedding provider (the training-config default) computes a deterministic hash-based embedding locally with no external call, so it is the fastest option and the one to default to for a client’s first onboarding pass unless they have specifically asked for a semantic embedding model. Any API-backed embedding or LLM provider (embedding_base_url, llm_provider other than disabled) adds per-row network latency on top of that baseline, and a monthly token budget (llm_monthly_token_budget) that can throttle a large run. Set client expectations qualitatively — “minutes, scaling with row count and embedding provider choice” — rather than promising a fixed number; if a training job needs to be re-run with a different config, the train route enqueues a genuinely new job (its idempotency key hashes the current config), so nothing is lost by adjusting training-config and re-training.

​

Production email delivery vs. Mailpit

Locally, CAUSELOOP_EMAIL_PROVIDER defaults to console/local-smtp and invitations land in Mailpit’s captured-mail UI. services/api/auth/mailbox.py’s email_delivery_readiness() forbids both local providers outright in production (CAUSELOOP_ENVIRONMENT=production). The supported production providers are: CAUSELOOP_PUBLIC_APP_URL must also resolve to an absolute HTTPS origin in production — invitation links are built from it (public_app_link()), and readiness fails closed if it’s missing or not HTTPS. Two things to know operationally, both already noted in docs/CLOUD_OPERATIONS.md:

• Readiness passing does not mean a message was actually delivered. /health/ready’s email_delivery check only proves the provider contract is structurally complete (sender address, region/endpoint present) — it does not send a probe email. Before onboarding the first real client, send one real invitation to a controlled inbox you own and confirm it arrives; on AWS this also means confirming the SES identity is verified and the account is out of the SES sandbox, or arbitrary client recipients stay blocked even though every container looks healthy.
• Delivery is at-least-once, not exactly-once. The invitation row and its encrypted send_email job commit in one transaction (7-day token expiry, INVITATION_LIFETIME in services/api/auth/tenant_auth.py), so the invitation itself can never be silently lost — but Azure reuses the job UUID as its provider operation id (naturally idempotent), while Amazon SES has no such token, so a worker crash after SES accepts a message but before the database records success can duplicate the email. It cannot duplicate or lose the underlying invitation state.

GET /admin/tenants/{tenant_id}/invitations reports delivery_status and delivery_attempts per invitation by joining to the underlying send_email platform job — use it as your source of truth for “did the client actually get the email,” not just the 201 from the invite call.

​

Common failure playbook

​

Ingesting more data after go-live

The pipeline above describes the client’s first dataset, but ingest/commit is not a one-time gate — a live client can have new historical data uploaded and committed the same way at any point, producing a new Dataset row at version = max(existing) + 1 in the same tenant schema. Re-committing the exact same file is safe: content_hash-based deduplication (ON CONFLICT (tenant_id, dataset_id, content_hash) DO NOTHING) means a byte-identical row landing in the same dataset version twice is a no-op, not a duplicate. If the new data should actually retrain the model rather than just sit alongside the existing checkpoint, re-run the train → activate steps above — training re-run against a live tenant follows the exact same job and idempotency-key mechanics as the first pass, and GET /admin/tenants/{tenant_id}/retrain-recommendation gives a full/incremental/current recommendation based on how much the dataset has grown since the active checkpoint was trained (see Training and checkpoints).

​

QA the client’s view before go-live

Because tenant users can’t sign in until go-live, the only way to see the console exactly as the client will see it — with their real ingested data, their activated checkpoint, and their share_onboarding_outputs choice applied — is staff impersonation: POST /admin/tenants/{tenant_id}/impersonate. This mints a real tenant session for the tenant’s earliest-created active user and is worth doing as a deliberate QA step between “activate” and “go-live,” not just as a support tool after the client is already live. Every use writes an unconditional control.audit_log row (action="employee_impersonation"), so there is no way to do this QA pass without it being auditable — treat it as a real, logged access grant into the client’s data, not a read-only preview mode.

​

Related

• Client onboarding pipeline
• Monitoring and incidents
• Security model
• AWS deployment