services/api/onboarding_api.py unless noted), which durable worker consumes the resulting job, and a psql query to confirm it landed. Bring the stack up first:
Tenant creation only happens from the New client wizard (
/admin/tenants/new). Once the tenant exists it appears in the Clients directory in any lifecycle state, and clicking into it opens its own detail page (/admin/tenants/{tenantId}) with the remaining steps as independent actions — that per-step page, not the wizard, is what this walkthrough narrates from Provision onward, because each action there maps 1:1 to one backend call you can verify in isolation. The wizard offers a faster, linear path through most of the same calls — plus two the detail-page path never touches, PUT .../data-policy and POST .../sources/seed — see the callout at the end of Activate for exactly how it differs.psql checks below run through the Compose Postgres container:
Tenant lifecycle
control.tenants.lifecycle_state is the state this whole walkthrough drives forward. The transitions below are reachable through this pipeline (services/api/tenancy/provisioner.py, services/api/onboarding_api.py):
provision (POST .../provision) also accepts a tenant already in a provisioning state as a no-op guard, but no code path in this repository ever writes that value — every tenant observed here goes draft straight to onboarding once its provisioning job’s last step (activate_onboarding) completes.1
Staff sign-in
Open
http://127.0.0.1:13000/login, select the Causeloop Employee tab, and sign in with the local-only credential admin@causeloop.local / Local-Admin-Pass1! (seeded by the local-admin Compose service — see Local product stack). You land on /admin, the Clients directory.- Endpoint:
POST /admin/login(services/api/auth_api.py), backed bylogin_employeeinservices/api/auth/employee_auth.py. - Verify:
2
Create the tenant
In the rail, click New client, fill in Organization name (the slug auto-derives via Now leave the wizard — click Clients in the rail, then click the new tenant’s row to open its detail page. The rest of this walkthrough happens there.
slugify() unless you edit it directly), industry, region, and a primary admin email, then Continue →. This is the only step that runs inside the wizard — it creates the draft control-plane row and nothing else; no tenant schema exists yet.- Endpoint:
POST /admin/tenants→create_tenant, which validates the slug (validate_tenant_slug, lowercase alphanumeric + hyphens, 3–63 chars) and callscreate_draft_tenant(services/api/tenancy/provisioner.py). - Verify:
This walkthrough uses
acme as the example slug throughout. tenant_schema_name() (services/api/tenancy/db.py) builds the schema name as f"tenant_{slug}", preserving any hyphens in the slug verbatim — so a slug like your-slug gets the schema tenant_your-slug, which requires double-quoting (SET search_path TO "tenant_your-slug") because Postgres identifiers can’t contain a bare hyphen otherwise. A hyphen-free slug avoids that quoting entirely.3
Provision
On the tenant detail page, Step 1. Provision shows a Provision tenant button. Click it and watch The detail page polls
provisioner-worker’s log:- Endpoint:
POST /admin/tenants/{tenant_id}/provision→ensure_provision_job, which synchronously inserts acontrol.platform_jobsrow (job_type='provision_tenant',status='queued') before any background work starts —provisioner-workerflips it torunningwhen it claims the job — so the immediate next poll never reads a stalenot_started. - Worker job:
provision_tenant, consumed only byprovisioner-worker(the only worker holding owner-levelMIGRATIONS_DATABASE_URL).provision_tenant()runs six idempotent, individually-tracked steps:create_schema,run_tenant_migrations,grant_app_rw,seed_role_profiles,verify_object_storage,activate_onboarding. The last step flipscontrol.tenants.lifecycle_statefromdraft/provisioningtoonboarding. - Verify:
GET /admin/tenants/{tenant_id}/provision/status every 2s while running and shows the tenant’s Postgres schema name once done.4
Upload historical data
Step 2. Historical data ingest now shows a file picker. Choose a CSV, XLSX, JSON, or JSONL file (
packages/causegraph/src/causegraph/onboarding/ingestion.py dispatches on the extension) — a representative issue/finding register works well. The API responds with a row count, a suggested column mapping (issue_id, text, created_at), and any columns it heuristically flags as PII (matched against hints like email, ssn, account_number in onboarding_api.py’s _PII_COLUMN_HINTS). Review the mapping, then click Commit dataset v1.- Endpoints:
POST .../ingest/upload(stores the raw file in MinIO under the tenant’s storage prefix, versioned, and parses/previews it — nothing is committed to Postgres yet) thenPOST .../ingest/commit(normalizes rows per the mapping and writesdatasets+recordsrows in the tenant schema). - Worker job: committing enqueues
materialize_insights(enqueue_platform_job, idempotency keymaterialize:{tenant_id}:{dataset_id}:{version}), consumed byproduct-worker. This is what turns raw records into the insight collections the client console reads later. - Verify (tenant-schema tables are unqualified, so set the search path first):
5
Training configuration
Step 3. Training configuration is fixed to the deterministic, offline embedding provider (
hash, dimension 64) and lets you pick the clustering algorithm (hdbscan, kmeans, or mixed_graph_greedy_modularity). LLM narrative enrichment is disabled in this runtime. Click Save configuration.- Endpoint:
PUT .../training-config— upsertstenant_schema.settings.key_values->'training_config'and a matchingllm_settingsrow (any API key you set here is Fernet-encrypted viaencrypt_secret/SOURCE_CREDS_KEY; only its hint is ever returned). - Verify:
6
Train
Step 4. Train shows a Train model button once a dataset exists.The detail page polls
- Endpoint:
POST .../train→ looks up the tenant’s latest dataset and saved training config, hashes the config, and enqueuestrain_tenant_modelwith idempotency keytrain:{tenant_id}:{dataset_id}:{config_hash}(retry_terminal=True, so a fixed and rerun job is allowed to run again even from a terminalfailedstate). - Worker job:
train_tenant_model, consumed byproduct-worker. - Verify:
GET .../train/status every 2s while running and invalidates the checkpoints list the moment it reports succeeded.7
Checkpoint & activate
Once training succeeds, Step 5. Activation & output sharing lists every
model_checkpoints row (version, algorithm, cluster count). Click Activate on the version you want live, and set Share onboarding outputs with the client — this decides whether the client sees this training run’s results on first login or starts from an empty dashboard.- Endpoint:
POST .../activate(ActivateCheckpointRequest{checkpoint_version, share_onboarding_outputs}) — deactivates every other checkpoint, activates the chosen one, re-enqueuesmaterialize_insightsagainst the current dataset withclient_visibleset from your choice, and setscontrol.tenants.lifecycle_state = 'review'. - Verify:
The New client wizard’s final step bundles this Activate call together with Invite and Go live into one Go live → click, using the admin email you typed on the very first wizard screen and a hardcoded
org_admin role. It does not wait for that invite to be accepted before going live. Everything downstream in this walkthrough still applies either way — only the order in which you personally click things changes.8
Invite a user
Step 6. Invite users takes an email and a role profile (
org_admin, risk_admin, analyst, or viewer — the default role profiles provision_tenant seeded per tenant). Submit Invite.- Endpoint:
POST .../invite→create_invitation(services/api/auth/tenant_auth.py) writes acontrol.tenant_invitationsrow (token hashed, 7-day expiry) and enqueues asend_emailjob with the accept-invite link. - Worker job:
send_email, consumed byproduct-worker, delivered over SMTP to Mailpit. - Verify:
9
Accept the invite in Mailpit
Open
http://127.0.0.1:18025, find the message (“You’re invited to acme on CauseLoop”), and open its accept-invite link (/accept-invite?workspace=acme&token=...). Set a full name (optional) and a password (8+ characters), then submit.- Endpoint:
POST /accept-invite(services/api/auth_api.py→accept_invitation) — creates (or reactivates) theTenantUserrow, assigns the invited role profile, marks the invitationaccepted, then immediately attempts to log the new user in. - Verify:
10
Go live
Back on the tenant detail page, Step 7. Go live’s button is clickable as soon as the tenant is provisioned — the UI never gates it client-side. Its three real preconditions are enforced only server-side, checked live by
POST .../go-live:- at least one dataset ingested,
- an active model checkpoint,
- at least one invitation in
pendingoracceptedstatus (acceptance is not required — a pending invite is enough).
409 naming exactly what’s missing ("Cannot go live yet: <missing items>.").- Endpoint:
POST .../go-live→ setscontrol.tenants.lifecycle_state = 'live'. - Verify:
11
Open the portal, or sign in as the client
Back in the Clients directory, the tenant now shows a Live chip and an Open portal → button.
- Open portal (impersonate):
POST .../impersonatemints a realcl_tenant_sessioncookie for the tenant’s earliest-created active user and redirects to/— this is a genuine access grant, not a read-only preview, so it is unconditionally audit-logged (control.audit_log, actionemployee_impersonation) regardless of whether anyone reviews that log. - Or sign in directly: open
http://127.0.0.1:13000/login, keep the default Client sign-in tab, and enter the workspace (tenant slug) plus the email/password set in the Accept-invite step. - Verify:
Other common stalls
The console never leaves its loading/locked skeleton
The console never leaves its loading/locked skeleton
Check
GET /health/ready directly (curl http://127.0.0.1:18000/health/ready or through the proxy at http://127.0.0.1:13000/api/backend/health/ready). It names exactly which dependency — database, worker, object_storage, credential_encryption, email_delivery, or redis — is failing. worker most commonly means one of the two durable workers hasn’t written a control.runtime_heartbeats row in the last 45 seconds; check docker compose ... logs product-worker provisioner-worker for a crash loop.Provisioning appears stuck at 'not started'
Provisioning appears stuck at 'not started'
A poll landing before the provisioning job’s first database write reads
not_started, not running — ensure_provision_job closes this race at the source by inserting the job row synchronously before returning 202. The New client wizard (tenants/new/page.tsx) keeps polling on not_started as a defensive backstop, but the tenant detail page (tenants/[tenantId]/page.tsx) only refetches while status is running — a poll that lands on not_started there stops polling outright, so a manual refresh may be needed. If it’s still not_started after several seconds, check provisioner-worker’s logs directly; it may not have started (see Local product stack’s startup-order diagram — it depends on migrate and minio-init completing).Upload returns 503
Upload returns 503
ingest/upload raises 503 Object storage is unavailable (ObjectStoreError) if MinIO isn’t reachable or minio-init hasn’t finished creating/versioning the causeloop-uploads bucket yet. Confirm with docker compose ... ps minio minio-init — minio-init should show Exit 0, not still running or failing.Ingest commit returns 422
Ingest commit returns 422
IngestError (unsupported file extension, empty workbook/CSV, invalid JSON, or a mapping that points at a non-existent column) surfaces as 422 with the specific reason in detail. The failed upload is marked status='failed' (mark_upload_failed) rather than left ambiguous — re-upload a corrected file rather than retrying the same upload_id.Invitation shows 'email missing' in the tenant detail page's list
Invitation shows 'email missing' in the tenant detail page's list
list_invitations looks up each invitation’s delivery job by idempotency key (email:tenant-invitation:{invitation_id}); "missing" means no matching send_email job row was ever found in control.platform_jobs. This can’t be caused by a stopped product-worker or an unhealthy Mailpit — create_invitation enqueues the send_email job in the same transaction as the invitation row, so a healthy invite always has at least a queued or failed delivery job. "missing" only happens if the job row itself was removed after the fact (or the invitation predates the delivery-job wiring).