Ops runbooks

Short responses to the failures tenants actually hit. Each runbook lists the symptom, then checks in order, then the escalation pack. Surfaces from the tenant-lifecycle sources.

R1: Tenant stuck on /preparing

Symptom: fresh tenant shows /preparing past the expected window.

1. Confirm tenant hostname and provisioning start.
2. Poll GET /api/v1/admin/tenants/[id]/provisioning to see which step is unreached. Check evaluateTenantReadiness output.
3. If the workflow is still running: wait, tell the customer admin it is normal, give a check-back time.
4. If a step failed or wedged (for example vercel_alias, mint_workos_org, token-gated/overridable ai_gateway_key): do not have the customer re-register. Note step, error, timestamp; repair from the ops side.
5. Verify clear with a fresh identity: preparing, then /onboarding/*, then dashboard.

Escalation pack: tenant id, hostname, failing step, error text, provisioning-start time.

R2: Hostname does not resolve or serves wrong content

Symptom: <slug>.preshos.com does not load, errors, or shows the wrong tenant.

1. Confirm the exact hostname (intake typos in slugs are common).
2. Check the tenant’s hostname assignment; check vercel_alias binding (hostname to intended deployment), especially after redeploys.
3. Check wildcard/DNS baseline (*.preshos.com) and whether a custom hostname was entered at Create.
4. Verify with a fresh identity in a clean session (no cached auth from another tenant).

Escalation pack: hostname, expected vs. actual, last-known-good time, recent deploys, alias state.

R3: Onboarding rescue (owner blocked)

Symptom: owner cannot complete the wizard or reach first value.

1. Confirm they are on the tenant hostname with the invited identity (owner email derived server-side at Create, so verify the right person).
2. Unblock on required vs. optional: org-name is the hard requirement; domain, branding, members, connect deferred.
3. Wizard errors (app/(onboarding)/onboarding): capture exact step and message; stop duplicate-tenant attempts.
4. No environment visible: provision or point at one (seed_default_agents should have covered this, so check whether it ran) before continuing.
5. Walk to first value (Day-0) and verify the recorded outcome yourself.

Escalation pack: tenant id, hostname, identity, wizard step, error text, environment state.

R4: Connector provisioning intake

Symptom: tenant requests a system connection (or reports one missing).

1. Record intake: tenant hostname, environment, target system, business need, urgency, requester.
2. Check the environment’s live tool list and binding state (settings.object_mappings for org, package, connection). Sometimes a mapping exists but is not enabled in that environment.
3. Registered-adapter check: HubSpot and ConnectWise PSA exist in provider.ts; anything else (including anything “Salesforce”) is not an implementation until the registry says so. Communicate that code is not connected; never promise dates the team has not set.
4. Track to done in this order: provisioned, credentialed (holder, product flow), enabled in environment, bindings configured, verified in test session, requester notified.

Escalation pack: intake record, environment, target system, binding coordinates, definition of done.

R5: Cross-tenant confusion

Symptom: wrong data, missing powers, missing records.

1. Ask for the hostname first, always.
2. Verify identity and role in that tenant; partner-admin reach goes through /my-tenants. Confirm which tenant they are actually in.
3. Verify org scope on the records (org_id vs company_id).
4. Fix bookmarks; re-grant in the right tenant if a grant landed wrongly.

Escalation pack: both hostnames, identity, expected vs. actual, session/record references.

R6: Never suspend or deprovision org 1

Symptom (prevented): a suspend/deprovision action targeting org 1.

1. Abort the action.
2. Verify org-1 exclusion exists in the tooling path used. If it does not, file it as a bug before doing anything else.
3. Confirm org 1 healthy (control plane intact) before resuming normal ops.