Runbook: Incident Response

Who to Contact

See OWNERS.md for full contact list.

Role	Responsibility
Platform Lead	Overall incident ownership, escalation to Eusden
On-call Engineer	First responder, initial triage
Eusden	Production deploy authority, major architectural decisions

Production deploys are restricted to Eusden. For incidents requiring a production rollback or hotfix deploy, Eusden must be looped in.

---

Severity Levels

Severity	Definition	Response Time
P0	All clients unable to send/receive messages; auth down	Immediate (wake up)
P1	Major feature broken for all orgs (queue down, billing broken)	30 minutes
P2	Feature broken for some orgs; degraded performance	2 hours
P3	Minor issue; workaround available	Next business day

---

First Response Checklist

1. Check Railway dashboard — are all services running?

   railway logs --service humanwork-api | tail -50
   railway logs --service humanwork-agent | tail -50
2. Check health endpoints:

   curl https://api.humanwork.ai/health
   curl https://api.humanwork.ai/ready
3. Check Vercel dashboard for frontend deploy status
4. Check PostgreSQL connectivity: GET /ready will fail if DB is unreachable
5. Post status update in team Slack channel within 15 minutes of incident start

---

Railway Service Down

Symptoms: GET /health returns non-200; Railway shows service as crashed or restarting.

Steps:

1. Check logs: railway logs --service humanwork-api --lines 100
2. Look for: OOM (Out of Memory), migration failure, missing env var, crash on startup
3. If OOM: scale up memory in Railway dashboard (Settings → Resources)
4. If migration failure: check DATABASE_URL and DATABASE_SSL=true are set; run migration manually
5. If env var missing: check Railway Variables for the service; compare against .env.example
6. Restart service: Railway dashboard → Service → Deployments → Restart
7. If restart fails: rollback to previous deploy

   railway rollback --service humanwork-api --deployment <previous-deploy-id>
   Railway keeps the last 5 deployments.

---

Vercel Deploy Failed / Frontend Down

Symptoms: https://app.humanwork.ai returns 500 or shows stale build.

Steps:

1. Check Vercel dashboard → Deployments → find failing deploy → view build logs
2. Common causes: TypeScript build error, missing env var, Next.js build timeout
3. To rollback: Vercel dashboard → Deployments → select previous successful deploy → "Promote to Production"
4. To redeploy from CLI:

   cd frontend && vercel --prod

---

Database Connection Errors

Symptoms: GET /ready fails; API logs show ECONNREFUSED or connection timeout.

Steps:

1. Check Railway Postgres service status in dashboard
2. Verify DATABASE_URL env var is set on the API service
3. Verify DATABASE_SSL=true is set (Railway Postgres requires SSL)
4. Check if connection pool is exhausted (Railway Postgres has a connection limit per plan)
   • If exhausted: restart the API service to release connections; consider adding connection pooling (PgBouncer)
5. If Postgres service is crashed: Railway dashboard → humanwork-postgres → Restart
6. Verify extensions are installed after restore:

   railway run --service humanwork-postgres \
     psql "$DATABASE_URL" -c "\dx"
   Expected: vector, pg_trgm extensions present.

---

Agent Errors Spiking

Symptoms: Expert queue filling up faster than usual; POST /chat or POST /v1/chat returning errors; API logs show agent timeouts.

Steps:

1. Check agent logs: railway logs --service humanwork-agent | tail -100
2. Look for: OpenAI API rate limits (429), model errors (500), OOM
3. If OpenAI rate limit: check usage at platform.openai.com; reduce throughput or upgrade tier
4. If agent is OOM: scale memory in Railway dashboard
5. To disable auto-send temporarily (route everything to Expert queue):
   • In Railway Variables for humanwork-api, set CONFIDENCE_THRESHOLD_OVERRIDE=101
   • All responses will route to Expert queue regardless of agent confidence
   • Remove the override when agent is healthy
6. If agent service is repeatedly crashing, restart it:

   railway up --service humanwork-agent

---

WhatsApp (Twilio) Not Receiving Messages

Symptoms: WhatsApp messages not arriving in API; clients reporting messages to Specialist not getting responses.

Steps:

1. Check API logs for Twilio webhook activity: railway logs --service humanwork-api | grep whatsapp | tail -50
2. Check Twilio Console → Monitor → Errors for webhook delivery failures
3. Verify webhook URL in Twilio Console matches https://api.h.work/channels/whatsapp/inbound
4. Verify TWILIO_SUBACCOUNT_TOKEN in Railway env vars matches the subaccount shown in Twilio Console
5. If webhook is being rejected with 403: TWILIO_SUBACCOUNT_TOKEN is wrong — update in Railway and redeploy
6. Twilio retries failed webhooks for up to 4 hours — messages are not lost unless the outage exceeds that window

---

Expert Queue Not Delivering Notifications

Symptoms: New queue items not appearing in real-time; experts must refresh to see new items.

Steps:

1. Check if WebSocket connection is established: browser DevTools → Network → WS tab
2. Check API logs for Socket.io errors
3. Check NEXT_PUBLIC_WS_URL env var is set correctly in Vercel (should be wss://api.humanwork.ai)
4. Verify CORS: ALLOWED_ORIGINS env var on API must include https://app.humanwork.ai
5. If Socket.io failing, queue still functions — experts can poll manually. Not a P0.

---

Rollback Procedure

# Railway: list recent deployments
railway deployments --service humanwork-api

# Rollback to a specific deployment
railway rollback --service humanwork-api --deployment <deploy-id>

# Vercel: promote a previous deployment
# Go to: vercel.com → Project → Deployments → click previous deploy → "Promote to Production"

---

Post-Incident

1. Write a brief post-mortem (timeline, root cause, fix, prevention)
2. Create GitHub issues for any systemic gaps found
3. Update this runbook if the incident revealed a missing procedure