Runbook: Railway Outage

Use this runbook when Railway itself is experiencing an outage (not a single service crash).

How to Confirm It's Railway

1. Check https://status.railway.app for active incidents
2. Verify multiple services are affected simultaneously (not just one crashed service)
3. Check team Slack — others will likely be reporting the same thing

During a Railway Outage

Immediate (< 5 minutes)

1. Post in team Slack: "Railway outage in progress — monitoring status.railway.app"
2. Check if Vercel frontend is still serving (it's independent of Railway)
   • If frontend is up: clients can still load the app but API calls will fail
   • Post a status note if you have a client-facing status page
3. Do not attempt to redeploy services during an outage — Railway deployments will also fail

If Outage Extends > 30 Minutes

1. Assess which functionality is affected:

   • API down → no new conversations, no queue processing, no auth
   • Postgres down → API will also be down (DB dependency)
   • Redis down → rate limiting falls back to in-process (ThrottlerModule); minimal impact
   • Agent down → new messages auto-escalate to Expert queue (fail-open behavior)
   • WhatsApp (Twilio) → Twilio delivers to API; if API is down, Twilio will retry for up to 4 hours

2. Communicate to affected clients if SLA breach is imminent

3. For extended Postgres outage (> 2 hours), assess data loss risk:

   • Railway Postgres has automatic backups — check Railway dashboard for last backup time
   • No action needed unless DB was corrupted before the outage

Recovery

When Railway recovers:

1. Check all service health: curl https://api.humanwork.ai/health and /ready
2. Check Railway logs for each service for startup errors
3. Verify migrations ran successfully (check API logs for migration output on startup)
4. Verify pgvector extension is still present:

   railway run --service humanwork-postgres \
     psql "$DATABASE_URL" -c "SELECT extname FROM pg_extension WHERE extname = 'vector';"
5. Test a full conversation flow manually
6. Check Expert queue for any items that were created during the outage but not delivered via WebSocket — they will appear on next queue load

Failover Options (Not Yet Implemented)

The current architecture has no hot standby. If Railway has a prolonged multi-hour outage affecting production, options are:

• Manual failover to Render/Fly.io: requires re-provisioning Postgres and running migrations; estimated 2-4 hours setup time
• AWS RDS + ECS: a pre-configured fallback stack does not exist; would take 4+ hours to provision from scratch

Formal DR planning is tracked as a future initiative. For now: Railway outages are the primary risk, and the mitigation is Railway's own redundancy and fast recovery (typical Railway incidents resolve in < 2 hours).