Your on-call is the
last line of defense.
A bad deploy fires PagerDuty. Someone wakes up. They read through Grafana, confirm the problem is real, find the right kubectl command, and watch dashboards for another 20 minutes to verify recovery. By then, thousands of users have already hit the bug.
The situation
"3:14 AM. PagerDuty fires. The senior engineer on call squints at Grafana, confirms it's real, and starts looking for the rollback runbook."
By the time they've identified the bad deploy, found the right command, executed it, and confirmed recovery, 22 minutes have passed. Error rates were elevated for every one of those minutes. Every user who hit checkout during that window got an error. The post-mortem will call this a "fast response." It wasn't fast enough.
The real problem
The rollback tools already exist. kubectl rollout undo works. The problem is the pipeline from "error rate spikes" to "rollback complete" runs entirely through a human being, who has to be awake, alert, and in front of a laptop to execute it. That pipeline has a minimum latency of 10–15 minutes on a good night. On a bad night, it's 45.
There's a second, quieter problem: most rollbacks are too broad. Rolling back the entire service when only 3% of users (in one region, on one feature flag) are affected is a blunt instrument. You're degrading the 97% to protect the 3%. And you still have to redeploy when you're ready to try again.
The fix
The rollback decision needs to happen before the human wakes up. That means a system continuously watching your SLO metrics (not alerts, which are already late) and triggering a scoped reversion the moment a measurement window closes over your threshold. No runbook. No approval. No redeploy. Just traffic back where it was, in under 10 seconds, with an automatic incident packet generated for the post-mortem.
Configuration
Define your rollback policy once.
A single stanza in your titan.yaml is all Phoenix needs. No runbooks, no on-call scripts.
# titan.yaml — phoenix rollback policy
rollback:
triggers:
- slo: api-error-rate
threshold: 0.5%
window: 2m
scope: cohort # only the affected cohort, not the whole service
mode: auto # no human required
dry_run: false # set true to simulate without acting
post_rollback:
notify: [slack, pagerduty]
packet: true # generate incident packet for Titan InsightThe transformation
Before and after Phoenix rollback.
How it works
Four steps. Under 10 seconds.
SLO breach detected, automatically
< 30s
median time to rollback decision
Phoenix continuously watches the SLO metrics you define. The moment a measurement window closes over your threshold, Phoenix triggers: no alert, no Slack message, no human in the loop.
Blast radius scoped to the failing slice
Scoped
only the failing slice reverted
Phoenix doesn't roll back your whole service. It identifies exactly which cohort, region, or feature-flag state is failing, and reverts only that. Every other user keeps running the new version.
Traffic restored in under 10 seconds
< 10s
median traffic restoration time
Once the scope is determined, Phoenix reverses the traffic shift. No redeploy, no pipeline run, no manual approval. The release is preserved: only its routing is changed.
Incident packet generated automatically
Zero
manual approvals required in auto mode
Every Phoenix action produces a structured incident packet: what SLO breached, which cohort was affected, how long the impact lasted, and what was reverted. Feeds directly into Titan Insight.
The status quo
How teams handle rollbacks today, and the cost.
The manual rollback playbook has three failure modes: too slow, too broad, and too dependent on a human being awake.
How we compare
DeployTitan vs. the alternatives.
| Capability | DeployTitan | Manual runbook | Feature flags only |
|---|---|---|---|
| Automatic rollback (no human needed) | ✓ | ✗ | ✗ |
| Scoped to failing cohort only | ✓ | ✗ (full service) | ✗ |
| Traffic restored in < 10s | ✓ | ✗ (6–25 min) | ~ (flag toggle only) |
| SLO-aware trigger | ✓ | ✗ (alert-based) | ✗ |
| Structured incident packet | ✓ | ✗ (manual postmortem) | ✗ |
See Phoenix roll back a live canary in under 10 seconds.
We'll trigger a real SLO breach in a sandbox environment and show you exactly what Phoenix does, automatically, before you'd even finish reading the alert.