A monitoring alert firing is not the end of your work — it's the beginning of a structured process. Teams that handle incidents well share a common characteristic: they have a documented playbook that removes decision paralysis under pressure. When it's 2 AM and production is down, the last thing your on-call engineer should be doing is deciding what to do next. The playbook tells them.
This guide covers the full incident response lifecycle for uptime-related incidents: detection, triage, communication, resolution, and postmortem. It includes status page update protocols, escalation chain design, and how to configure Vigilmon alert routing to feed this process correctly.
The Five Phases of Incident Response
1. Detection
2. Triage
3. Communication
4. Resolution
5. Postmortem
Each phase has specific actors, actions, and outputs. The playbook defines all five so that any engineer entering mid-incident knows exactly where things stand and what their role is.
Phase 1: Detection
Detection is the monitoring layer catching a failure before users report it. The quality of your detection directly determines your time-to-know — the gap between when the incident starts and when your team knows about it.
Detection Sources
Primary: Uptime monitoring alerts
Your uptime monitoring system is the first line of detection. Vigilmon dispatches checks from multiple geographic probe nodes simultaneously. An alert fires when a majority of those probes confirm the target is unreachable — not when a single probe has a bad moment.
This consensus requirement is critical for incident response: every alert that reaches your on-call engineer should represent a genuine failure. If your monitoring system generates false positives, engineers learn to hesitate before responding. Hesitation adds minutes to time-to-acknowledge. In production outages, those minutes matter.
Secondary: User reports
Users reporting failures via support tickets, social media, or direct contact. These are late-stage detection signals — the incident has been ongoing long enough for users to notice and take the time to report it. Incidents detected via user reports rather than monitoring represent a gap in your monitoring coverage.
Tertiary: Internal system alerts
Application error rate spikes, infrastructure resource exhaustion alerts, or deployment pipeline failures can indicate an incident before it fully manifests as user-visible downtime.
Detection SLO
Define a maximum acceptable time-to-detection (TTD) for each service tier:
| Service tier | Maximum TTD | Check interval | |---|---|---| | Critical production API | 3 minutes | 1 minute | | Standard production services | 10 minutes | 5 minutes | | Staging and internal | 15 minutes | 5 minutes | | Background jobs (heartbeat) | Window-dependent | N/A |
A 1-minute check interval means your worst-case detection time is 1 minute from failure onset. Add alert routing time (seconds for webhook delivery) and the engineer receives the page within 2–3 minutes of the incident starting. This is your TTD.
Detection Failure Modes
False negatives — monitoring misses a real outage: Monitoring is checking a health endpoint that returns 200 while the actual user-facing service is broken. Mitigation: check user-facing endpoints and validate response body content, not just internal health routes.
Delayed detection — monitoring interval is too long: A 15-minute check interval means a failure at minute 1 isn't detected until minute 16. Mitigation: use 1-minute intervals for critical paths.
Alert routing failure — monitoring fires but nobody receives it: Your webhook destination is down, your PagerDuty integration token expired, or the alert went to a channel nobody reads. Mitigation: test your alert pipeline end-to-end quarterly.
Phase 2: Triage
Triage is the 5–15 minutes after detection when the on-call engineer determines what is broken, how badly, and what needs to happen next.
Triage Checklist
When an alert fires, the on-call engineer should work through this list:
Confirm the incident (< 2 minutes):
- [ ] Is this a monitoring false positive or a real failure?
- [ ] Is the monitor check showing the failure consistently or intermittently?
- [ ] Can I reproduce the failure manually? (attempt the failing request from my browser or terminal)
- [ ] Are other monitors also alerting, or is this isolated?
Scope the impact (< 5 minutes):
- [ ] Which service is affected? (by name — "Payment API", not "monitor-4")
- [ ] Is this a complete outage or partial degradation?
- [ ] Which geographic regions are affected?
- [ ] Which users / customers are affected?
- [ ] Is the failure in our infrastructure or in a dependency?
Classify severity:
- [ ] P1: Complete outage affecting users now. Escalate immediately.
- [ ] P2: Partial outage or severe degradation. Begin resolution, escalate if unresolved in 15 minutes.
- [ ] P3: Minor degradation. Begin resolution within business hours. No overnight escalation.
Assign incident commander: For P1 incidents, designate one person as incident commander. This person is responsible for the incident process — not necessarily the technical resolution. Common failure mode: everyone is debugging; nobody is communicating. The incident commander ensures communication happens while others debug.
Triage Anti-Patterns
Diving straight to root cause: The instinct to immediately open databases and check application logs is natural but premature. First confirm scope — is this affecting all users or one customer? Is this one service or multiple? Root cause investigation before scope confirmation can waste time investigating the wrong thing.
Solo triage without communication: When an engineer receives a P1 alert at 2 AM and starts debugging without notifying anyone, the incident is untracked. If they resolve it quickly, great. If they don't, 40 minutes pass before anyone else knows there's an incident. Notify immediately; investigate in parallel.
Premature root cause confidence: "It's probably the database" is a hypothesis, not a diagnosis. Triage should scope and classify; root cause comes in resolution.
Phase 3: Communication
Communication during an incident is as important as technical resolution. Users who don't know what's happening create support volume, social media noise, and churn. Users who receive timely, honest updates tolerate incidents much better.
Internal Communication
Incident channel: Create a dedicated Slack channel for each P1 incident: #incident-2026-06-30-payment-api. All incident communication goes there. This creates a real-time log of the incident for postmortem review and keeps the noise out of general engineering channels.
Status updates: Post a brief update every 10 minutes until the incident is resolved. Even if you have nothing new to report: "Still investigating. No user data affected. Will update in 10 minutes." Silence creates uncertainty. Uncertainty leads to stakeholder interruptions that pull engineers away from resolution.
Stakeholder notification: For P1 incidents affecting users, notify:
- Engineering lead immediately
- Customer success / support team so they can handle inbound
- Product leadership within 30 minutes of P1 declaration
- Executive team if the incident extends beyond 1 hour
External Communication: Status Page Updates
Status page updates are the external equivalent of your internal incident channel. Users and customers who check your status page during an incident need timely, honest information.
Status page update cadence:
| Incident age | Status page action | |---|---| | 0–10 minutes | Identify impact; post initial update: "Investigating reports of issues with [service]" | | 10–20 minutes | Update with scope: "We have identified [X] as affected. Engineers are working on resolution." | | Every 20 minutes | Post "Update: [brief status]. Estimated resolution time: [estimate or 'not yet determined']" | | Resolution | Post "Resolved: [service] is fully operational. We will follow up with a postmortem." |
What to say on your status page:
Use plain language. Avoid jargon that confuses non-technical users. Be honest about what you know and what you don't:
- "We are experiencing intermittent failures in our payment processing API. Our team is actively investigating." ✅
- "There is a P1 SEV1 incident in prod env affecting the payments microservice due to a suspected database connection pool exhaustion." ❌ (technical, not user-oriented)
Include the user impact: "During this period, some users may be unable to complete purchases." Users want to know if this affects them — tell them directly.
What not to say:
- Don't speculate about root cause publicly until you're confident.
- Don't promise resolution times you can't keep. "We expect resolution by 3:00 PM" that slips to 5:00 PM damages trust more than "Resolution time not yet determined."
- Don't minimize the impact. Users experiencing failures know they're real; telling them "intermittent issues affecting a small number of users" when the service is fully down is visibly dishonest.
Vigilmon Status Badge
Vigilmon provides an embeddable status badge you can include on your website or developer portal. When monitors are reporting outages, the badge automatically reflects degraded status — giving users a real-time indicator without manual status page updates for every check failure.
For planned maintenance, update your status page in advance and inform users via email. Users who know maintenance is coming don't generate support tickets when it happens.
Phase 4: Resolution
Resolution is the technical process of restoring service. The structure here depends heavily on your stack, but the incident response process around resolution is generic.
Resolution Roles
Incident commander: Tracks time, manages communication cadence, decides when to escalate, and declares resolution. Does not debug unless the team is small enough that there's no option.
Technical lead: Drives root cause investigation and directs remediation. One person, not a group. When multiple engineers are independently trying different fixes, they can interfere with each other and make diagnosing the fix harder.
Support liaison: Monitors inbound support volume and communicates findings back to the incident channel. This role prevents support team interruptions from pulling engineers out of debugging flow.
Resolution Steps
Stabilize first, investigate second: If there's a quick mitigation available — restart a service, failover to a backup, disable a feature flag — do it first. Restore user access, then investigate root cause. A 20-minute outage resolved by a restart, followed by thorough investigation, is better than a 45-minute outage while you understand exactly why before taking action.
Document your investigation as you go: In the incident Slack channel, post what you've checked and ruled out. "Database connections: normal. Application error logs: seeing connection timeouts to Redis. Cache cluster appears to be the issue." This timeline becomes the postmortem source material.
Define resolution criteria before calling it resolved: What does "resolved" mean for this incident? "HTTP 200 returning" may not be sufficient if the service has a warmup period where it returns 200 but is functionally degraded. Define the criteria — all monitors green, manual testing confirms core flows, error rate returned to baseline — before declaring resolution.
Hold for 10 minutes after stabilization: Premature resolution declarations followed by immediate re-escalation are demoralizing and confusing. After the service appears stable, hold the incident open for 10 minutes while monitoring confirms consistent recovery before posting the resolved update.
Escalation Triggers
Escalate to the next level when:
- Triage has been running for 15 minutes with no scope clarity
- Root cause investigation has been running for 30 minutes with no leading hypothesis
- Resolution requires access, tooling, or knowledge held by a specific engineer not currently on call
- The incident scope has expanded beyond initial assessment
- External communication requires executive sign-off
Define your escalation chain before incidents happen:
Level 1: Primary on-call engineer
↓ (5 min no acknowledge, or 30 min no resolution)
Level 2: Secondary on-call engineer
↓ (15 min no acknowledge from level 2, or 45 min no resolution)
Level 3: Engineering lead
↓ (1 hour no resolution, or user-impacting P1)
Level 4: VP Engineering / CTO
Document who is in each level for each rotation period. An escalation chain that nobody updates becomes a chain that escalates to people who left the company.
Phase 5: Postmortem
The postmortem converts an incident from a cost into an investment. Done well, postmortems prevent future incidents, improve detection speed, and improve incident response processes.
Blameless Culture
Postmortems that assign blame produce engineers who hide mistakes. Postmortems that treat incidents as system failures — even when a human action contributed — produce engineers who share information freely.
A blameless postmortem asks:
- What conditions made this failure possible?
- What in our systems, processes, or tooling allowed this to happen?
- What would have prevented it, or detected it sooner?
Not: who made the mistake that caused this?
The goal is action items that reduce future risk — not accountability for past events.
Postmortem Structure
Summary (2–3 sentences): What failed, for how long, and how many users were affected.
Timeline: Chronological log of the incident, from first detection to resolution. Include timestamps. This should be reconstructible from your incident Slack channel — which is why you post updates there in real time.
Root cause: One clear sentence. Not "multiple contributing factors" — that phrase usually means the investigation wasn't completed. Dig until you can state the root cause specifically.
Contributing factors: What conditions made this failure possible or harder to resolve? Examples: no test coverage for this path, monitoring only checked status code not response body, runbook was outdated, the service had no circuit breaker.
Detection analysis:
- How was the incident detected? (monitoring, user report, internal alert)
- How quickly was it detected after onset?
- Could detection have been faster? How?
Resolution analysis:
- What steps were taken?
- Were any steps ineffective or counterproductive?
- Were the right people involved at the right times?
- What slowed resolution that could be systematized?
Action items: Specific, assigned, time-bounded tasks that reduce the risk of recurrence or improve response. Not "improve monitoring" — "add response body validation to the payment API monitor by [date], assigned to [name]."
Postmortem SLO
Postmortems should be completed within 5 business days of the incident while context is fresh. Postmortems written two weeks later are incomplete; the timeline is fuzzy and contributing factor memory has faded.
For P1 incidents, publish the postmortem internally and share a customer-facing version with affected enterprise customers. The customer-facing version covers timeline, impact, and what you've done to prevent recurrence — not root cause details that might reveal security or business-sensitive information.
Vigilmon Alert Routing for Incident Response
Webhook Configuration for Severity Tiers
Configure Vigilmon webhooks to route alerts to your incident management tools based on monitor naming and criticality:
// Vigilmon DOWN webhook payload
{
"monitor_id": "mon_xyz789",
"monitor_name": "Production API - Payment Processing",
"monitor_type": "http",
"status": "down",
"timestamp": "2026-06-30T02:47:33Z",
"consecutive_failures": 3,
"response_time_ms": null
}
# webhook_router.py — route by monitor name prefix
def classify_incident(monitor_name, status):
if "Production" in monitor_name and status == "down":
return "P1"
elif "Production" in monitor_name:
return "P2"
elif "Heartbeat" in monitor_name or "Backup" in monitor_name:
return "P2"
elif "Staging" in monitor_name:
return "P3"
return "P3"
Monitor Naming for Self-Describing Alerts
The alert that reaches your on-call engineer should be immediately actionable without a lookup. Use monitor names that describe the service, environment, and check type:
# Good monitor names
Production API - Payment Processing - Health Check
Production API - Authentication - Login Endpoint
Staging API - Homepage Load
Production DB - Connection Port TCP
Production SMTP - Delivery Heartbeat (daily)
SSL Certificate - api.example.com
# Poor monitor names
api-check
monitor-1
test
production
An alert that says "Production API - Payment Processing - Health Check is DOWN" is immediately actionable: it's production, it's the payment path, it's the health endpoint. An alert that says "monitor-1 is DOWN" requires a dashboard lookup before triage can begin.
Heartbeat Monitor Configuration
Configure heartbeat monitors for every scheduled job that affects users:
| Job | Typical duration | Heartbeat window | Severity | |---|---|---|---| | Payment reconciliation | 45 min | 75 min | P1 | | Email notification sender | 5 min | 15 min | P2 | | Report generation | 20 min | 35 min | P2 | | Database backup | 30 min | 50 min | P2 | | Log rotation | 2 min | 10 min | P3 |
Window at 150–200% of typical duration to account for load variance without generating false positives.
Incident Response Quick Reference
On alert receipt (on-call engineer):
- Acknowledge the alert within 5 minutes
- Manually reproduce or confirm the failure
- Classify severity (P1/P2/P3)
- Open incident Slack channel for P1
- Notify engineering lead for P1
- Post initial status page update for P1
Every 10 minutes during P1:
- Post status update in incident channel (even if no change)
- Post status page update
Resolution gates:
- All monitors green for 10+ minutes
- Manual testing confirms core user flows
- Error rate at baseline
Postmortem:
- Complete within 5 business days
- Assign all action items with owners and dates
- Share customer-facing version with affected enterprise accounts
Conclusion
Incident response quality is a competitive advantage. Teams that detect outages quickly, communicate clearly, and restore service efficiently retain users and trust even through severe incidents. Teams that detect outages slowly, communicate poorly, and resolve erratically compound the damage of every incident with preventable additional harm.
The playbook isn't about making incidents more bureaucratic — it's about making them faster and less stressful. When every engineer knows the detection criteria, the triage checklist, the communication cadence, and the escalation chain, nobody has to make decisions under pressure. They execute a documented process.
Vigilmon's multi-region consensus alerting is the detection foundation: every alert that reaches your on-call engineer represents a genuine failure confirmed by independent probes from multiple geographic locations. Combined with clear alert routing, descriptive monitor names, and heartbeat monitoring for background jobs, it feeds the incident response process with reliable signal.
Start building your detection layer at vigilmon.online — free account, consensus alerting by default, webhook notifications to your incident management platform, up and running in under 5 minutes.
Tags: #monitoring #incidentresponse #uptime #oncall #sre #devops #vigilmon #playbook #2026