On-call at 2am with an unknown alert

What the Prompt Is Testing

A prompt such as, "Tell me about a time you handled an unfamiliar production alert," is testing whether you can:

• assess user impact and scope before changing production;
• communicate clearly while the facts are incomplete;
• form and test a hypothesis, choose a safe action, and escalate when needed; and
• turn the incident into learning through documentation and prevention.

If you have not been on call at 2am, use a real incident, support escalation, or debugging session that shows the same judgment. State the difference instead of presenting this payment example or a planned response as your own history.

30-Second Opening

State the incident, your role, your first decision, and the outcome. Replace the brackets with facts from your own experience:

During [real incident], [service or users] experienced [impact]. I was [role]
and was responsible for [scope]. I first [safe assessment or containment], then
used [signal or hypothesis] to [action]. We [measured result], and afterward I
helped add [runbook, alert, test, or other prevention].

This is a template for organizing a true story, not a script to claim verbatim.

A Practical Incident Framework

In an illustrative scenario, you have been woken up by a PagerDuty alert. It is 2am. The alert is PaymentService_HighErrorRate. You have been on call for two weeks but have not touched the payments service before.

A common instinct is to immediately search logs, change something, or wake up a senior engineer. A safer sequence starts with impact and scope, then moves to evidence, communication, and a controlled action.

Step 1 (first minute): Assess severity before anything else

Questions in this order:
1. What is the user impact? (Are users actively failing, or is this a metric threshold?)
2. What is the scope? (100% of traffic? One endpoint? One region?)
3. Is this getting worse, stable, or improving?

A p99 latency spike that's already trending down might resolve itself. An error rate at 100% requires immediate action. These need different responses.

Check: your monitoring dashboard, not just the alert. Alerts fire on thresholds; dashboards show trends.

Step 2 (first 2 minutes): Post a status update before investigating

This feels backwards but it's critical. If you don't post anything, your manager, other on-call engineers, and downstream teams will start paging you.

Template for your incident Slack / Pagerduty note:
"[2:03 AM] Paged for PaymentService_HighErrorRate. 
 Currently assessing scope and impact. 
 Will update in 15 minutes or if I need to escalate."

This buys you investigation time and establishes that someone is engaged.

Step 3 (next 8 minutes): The structured investigation

Step 4: Escalate early, not late

Trying to "figure it out before waking anyone else up" can increase risk. Escalate if:

• You've spent 15 minutes without a clear hypothesis
• The impact is growing, not contained
• The service is outside your expertise
• You need a second set of eyes on a risky fix

Escalation message template:
"Hey, I need eyes on PaymentService. 
 Error rate is at 40% for 15 minutes. 
 No recent deploys. 
 I've ruled out DB latency (normal) and external rate limits (normal). 
 My next hypothesis is X but I'm not confident. Can you join a call?"

Early escalation is a normal incident-control practice. During an active incident, a second responder can add context, challenge a risky hypothesis, or take over communication.

The Fix Discipline

Two rules that prevent making things worse:

Rule 1: Never apply a fix you don't understand. "Restarting the service" is not a fix — it's a gamble that restoring state will clear the problem. If it works, you'll get paged again tonight, or next week. If it makes things worse, you've made the incident harder.

Rule 2: Change one thing at a time. If you change the config, restart the service, and roll back the deploy simultaneously, and things improve, you don't know what fixed it. You've made the incident harder to learn from.

Before any fix: 
"My hypothesis is X. This fix targets X. 
 If it works, I expect to see the error rate drop within 2 minutes. 
 If it doesn't work, I'll revert immediately."

Write this in the incident channel before you apply the fix. It forces clarity and creates an audit trail.

The Post-Fix Checklist

After the error rate drops, you're not done:

☐ Update status: "Error rate back to normal at 2:37 AM. Root cause: [brief]"
☐ Watch for 10 minutes to confirm stability
☐ Document what you found and what you changed in the incident channel
☐ Assess: is there work needed during business hours?
☐ Write an incident summary if severity was high enough
☐ File a ticket for any quick fixes you applied that need proper solutions

It is tempting to treat the incident as finished as soon as the metric recovers. A short record of the evidence and changes made helps the next responder and gives the team a basis for follow-up.

Build a 2–5 Minute STAR Answer

Use one real incident or debugging session. Replace bracketed details with your own evidence; do not present the illustrative payment alert as a personal event.

When asked about being on call or responding to incidents, the answer should make these qualities visible:

• Systematic thinking — did you follow a process, or thrash?
• Communication — did you keep stakeholders informed?
• Escalation judgment — did you know when you were in over your head?
• Post-incident discipline — did you document and follow through?

The best answers include: the specific signals you looked at, the hypotheses you formed, the points where you chose to escalate and why, and what the post-incident work looked like.

Situation (30–45 seconds)

Describe [the service or system], [the user impact], and [the alert or symptom]. Include why it mattered and what you knew when the incident began.

Task (15–30 seconds)

State your role and responsibility: [first responder], [service owner], [incident coordinator], or [investigator]. Name the immediate goal, such as [restore the critical path safely] or [determine whether the alert represented real user impact].

Action (1–3 minutes)

Walk through the sequence you actually followed: [assess scope], [post a status update], [check recent changes], [form and test a hypothesis], [escalate], and [apply or roll back one controlled change]. Explain what you deliberately did not change and why.

Result (30–60 seconds)

State the measured or observable outcome: [error rate returned toward baseline], [users recovered], or [the alert was tuned]. Then add [the runbook, test, alert, or incident follow-up] that reduced recurrence risk. If the outcome is not fully known, say that and explain how it will be measured.

The outline takes about 3–4 minutes at a normal pace. Keep the operational framework as technical depth, and spend the interview answer on your decisions, communication, and evidence.

Weak: "I got paged and eventually figured out the problem and fixed it."
Illustrative strong shape (replace the brackets; do not claim this event unless it happened to you):
"At [time], I was paged for [alert] affecting [scope]. I checked [signal] and
 found [impact or trend]. After confirming [rollback or mitigation] was safe,
 I [action], and [metric] moved toward [baseline] within [measured time]."

Choose and Adapt a Real Story

Choose a real event where you can explain your own decisions, not only the final fix. Useful stories include:

• a production incident, support escalation, or failed deployment;
• an alert that was noisy, incomplete, or unfamiliar;
• a moment when you had to choose between containment, rollback, and more investigation; and
• a follow-up that improved a runbook, alert, test, or ownership process.

The event does not need to happen at 2am or involve PagerDuty. If you were not the primary responder, say who was and describe your part: [gathering evidence], [coordinating communication], [testing a hypothesis], or [implementing the follow-up].

Use this honest bridge when you lack a direct on-call story: "I have not been the primary responder for a 2am payment incident. My closest example is [real event], where I [actual contribution]. The same principle would apply here: [hypothetical adaptation]." Keep the final clause clearly hypothetical.

Follow-up Questions and How to Answer

"Why did you check impact before logs?" Explain that scope and trend determine urgency and help you choose between containment, deeper investigation, and escalation.

"Why did you not just restart the service?" Describe the hypothesis, the risks of a restart, and the safer reversible action you considered. If a restart was appropriate, explain what evidence made it safe and what you monitored afterward.

"When did you escalate?" Name the trigger—growing impact, no credible hypothesis after a defined interval, unfamiliar ownership, or a risky change—and say what context you gave the next responder.

"What if there had been no recent deploy?" Show the next branch of the investigation: dependency health, database behavior, traffic, queues, capacity, or provider status, chosen according to the alert and service architecture.

"What changed after the incident?" Point to one concrete prevention step and how it will be tested or measured. Do not claim the incident can never recur.

Common Interview Mistakes

• Starting with a tool or log query before explaining user impact and scope.
• Listing several simultaneous changes without a hypothesis, expected result, or rollback plan.
• Treating escalation as failure instead of explaining how it reduced risk.
• Overclaiming your role, root-cause certainty, recovery time, or prevention result.
• Presenting this article's payment scenario or a proposed response as personal experience. Use "I did" for work you actually did and "I would" for a hypothetical adaptation.

Test Your Understanding

1. What should come before a production change?

Assess impact and scope, form a hypothesis, and state what result you expect from the change.

2. When is escalation appropriate?

When impact is growing, the next action is risky, the service is unfamiliar, or you cannot form a credible hypothesis within the agreed interval.

3. What makes the Action part of STAR convincing?

The sequence of evidence, decisions, communication, and controlled changes—not merely the tool names.

4. What should you say if you were not the person who fixed the incident?

Give that person credit and explain your own contribution precisely, including what you learned and what changed afterward.

Quick Recap

1. Before investigating, assess severity: what's the user impact, what's the scope, and is it getting worse?
2. Post a status update before you start debugging — it prevents escalation pile-on and establishes ownership.
3. Investigate in order: scope → recent change → service-specific signals. A recent change can narrow the search faster than debugging the symptom without context.
4. Escalate early: spending 15 minutes without a clear hypothesis is the escalation trigger, not the thing to push through alone.
5. Document before and after every fix: hypothesis, expected outcome, result. The audit trail matters for the post-mortem.

Related Skills

• Incident response: assessing impact, containing harm, and coordinating a safe recovery.
• Debugging: turning signals into hypotheses and testing them in a controlled order.
• Observability: designing alerts and dashboards that show scope, trend, and service health.
• Communication and ownership: keeping responders and stakeholders aligned before, during, and after the incident.