Skip to content
~/leonardo.casoni

incident management

Incident Management: what to do when something breaks

Incidents happen to everyone. What makes the difference is how we face them: calmly, in an orderly way, and willing to learn.

What it is

An incident is an unplanned event that interrupts a service or lowers its quality. Incident Management is the process that restores normal service as quickly as possible, limiting the impact on users and business.

The goal is not to never have incidents, because that is impossible. The goal is to notice early, respond well and not repeat the same mistake.

Not to be confused

Event

Something that happens in a system: a log line, a metric that changes. Most of the time it is normal.

Alert

An event that needs attention and notifies someone.

Incident

A real or imminent disruption that must be handled now.

Problem

The underlying cause of one or more incidents. It is studied to keep them from coming back.

The life of an incident

  1. 01

    Detect

    The problem shows up through monitoring, an alert or a user report. Better if before the users.

  2. 02

    Classify

    You assess impact and urgency and assign a priority, which decides who is involved and how fast.

  3. 03

    Respond

    You call the right people, open a channel or ticket and name who coordinates.

  4. 04

    Mitigate and resolve

    First restore the service (a rollback, a restart, a move to another system), then investigate the cause.

  5. 05

    Communicate

    You update colleagues, customers and managers regularly, even when the cause is still unclear.

  6. 06

    Learn

    You run a blameless review, decide concrete actions and check they get done.

Roles during an incident

In a serious incident, clear roles keep everybody from doing everything while nobody coordinates.

Incident Manager

Coordinates, decides and keeps the big picture. Not the person who fixes it first-hand.

Responders

The technical experts who investigate and apply fixes.

Communications

Updates stakeholders and customers with clear, regular messages.

Scribe

Keeps the timeline of what happens and when: gold for the final review.

An example of priority levels

It is just an example: every company defines its own levels and response times.

PriorityExampleTypical reaction
P1 · CriticalThe main service is down for everyone.Everyone, right away, even at night, with continuous updates.
P2 · HighAn important feature is degraded or down for many users.Immediate action by the on-call person.
P3 · MediumA partial problem, with a workaround.Handled during working hours.
P4 · LowA minor annoyance, with no impact on the service.Goes to the backlog and gets planned.

How it is measured

MTTD
Mean time to notice the problem.
MTTA
Mean time to acknowledge the alert.
MTTR
Mean time to restore the service.
Incidents by priority
How many happen, by priority, over time.
SLA / error budget
How much disruption you can afford compared with the commitments made.

What works

  • Actionable alerts: every alarm should lead to a clear action.
  • Up-to-date runbooks that are easy to find.
  • A sustainable on-call, with fair rotations and good handovers.
  • Communicating early, even just to say “we are investigating”.
  • Mitigate first, understand later: bring the service back, then look for the cause.
  • Blameless reviews, with actions assigned to someone and to a date.
  • Practicing: simulate incidents (game days) before they really happen.

Common mistakes

  • Too many alerts, to the point of ignoring them (alert fatigue).
  • No coordinator: everyone jumps in and nobody has the full picture.
  • Looking for who is to blame instead of the cause.
  • Writing the post-mortem and then not completing the actions.
  • Communicating only after the incident is over.

This is an introductory guide, written the way I understood things while working in the field. Every company organizes itself differently: take it as a starting point, not as a manual.

Keep reading