Skip to content
~/leonardo.casoni

disaster recovery

Disaster Recovery: getting back to work after a serious failure

It is not a matter of if it will happen, but when. A good disaster recovery plan does not prevent the problem: it tells you what to do and how long it will take.

What it is

Disaster Recovery (DR) is the set of plans, tools and procedures to restore systems and data after a serious event: the failure of a whole cloud region, a human error, a ransomware attack, data loss.

It is part of a wider topic, Business Continuity, which is about how the company keeps working during and after a crisis, with people and processes, not only systems.

Not to be confused

High availability protects against small failures, disaster recovery against big ones. You need both.

Backup

A copy of the data, to restore from when it is lost or damaged.

High availability

Redundancy to survive a failure with no interruption, for example across several zones.

Disaster Recovery

Recovery after a serious event, often in another place or region.

Business Continuity

Keeping the whole business going, people and processes included.

Two numbers to decide: RTO and RPO

The lower the numbers, the more the solution costs. That is why not every system has the same targets.

RTO
Recovery Time Objective: how quickly the service must be back. With a 4-hour RTO, after the disaster you have 4 hours to restart.
RPO
Recovery Point Objective: how much data you can afford to lose, measured in time. With a 15-minute RPO you need copies or replicas at least every 15 minutes.

The strategies, from simplest to most expensive

These are the four strategies also described in AWS guidance. Times are indicative and depend on the system.

StrategyHow it worksRecoveryCost
Backup and restoreYou keep copies of the data and, if needed, rebuild everything from there.HoursLow
Pilot lightA minimal core, for example the replicated database, always runs on the other site. The rest is switched on when needed.Tens of minutes or hoursMedium-low
Warm standbyA scaled-down but working version of the system is always on and ready to grow.MinutesMedium-high
Multi-site active/activeSeveral sites serve traffic together: if one fails, the others keep going.Almost noneHigh

How to build a plan

  1. 01

    Understand what matters

    List the services and assess what stopping them costs: this is the business impact analysis.

  2. 02

    Set RTO and RPO

    For each service, together with the people accountable for the business, not alone.

  3. 03

    Choose the strategy

    Look for the balance between targets and cost, service by service.

  4. 04

    Automate

    Infrastructure as code to recreate environments, and automatic backups that you check.

  5. 05

    Write procedures and roles

    Who does what, and in which order. One copy must be reachable even when the systems are down.

  6. 06

    Test and improve

    Run restore tests, simulations and failover drills at regular intervals.

Backups done well

  • The 3-2-1 rule: at least 3 copies of the data, on 2 different media or systems, with 1 off-site.
  • Immutable or isolated copies, to withstand ransomware.
  • Encryption of copies and access control.
  • Monitoring of backup jobs: a backup that fails silently is worse than no backup.
  • Regular restore tests. Until you have restored it, you do not know whether it works.

Common mistakes

  • Backups never restored, found useless on the wrong day.
  • Forgetting dependencies: DNS, identity and permissions, secrets, certificates.
  • Keeping the procedures only inside the system that is down.
  • Confusing high availability with disaster recovery.
  • Not updating the plan when the infrastructure changes.
  • Deciding RTO and RPO without involving who is accountable for the business.

This is an introductory guide, written the way I understood things while working in the field. Every company organizes itself differently: take it as a starting point, not as a manual.

Keep reading