Black Hawk Down

I’m guessing a lot of us in the IT and Security space have experienced the gut wrenching feeling of not receiving that ICMP ping reply you were expecting from a production system, be it a firewall, switch, or server. Was there a recent configuration change that happened prior to the last reboot? Did a hard drive die? Firmware corruption? Did the new intern accidentally flip a power switch or thought they were in their home directory when they typed sudo rm -rf with their new privileges?

Thankfully, we have backups to help us get back on our feet and restore the systems to a production state. For decades, backup solutions have been architected to help system and network administrators cope with the uncertainty related to hardware, software, and even the people who manage them. And more often than not, backups have historically been used to bring one, or even a handful of systems back from the brink. Replace a drive, restore the system from a backup, and in a few hours, everything is back to status quo.

But what happens to an organization if most or all of their systems go lifeless? If the restoration process for one (1) server is, for argument’s sake, four (4) hours, how long will it take to restore all 150 servers in my enterprise? 600 hours? Worst-case scenario, that’s 25 days of working 24 hours a day. In this timeframe, can my organization withstand not being able to do most things that are required to continue generating revenue? What about supporting critical infrastructure and ensuring lifesaving equipment in hospitals is not interrupted?

Colonial Pipeline Down

Traditional backup solutions and architecture were never really designed for the rapid restoration of many systems at once. But this is now the challenge for many organizations, since the scenario described above has happened to many businesses as a result of successful ransomware attacks. One minute all systems are healthy, the next minute you have a message stating, 'all your database are belong to us, we require $2 million in bitcoin, if you refuse, we release all your sensitive data on dark web. Please call 888.555.5555 to contact our tech support if you need help converting or sending bitcoin.'

As a result, organizations hit with successful ransomware attacks are trying to rely on traditional backup systems, which were never designed to be attack response systems. Of course, that's only IF the organization even has backups, since the threat groups working the ransomware game know they’ll have a higher likelihood of receiving their demanded ransom if they can find and destroy the target’s backups prior to the encryption attack.

The Paradigm of Traditional Backup Solutions Has Shifted

What was once a novel attack scenario has evolved into a global enterprise business model, full of ransomware-as-a-service providers selling their proven ransomware for a share of the ransom profits. It is crucial that IT leadership teams require multi-factor authentication (MFA) on backup administrator accounts, properly segment backup systems and disaster recovery sites from corporate networks, and begin looking at solutions like immutable snapshot systems that are exceptionally difficult to tamper with and can restore systems relatively quickly.

It is equally as crucial that business leadership teams change how they view backups, disaster recovery, and business continuity. What systems that support the most critical aspects of the business need to be recovered first? How long will that take? Is that length of time acceptable? If not, what will it cost to deploy systems that can restore the critical services within that acceptable timeframe? What are the contractual and legal obligations to clients and partners? How much would the organization be fined for regulatory violations as a result of a ransomware attack if it was found not to have performed adequate due diligence in securing that now-tampered or stolen data?

Be Like Water, My Friend

The situations that backup solutions are being asked to handle have evolved along with technological advances and the associated malicious use of technology. And like most aging technologies, traditional backups still have an important role for organizations, like restoring that SQL database that fell down. But the same solutions are not enough to rapidly and effectively restore systems that have been subjected to the full assault of a successful ransomware attack. Organizations must continue to look forward and adapt to these changing tactics.