On the morning of July 19, 2024, the world awoke to disruption on a global scale, with critical systems for banking, airlines, and innumerable other companies knocked offline.

The outage affects machines using the Microsoft Windows operating system. Fixes and postmortem are in the works, but it is undetermined as of this writing how long the outage will last.

Related Company Profiles

View all

However one thing is clear – the culprit is not a rogue band of hackers or nation-state but rather, security vendor CrowdStrike.

Systems testing

As the fallout from this incident continues, there needs to be a discussion at high levels in corporate IT departments concerning policy on software updates. In the pre-cloud era, most updates to end-user systems and servers were first vetted and tested by the IT department. The process looked something like this:

  • Review update changelog – note any possible problems and decide if the update has enough changes to warrant a roll-out;
  • Install the update on isolated test system(s);
  • Test the update for problems and functionality;
  • Roll the update out to a small subset of users, preferably those close by and in the IT department;
  • If all seems well over the next few days, roll out to a larger test group;
  • If that goes well, schedule the update corporate-wide in batches, staggered to ensure that the updates do not all happen at once and crash network/server resources.

In the case of equipment like Ethernet switches and routers, testing to ensure that updates do not cause widespread outages can take months. Dedicated storage systems, server BIOS upgrades and similar high-risk patches all need to be very carefully tested and rolled out.

Cloudy skies

But the cloud has changed things, with updates happening automatically. There are a lot of good reasons for automatic updates, among them the time it saves IT staff and most importantly how much better it is from a cybersecurity standpoint.

How well do you really know your competitors?

Access the most comprehensive Company Profiles on the market, powered by GlobalData. Save hours of research. Gain competitive edge.

Company Profile – free sample

Thank you!

Your download email will arrive shortly

Not ready to buy yet? Download a free sample

We are confident about the unique quality of our Company Profiles. However, we want you to make the most beneficial decision for your business, so we offer a free sample that you can download by submitting the below form

By GlobalData
Visit our Privacy Policy for more information about our services, how we may use, process and share your personal data, including information of your rights in respect of your personal data and how you can unsubscribe from future marketing communications. Our services are intended for corporate subscribers and you warrant that the email address submitted is your corporate email address.

Case in point, antivirus definition updates were one of the first widespread examples of automatic updating. Automatic updates also prevent zombie unpatched systems from propagating malware or providing other security holes.

The primary driver for widespread automatic updates has been cybersecurity. The ability to very quickly and automatically patch systems in response to real time threats is compelling.

Time for a rethink

Two factors should be making enterprise IT rethink the assumptions around automatic updates. First, is the loss of control. The responsibility of ensuring that updates work smoothly moved from enterprise IT’s hands and into the hands of vendors. Enterprise IT has no control over those vendors, or control over their practices, and often hasn’t even investigated those vendors’ practices. Big vendors serving thousands of companies have extensive quality control and practices, right?

In truth, most vendors do have excellent quality control and safe practices. But its still a trust relationship where the control really lies with the vendor.

We are also seeing more of the so-called ‘supply chain’ attacks, that attack components of a software vendor’s solution, such as libraries and services. This is another instance where IT has been simply trusting the vendor to ensure that those software components are secure and up to date with the automatic updates.

Putting on the systems updating brakes

Enterprises need to consider putting the brakes on automatic updating and moving to a process where IT is more involved and in the path of approval. Responsibility lies with the enterprise, even if the mistake is a vendor’s.

This will require more time and resources, but it’s a luxury to offload the responsibility for software updates to vendors. Airplane pilots inspect their planes themselves, despite having seasoned ground crew and mechanics on hand. In short, everybody must agree that all lights are green for a plane to fly.

Enterprises need to again step up and take an active role in managing software systems updates, even urgent ones. The responsibility and accountability needs to come back in-house. Otherwise the next time a fiasco like the CrowdStrike one occurs, enterprises won’t have anyone to blame but themselves.