A new report out from the US government confirms earlier reports that a power outage in the Ukraine was the result of a hacker, likely Russian. Despite some earlier false alarms this would appear to be the first publicly confirmed major utility outage that can be attributed to hackers. Many of us in the security industry have been predicting this for years and there is also a long way to go in order to mitigate this threat now that it has finally come to pass. The risk goes beyond power plants as well: sewage treatment systems, traffic lights, CCTV systems, air traffic control, and many other pieces of critical infrastructure are now online.

The fact that attackers are looking to target infrastructure devices and other industrial control systems is not new. A pair of research papers from Trend Micro describe attacks that they observed on a set of honeypots, simulating water treatment systems, that they created and exposed to the Internet. While some of the attackers immediately took actions that would have caused real damage had they not been connected to cleverly designed simulations of their target systems others appeared more interested in creating backdoors that they could use to re-gain access at a later date. It is also relevant that the majority of the attacks were launched from China and Russia, two countries that regularly end up on top of the list of usual suspects when these sorts of attacks occur.

The idea that someone is installing backdoors in our infrastructure and waiting for a reason to use them, a sort of sleeper cell, presents much more of a risk than immediate attacks: when systems are breached and immediately damaged the consequences should be relatively limited in scope and duration but when a coordinated attack is launched on many systems that have been compromised over the course of years the threat of severe long-term damage is much greater. For example: if one power plant were to be brought down every week for a year it is likely that the rest of the power grid could handle the extra load while the offline plants are repaired and brought online one by one but if dozens of power plants were brought offline simultaneously the result would be a widespread and long duration outage. If a widespread attack like this were strategically timed, perhaps during a winter storm in the northern latitudes, the consequences could be even more severe as homes lose heat while homeowners are trapped inside.

There are two fundamental security problems that make this threat possible: Much of the software that powers industrial control systems was not designed with security in mind, and these types of systems are being connected and exposed to the Internet.

Many of the network protocols that industrial control systems utilize have their origins in the 1970's, an era when breaking into a network would require gaining physical access to a facility. There was no emphasis placed on the standard security protocols in use today, e.g. properly authenticating commands, checking authorization, and encrypting connections. It was implicitly assumed that every device on the network could be trusted and that any command must be legitimate. Now that attackers can gain remote access to these networks the problems presented by these insecure protocols becomes more acute: if the systems are designed to trust any command that comes across the network then they will treat any commands issued by an attacker as legitimate. A Department of Homeland Security advisory issued in response to a vulnerability in a Siemens control unit a few years ago describes this underlying issue succinctly "According to ICS-CERT analysis, the ISO-TSAP protocol is functioning to specifications; however, authentication is not performed nor are payloads encrypted or obfuscated. [...] Like ISO-TSAP, many protocols used in industrial control systems were intentionally designed to be open and without security features."

Protocol design failures aside, industrial control systems are also susceptible to the same types of exploitable vulnerabilities that we find in even modern software, from buffer overflows to backdoor passwords. The more security researchers look at these systems the more vulnerabilities they find and vendors have not been very proactive about fixing these issues. Even more concerning is that many administrators seem to be taking an "if it ain't broke don't fix it" approach to patching these types of systems which leaves known and fixable vulnerabilities exposed. Unfortunately these systems are invisibly broken in that the potential consequences of an attack are much greater than some scheduled downtime for preventative maintenance.

We can't rely on the concept that our systems are so obscure that an attacker won't find them. The Shodan search engine indexes devices that are connected to the Internet and makes it quick and easy to find these types of systems. If a new vulnerability is identified in a particular make, model, and version of a device it only takes seconds to punch those parameters into a search and find hundreds, if not thousands of potential victims. Custom scanning tools can be even more effective in looking for exploitable systems as networks have gotten faster and scanning the entire Internet for vulnerabilities can be accomplished in under an hour.

In light of these design issues and software vulnerabilities, exposing systems with the potential to cause real-world damage to hackers half a world away is clearly a dangerous thing to do yet there are thousands upon thousands of these systems connected to the Internet with no effective firewall to prevent attackers from finding and exploiting them. I'm hard pressed to imagine a valid business reason for taking the risk of exposing these systems to the Internet but yet there they are. I suspect this isn't so much a conscious business decision as it is a result of a pervasive "connect everything to the network" mentality but the lack of firewall rules to prevent access to these systems is inexplicable. Even with firewall rules in place, there are plenty of ways for an attacker to gain a foothold in a target network and if an industrial control system is designed to "trust everything on the network" then any vulnerability in the network that the control system is attached to should be considered a vulnerability that exposes the control system itself.

In an ideal world the vendors behind these devices would design more secure protocols and look for vulnerabilities in their own products while network administrators would regularly apply security patches and harden the configurations of these devices to the best of their ability. Unfortunately these types of devices have a long service life and retroactively fixing them all likely isn't feasible. We can only hope to put enough pressure on the device manufacturers to improve the security of the next generation of devices so we are not still dealing with this problem 50 years from now.

So the answer then is to air-gap these systems: disconnect them from the Internet, the corporate LAN, and every other network that there is no justifiable business reason for them to be connected to. We can't trust the software or the protocols so we must configure these systems the way they were designed to be in the 1970's: on an isolated network protected by physical security. If a Russian or Chinese hacker can't attack a target over the Internet then the target likely will not be attacked simply owing to the increased cost and potential consequences of attempting to infiltrate a facility.

And to all the IT personnel and business managers who think this is an issue that doesn't affect them I would recommend taking a few minutes to search for your own devices on the Internet and search for known vulnerabilities in the devices you have. You might be surprised what you find if you only look, after all the attackers already are.