The Internal Revenue Service is in the news again as a result of last year's hack as they've drastically increased their estimate of the number of people affected. This highlights a recurring problem that we see with breaches: companies often simply aren't aware that they've been breached and, even once they figure it out, aren't capable of quickly and accurately determining what was taken and how to fix the underlying problems.

Missing the Point

In the cases of the IRS this saga started in March of 2015 when security journalist Brian Krebs posted a warning about flaws in the IRS' identity verification processes that would allow identity thieves to illicitly collect taxpayers' W2 form information that could then be used to file fraudulent returns in their name. Just under 2 months later the IRS admitted that these flaws led to about 100,000 cases where tax transcripts had been compromised between February and May of that year. This begs the questions: did the IRS know about these flaws before Krebs reported on them, if they did know then why weren't they fixed sooner, and why weren't they fixed immediately after Krebs' report?

It wouldn't be surprising if the IRS only found out about their security issues as a result of Krebs' reporting, after all it is common for organizations to learn that they have been breached only when contacted by a third party.

To pick one high profile example from the private sector, Target only found out about their massive credit card breach when contacted by the Department of Justice and United States Secret Service as detailed in their CFO's testimony before congress. The way these payment card breaches are typically discovered is that the banks themselves purchase stolen card numbers from underground markets and check to see if there is a single location where the stolen cards were all used. In Target's case millions of card numbers went up for sale on these underground markets and were quickly tracked back to Target as detailed by Krebs (who originally broke the Target story) and his contacts within the banks.

While stolen card numbers are typically sold on easily accessible marketplaces and banks have a financial interest in monitoring these marketplaces for stolen card numbers in order to prevent fraud the same is not true in many other industries. If social security numbers, medical records, account credentials, intellectual property, or other valuable information is leaked it would be much less likely to be found and reported by a third party and an affected organization may remain ignorant of an ongoing attack for a long period of time, racking up damages all the while.

Damage Assessment

Finding out that a breach has taken place it only the first step however. In the aftermath an organization must still determine how to clean up the mess, including determining what the attackers took. This is especially important when personal information is involved as quick disclosure it often required by law. Sadly organizations also tend to fall short in this area as well.

This is the second time the IRS has revised the number of accounts believed to have been stolen, first going up from the original 114,000 estimate to 334,000 in August 2015 and now going up to over 700,000 nearly a year after Krebs originally reported the issue to the public. This does not inspire confidence in the IRS' ability to forensically examine their own systems.

Once again the IRS is not alone, many organizations are forced to revise their breach numbers in the weeks and months after initially disclosing an incident (although a year later is a bit longer than is typical). To pick on Target again, their initial reports on December 19th indicated 40 million stolen payment cards they added another 70 million user accounts to that total 3 weeks later on January 10th.

When breaches involve more than just payment card numbers this inability to accurately determine impact can have more serious consequences. As an example of this we can look to a breach at Symantec that was announced in 2012 after source code from various Symantec products was posted on the Internet. The code had actually been stolen in 2006 in a breach that Symantec was aware of but, as Symantec stated to Wired, their investigation into what was taken was inconclusive and it seems they basically ignored the breach from that point until they started receiving ransom demands 6 years later.

The risk here is that an attacker can review the source code looking for vulnerabilities that can be exploited to compromise users of the affected Symantec products. In the aftermath of the disclosure Symantec played down the possibility of harm from any such vulnerabilities due to the age of the code and changes that had happened over the previous 6 years. This statement was then revised after source code for a version of PCAnywhere that was still in use was released.

The issue with Symantec's "don't worry about it approach" is that the real risk to the public isn't what could happen after the source code was disclosed 6 years after the breach, it's what may have happened during the 6 years between the hack and when the code resurfaced: we simply don't know who had the code and what they did with it. For all we know hackers may have been reviewing the code, developing, and using zero day attacks based on what they found. The same could be true of any stolen source code floating around today that we don't know about yet.

As more of our lives comes to depend on the proper functioning of computer programs, from the code that operates our vehicles to the medical devices implanted in our bodies, the possibility of deadly attacks resulting from these sorts of breaches becomes real. Determining whether source code has been compromised could be a matter of life or death. Fortunately most companies won't find themselves in such a critical situation but the loss of intellectual property could still have serious financial consequences.

Needles in Haystacks

The obvious reason why breaches are not being detected and why, even when they are, the effects are not immediately known is that many companies are simply not focused on the ability to detect and respond to breaches. Much of our efforts at information security over the past few decades have been focused on keeping attackers out, not what happens once they get in.

The raw data to detect a breach has always existed in the form of system logs. They tell us everything we need to know: who is logging in, when, from where, and what they're accessing. The problem has always been sorting through the mountains of logs that a network generates and making some sort of sense out of it.

Fortunately we now have many tools to help correlate the alerts that our security devices are generating with the logs on the network (Security Information and Event Management) and other intelligence information, such as the typical behavior of system users (User Behavior Analysis). This feeds into another security trap though: over-reliance on technology.

Target once again provides us with a case study in this regard: the company had paid $1.6 million to install the FireEye threat detection system on their network and the technology performed as designed, generating alerts when it detected the malware that eventually led to the infamous breach before any cardholder data had been send out from the network. Somehow this early warning was overlooked by Target's network security personnel and the rest is history.

Getting this detection ability right is hard work. Personnel will have to be available 24x7 to sort through many false positive alerts without becoming so jaded that they miss a real indicator of a breach. This requires both deep technical skills, knowledge of the environment being protected, and a certain type of analytical mindset. Putting all of this together can be difficult given the current shortage of skilled personnel in the information security space and many companies instead rely on managed security service providers as a more cost effective method of maintaining this capability.

Playing Detective

Once a potential breach has been found the focus shifts to containing the breach, determining the impact, and making the proper notifications. This requires a plan and even more skilled staff with knowledge of network forensics techniques. Being able to quickly isolate a breach and determine what actions the attackers took are key to being able to accurately report the impact of the breach.

The importance of the speed and accuracy of the response effort can't be understated as it will impact both the victim of the breach and their customers. Laws typically require the speedy disclosure of breaches affecting personal information and, even where they do not, leaks or bragging by hackers may expose a breach to the public who will want quick answers. Delaying a response leaves customers exposed, or at the very least wondering if they are exposed, while repeatedly restating the scale of the breach as investigations turn up new information keeps the breach in the public consciousness with the associated negative impact on the brand increasing as a result.

Besides the direct impact of a breach it is also important to swiftly identify the vulnerabilities that led to the breach, other associated vulnerabilities that could lead to a new breach, and how they can be remediated. The IRS provides us another example of failure in this regard: Krebs pointed out in August of 2015 that the functionality on the IRS website that allows taxpayers who had already been victims of identity theft to retrieve their PIN numbers used the same "security questions" that fraudsters were already abusing to retrieve taxpayers' W2 forms. In March 2016 this flawed functionality was still present and Krebs reported on new cases of fraudulent tax returns involving stolen PIN numbers. It wasn't until March 7th, 2016, nearly a year after Krebs' initial story about fraudulent returns and over a years since the fraud began, that this functionality was disabled by the IRS. The IRS' inability to quickly identify and address the vulnerabilities in their system has clearly impacted both them and their "customers", the taxpayers.

When it comes to planning there are numerous considerations that have to be taken into account: What systems are non-critical and can be isolated pending investigation? Under what conditions can even critical systems be taken offline? Who is responsible for leading the breach investigation and cleaning? Who will be on call to assist in the investigation and cleanup? The time to develop and test this plan is well before a breach happens yet many companies simply do not have any sort of documented incident response plan or forensics personnel immediately available.

Hopefully a breach should be a rare event for any given company so incident response and forensics staffing is an area where companies often utilize retainers with third-party specialists who can maintain the skills and experience necessary for a quick response.

Hope for the Best, Prepare for the Worst

We have to accept that every network is a potential target and that, despite the security measures we have taken, there are enough vulnerabilities floating around on our networks that they will eventually be breached, if they're not already. We also need to recognize that, despite the promises of some security tool vendors, technology alone is not enough; we need skilled personnel to make that technology effective.

Considering a network to be secure without proper detection and response plans, tools, and personnel would be like having a bank with a vault but no alarms, guards, or cameras. Trouble will almost certainly come and when it does there won't be much anybody will be able to do about it.