Incident Response is about Where, When, and How

Last updated at Mon, 28 Oct 2019 17:38:05 GMT

"If and when" is old and busted.

"Where, when, and how" are the new hotness.

Incidents happen. There will always be a Patient Zero. "Where the incident happened, when you detect the incident, and how you responded" is what I believe matters.

I think the general public will appreciate measured response under attack to us fostering belief in 'perfect defense'. With this in mind, I want discuss a few thoughts prompted by eBay's response to this compromise.

Scoping is hard

Incident Handlers leap into action as events get flagged and high-fidelity notifications bring focus to an incident. On some occasions we find ourselves facing a confirmed breach.

No one wants this phone call, but eventually we all make or take that call. What happens next, and the decisions that follow, are what matter.

The team doesn't yet have confirmation on what exactly was taken. This starts out as a question of what systems are involved, what data is included, what users or business units are impacted. The most important thing at this stage is drawing a box around the systems that were compromised.

Handlers often work to approximate volume and type of 'accessed'—which in eBay's case may be upwards of 145 million users. If confirmed, eBay becomes the second largest breach as per http://datalossdb.org.

While scoping efforts are in flight, if properly staffed and instrumented, there will be a second effort working in tandem. The second team is hunting for Patient Zero, the root cause of the compromise, the initial point of entry.

Make no mistake, this is hard. Scoping is hard. Harder than PCI, scoping in incident response scenarios is wicked hard. Incident Handlers make the best decisions they can, with the data available *at that time*. Data gets updated. Assumptions get undermined. And then THE SCOPE CHANGES.

Note:

There are regulatory forces pushing for earlier breach notifications. I'm betting that regulators have never worked as incident handlers. In my experience verification of the breach, scope of the attack, and confirmation of loss should come long before determining what and how things get announced.

Notifications should be accurate, and done in a thorough manner. The breach is enough to shake consumer confidence. Being unsure about what was taken, when the events took place, which systems/environments were affected, and who was directly impacted should no longer be in question at the time of announcement.

Crisis Communications ain't easy.

Unlike this blog post (which was edited and discussed by a few people), a breach notification needs departmental sign off. By IT. By the business lead. By PR and Marketing... AND THEN LEGAL.

Consensus is hard.

eBay faced an unfortunate situation where a release was redacted, breach size was announced and redacted, and then left unconfirmed. And that's okay, because the data was only accessed... not copied by the attacker. (Can someone explain that one to me?)

Crisis Communications shouldn't be easy.

As solutions and service providers, we all seek to eliminate friction for our customers, partners, and guests. One of my favorite User Experience (UX) designers always taunts me with the thought that 'uncertainty must be used with purpose.'

When emotions run high (as they do during a breach) Corporate Comms and PR teams seek to communicate the absolute minimum, stick to the facts, and minimize the perception of damage.

As champions advocating for the well-being of our customers and partners, we often face an uncomfortable discussion. A great example of this is something eBay has not (yet) done. In this case, I believe eBay would be wise to force a password reset for all users in the potential scope of this breach.

Yes, I know it won't be popular at first. Yes, I'm dead serious. It needs to be done.

In my experience, humans are very accommodating when you put their well-being ahead of their comfort. They are forced to invest in their relationship with you. In the long run, they will thank you for it.

Fraudsters gonna Fraudulate.

Yep, I'm coining words.

Someone, somewhere, posted a sample of what they claim is eBay's compromised database. If the sample is real, there is a lovely ray of sunshine in that the passwords appear very nicely hashed. More on this in a separate post.

eBay states this is not from their 'accessed' database. (Because the data was accessed, not compromised. Just so we're clear...ish.)

Major events like this breach will bring all kinds of fraudsters to the party. Fast, efficient, and well-resourced fraudsters will offer up fake samples to sell 'the database,' and others will follow suit. Others will wait for the legit database to be accessible, and will launch some solid phishing campaigns, equipped with extra data like the users' home address and birthday. The fraudsters that really get under my skin are the ones calling, posing to work for a credit card company, or collections agency.

Check in with your friends and family. You don't want to admit it, but you already have someone in mind that would fall for this. Call them.

Logging is hard.

Log All The Things. Read all the logs. #NotGonnaHappen

Most regulatory bodies like PCI have a lot to say about logging, and log review. The simple fact is logging is quite often poorly done, and has not been reviewed by the Incident Response team. Prior to PCI, most organizations did nothing about gathering and retaining logs. Think about how many Apache logs we still see today with one source IP—the internal interface of the firewall! (go read up on XFF)

Database logging—knowing exactly what was accessed—can mean a mountain of very expensive logging in terms of volume and I/O disk activity. During high tide seasons or system troubleshooting, log settings can be changed and forgotten.

Sometimes these logs are critical in a response scenario. I'm sure we will hear more on this in the near future.

Save your future self some pain, check in on key databases and verify you've got the right logs flowing, and that you can trace activity back to users and external sources.

User activity matters.

For those that know me, I'm not one to shamelessly plug products. That said, I want to highlight something. Unless you're Fortune 10 (or have a IT and InfoSec team operating near that general level of expertise and funding), you probably aren't keeping a close enough eye on your users.

We *do* have confirmation that this attack leveraged employee credentials. Rapid7, thanks to Metasploit being a part of our DNA, sees the world through predatory eyes. We understand attackers.

We know that credentials are the keys to the kingdom. If you find the notion of tracking where accounts are being logged in from, and detecting user events changing suddenly—you might want to check out UserInsight.

There will always be Patient Zero, or Infection Zero. How quickly your team becomes aware, and how elegantly you respond may ultimately determine how much credibility you gain or lose through the course of the event.