Whiteboard Wednesday:

Data Mining the Undiscovered Country

9月 27, 2017

In this week’s Whiteboard Wednesday, Bob Rudis, chief data scientist at Rapid7, revisits the presentation he gave at Rapid7’s 2017 UNITED conference. He digs deeper into how the data from Project Sonar and Heisenberg Cloud are being put to use by the Rapid7 Labs team, as well as the upcoming launch of a new study on headless browser HTTP scans.

To hear more from Bob on what the team is working on, check out his blog post here.


Video Transcript

Welcome to this week's Whiteboard Wednesday. My name is Bob Rudis, and I'm Rapid7's Chief Data Scientist. Today, we're bringing a bit of Rapid7's recent UNITED Conference to those of you who either couldn't make it or weren't able to catch the talk I co-presented with Derek Abdine, director of Rapid7 Labs.

Show more Show less

Now, first up is some news about Project Sonar. Back in 2014, Project Sonar was cranking out around 100 different studies each month. Now, we use the word study to describe our internet scans because each scan is crafted with a purpose to examine some protocol or service, and nothing we do is arbitrary.

Today, we're just about to hit 200 studies per month and this is only possible due to a complete revamp of the way we orchestrate, process, and disseminate scans and scan data. These improvements enable us to react more quickly to internet scale events, like WannaCry, and also help us to track changes to the make-up of the internet on a more granular scale, that all the data from these scans makes its way to scans.io, so researchers like you can help us in our mission to make the internet a safer place for everyone.

We're about to launch a new kind of study, Headless Browser HTTP Scans. Unlike our traditional IPD4 based index grabs, headless browser studies grab all the content from a set of target URLs. JavaScript, images, XML, HTTP request data, everything. We're still working out how and where we'll be able to share this data, but we're hoping to shed some light on the framework, usage, cross-site shared dependencies, and development trends, along with the exposure that comes along for the ride with all of that.

 Next is a quick refresher on how we use Project Sonar data to monitor and improve your security posture. Nexpose and Metasploit Pro customers have direct access to Sonar data in product, but anyone can use this data to see if you're exposing dangerous services, like SMB, to opportunistic attackers. Rather than force you to screenshot command line calls to curl and grep, we've provided a full example in companion blog, Data Mining: The Undiscovered Country, that you can find on the Rapid7 blog. Just search for it, Google it, you'll find it. It's right up there. With just two commands, one to download a Sonar study and the other to search for IP addresses in your perimeter, you can get an independent view of whether you're exposing services you thought you weren't.

This is important since we have daily evidence through our other research platform, Heisenberg Cloud, that opportunistic attackers abound and are ready to take advantage of any errant exposure in your perimeter. In fact, we've seen a near exponential increase in the number of SMB probes since the WannaCry disaster earlier this year and scans for other Microsoft protocols are also on the rise. It will likely take you less than 15 minutes to check for your own exposure after each new Sonar study is published. Plenty of time to kick off a search, grab some coffee and comeback to a hopefully empty result. Now finally, in a world where cybersecurity folk constantly feel like we're battling windmills, we have some good news.

Earlier this year we used Project Sonar to help identify exposures in radio station broadcast equipment. You may remember a set of incidents back in the February timeframe where protestors took over airways of a few radio stations on the west coast to blast anti Trump music. They did this by taking advantage of a weakness in devices that use the network or the internet to carry broadcast audio from studios to transmitters. Now, our threat lead, Rebekah Brown, asked the labs team if we could get an idea of how many other radio stations might be vulnerable. We found a few thousand with similar exposure. Rather than just try to get the word out via blogs or news stories, we worked directly with the National Association of Broadcasters and drafted up an advisory that was sent to all NAB member stations, which is virtually all of the stations in the US.

Within 24 hours, we had a 50% reduction in exposure. Any veteran infosec pro knows how incredible that figure is. We've been regularly monitoring the exposure profile and are quite thrilled to say that there are just around 15% of vulnerable devices left out there. It's nice to know the defenders win one every now and again. Don't hesitate to ask us anything regarding Sonar, Heisenberg, or other aspects of our internet scale research. You can reach us at anytime at research@rapid7.com. That's it for this week's Whiteboard Wednesday. See you next week. 

Rapid7 Quarterly Threat Report: 2017 Q2

Want to learn more about the threat landscape in the second quarter of 2017? Download the report, hear from the researchers themselves, and more.

Learn more

National Exposure Index

Is the internet broken? You bet. Get to know overall threat exposure on the internet in this report.

Download Report