Last updated at Mon, 18 Sep 2017 16:27:58 GMT

On August 31, 2017, NoMotion’s “SharknAT&To” research started making the rounds on Twitter.

After reading the findings, and noting that some of the characteristics seemed similar to trends we’ve seen in the past, we were eager to gauge the exposure of these vulnerabilities on the public internet. Vulnerabilities such as default passwords or command injection, which are usually trivial to exploit, in combination with a sizable target pool of well-connected, generally unmonitored internet-connected devices, such as DSL/cable routers, can have a significant impact on the general health of the internet, particularly in the age of DDoS and malware for hire, among other things. For example, starting around this time last year and continuing until today, the internet has been dealing with the Mirai malware that exploits default passwords as part of its effort to replicate itself. The SharknAT&To vulnerabilities seemed so similar, we had to get a better idea of what we might be facing.

What we found surprised us: the issues are actually not as universal as initially surmised. Indeed, we found that clusters of each of the vulnerabilities are found almost entirely in their own, distinct regional pockets (namely, Texas, California, and Connecticut). We also observed that these issues may not be limited to just one ISP deploying a particular model of Internet router but perhaps a variety of different devices that is complicated by a history of companies, products, and services being bought, sold, OEM’d and customized.

For more information about these SharknAT&To vulnerabilities and Rapid7’s efforts to understand the exposure these vulnerabilities represent, please read on.

Five Vulnerabilities Disclosed

NoMotion identified five vulnerabilities that, at the time, seemed limited to Arris modems being deployed as part of AT&T U-Verse installations:

  1. SSH exposed to the Internet; superuser account with hardcoded username/password. (22/TCP)
  2. Default credentials “caserver” in https server NVG599 (49955/TCP)
  3. Command injection “caserver” in https server NVG599 (49955/TCP)
  4. Information disclosure/hardcoded credentials (61001/TCP)
  5. Firewall bypass no authentication (49152/TCP)

Successful exploitation of even just one of these vulnerabilities would result in a near complete compromise of the device in question and would pose a grave risk to the computers, mobile devices, and IoT gadgets on the other side. If exploited in combination, the victim’s device would be practically doomed to persistent, near-undetectable compromise.

Scanning to Gauge Risk

NoMotion did an excellent job of using existing Censys and Shodan sources to gauge exposure; however, they also pointed out that some of the at-risk services on these devices are not regularly audited by scanning projects like this. In an effort to assist, Rapid7 Labs fired off several Sonar studies shortly after learning of the findings in order to get current information for all affected services, within reason.

As such, we queued fresh analysis of:

  • SSH on port 22/TCP to cover vulnerability 1
  • HTTPS on 49955/TCP and 61001/TCP, covering vulnerabilities 2-4
  • A custom protocol study on port 49152/TCP for vulnerability 5

Findings

Vulnerability 1: SSH Exposure

Not having a known vulnerable Arris device at our disposal, we had to take a bit of an educated guess as to how to identify affected devices. In NoMotion’s blog post, they cite Censys as showing 14,894 vulnerable endpoints. A search through Sonar’s SSH data from early August showed just over 7,000 hosts exposing SSH on 22/TCP with “ARRIS” in the SSH banner, suggesting that these may be made by Arris, one of the vendors involved in this issue. There are several caveats that could explain the difference in number, including the fact that Arris makes several other devices, which are unaffected by these issues, and that there is no guarantee that affected and/or vulnerable devices will necessarily mention Arris in their SSH protocol. A follow-up study today showed similar results with just over 8,000. It is assumed that the difference in Rapid7’s numbers as compared to NoMotion’s is caused by the fact that Sonar maintains a blacklist of IP addresses that we’ve been asked to not study, as well as normal changes to the landscape of the public Internet.

A preliminary check of our Project Heisenberg honeypots showed no noticeable change in the patterns we observe related to the volume and variety of SSH brute force and default account attempts prior to this research. However, the day after NoMotion's research was published, our honeypots started to see exploitation attempts using the default credentials published by NoMotion.

September 13, 2017 UPDATE on SSH exposure findings

The researchers from NoMotion reached out to Rapid7 Labs after the initial publication of this blog and shared how they estimated the number of exposed, vulnerable SSH endpoints. They did so by searching for SSH endpoints owned by AT&T U-Verse that were running a particular version of dropbear. Repeating our some of our original research with this new information, we found nearly 22,000 seemingly vulnerable endpoints in that same study from early August concentrated in Texas and California.

Armed with this new knowledge, we re-analyzed SSH studies from late August and early September and discovered that seemingly none of the endpoints that appeared vulnerable in early August were still advertising the vulnerable banner, indicating that something changed with regards to SSH on AT&T U-Verse modems that caused this version to disappear entirely. Sure enough, a higher level search for just AT&T U-Verse endpoints shows that there were nearly 40,000 AT&T U-Verse SSH endpoints in early August and just over 10,000 in late August and early September, with the previously seen California and Texas concentrations drying up. What changed here is unknown.

Vulnerabilities 2 and 3: Port 49955/TCP Service Exposure

US law understandably prohibits research that performs any exploitation or intrusive activity, which rules out specifically testing the validity of the default credentials, or attempting to exploit the command injection vulnerability. Combined with no affected hardware being readily available to us at the time of this writing, we had to get creative to estimate the number of exposed and potentially affected Arris devices.

As mentioned in NoMotion’s blog, they observed several situations in which the HTTP(S) server listening on 49955/TCP would return various messages implying a lack of authorization, depending on how the request was made. Our first slice through the Sonar data from August 31, 2017 showed ~3.4 million 49955/TCP endpoints open, though only approximately 284,000 of those appear to be HTTPS. Further summarization showed that better than 99% of these responses were identical HTTP 403 forbidden messages, giving us high confidence that these were all the same types of devices and were all likely affected. In some HTTP research situations we are able to examine the HTTP headers in the response for clues that might indicate a particular vendor, product or version that would help narrow our search, however the HTTP server in question here returns no headers at all.

Furthermore, by examining the organization and locality information associated with the IPs in question, we start to see a pattern that this is isolated almost entirely to AT&T-related infrastructure in the Southern United States, with Texas cities dominating the top results:

The ~53k likely affected devices that we failed to identify a city and state for all report the same latitude and longitude, smack in the middle of Cheney Reservoir in Kansas. This is an anomaly introduced by MaxMind, our source of Geo-IP information, and is the default location used when an IP cannot be located any more precisely than being in the United States.

As further proof that we were on the right track, NoMotion has two locations, both in Texas. It’s likely that these Arris devices were first encountered in day-to-day work and life by NoMotion employees, and not scrounged off of eBay for research purposes. We’ve certainly happened upon interesting security research this way at Rapid7—it’s our nature as security researchers to poke at the devices around us.

Because this HTTP service is wrapped in SSL, Sonar also records information about the SSL session. A quick look at the same devices identified above shows another clear pattern -- that most have the same, default, self-signed SSL certificate:

This presents another vulnerability. Because the vast majority of these devices have the same certificate, they will also have the same private key. This means that anyone with access to the private key from one of these vulnerable devices is poised to be able to decrypt or manipulate traffic for other affected devices, should a sufficiently-skilled attacker position themselves in an advantageous manner, network-wise. Because some of the SharknAT&To vulnerabilities disclosed by NoMotion allow filesystem access, it is assumed that access to the private key, even if password protected, is fairly simple. To add insult to injury, because these same vulnerable services are the very services an ISP would use to manage and update or patch affected systems against vulnerabilities like these, should an attacker compromise them in advance, all bets are off for patching these devices using all but a physical replacement.

It is also very curious that outside of the top SSL certificate subject and fingerprint, there is still a clear pattern in the certificates: there is a common name with a long integer after it, which looks plausibly like a serial number. Perhaps at some point in their history, these devices used a different scheme for SSL certificate generation, and inadvertently included the serial number. Some simple testing with a supposedly unaffected device showed that this number didn’t necessarily match the serial number.

Examining Project Heisenberg’s logs for any traffic appearing on 49955/TCP shows only a minimal amount of background noise, and no obvious widespread exploitation yet in 2017.

Vulnerability 4: Port 61001/TCP Exposure

Much like with vulnerabilities 2 and 3 on port 49955/TCP, Sonar is a bit hamstrung when it comes to its ability to test for the presence of this vulnerability on the public internet.

Following the same steps as we did with 49955/TCP, we observed ~5.8 million IPs on the public IPv4 Internet with port 61001/TCP open. A second pass of filtering showed that nearly half of these were HTTPS. Using the same payload analysis technique as before didn’t pay dividends this time, because while the responses are all very similar -- large clusters of HTTP 404, 403, and other default looking HTTP response -- there is no clear outlier. The top response from ~874,000 endpoints looks similar to what we observed on 49955/TCP -- lots of Texas with some Cali slipping in:

The vast majority of the remainder appear to be 2Wire DSL routers that are also used by AT&T U-Verse. The twist here is that Arris acquired 2Wire several years ago. Whether or not these 2Wire devices are affected by any of these issues is currently unknown:

As shown above, there is still a significant presence in the Southern United States, but there is also a sizeable Western presence now, which really highlights the supply chain problem that NoMotion mentioned in their research. While the 49955/TCP vulnerability appears to be isolated to just one region of the United States, the 61001/TCP issue has a broader reach, further implying that this extends beyond just the Arris models named by NoMotion, but not necessarily beyond AT&T.

Repeating the same investigation into the SSL certificates on port 61001/TCP shows that there are likely some patterns here, including the same exact Arris certificate showing up again, this time with over 45,000 endpoints, and Motorola making an appearance with 3/4 of a million:

Examining Project Heisenberg’s logs for any traffic appearing on 61001/TCP shows there is only a minimal amount of background noise and no obvious widespread exploitation yet in 2017.

Vulnerability 5: Port 49152/TCP Exposure

The service listening on 49152/TCP appears to be used as a kind of a source-routing, application layer to MAC layer TCP proxy. By specifying a magic string, the correct opcode, a valid MAC and a port, the “wproxy” service will forward on any remaining data received during a connection to port 49152/TCP from (generally) the WAN to a host on the LAN with that specified MAC to the specified. Why exactly this needs to be exposed to the outside world with no restrictions whatsoever is unknown, but perhaps the organizations in question deploy this for debugging and maintenance purposes and failed to properly secure it.

In order to gauge exposure of this issue, we developed a Sonar study that sends to the wproxy service a syntactically valid payload that elicits an error response. More specifically, the study sends a request with a valid magic string, an invalid opcode, an invalid MAC and an invalid port, which in turn generally causes the remote endpoint to return an error that allows us to positively identify the wproxy service. Because this vulnerability is inherent in the service itself due to a lack of any authentication or authorization, any endpoint exposing this service is at risk.

As with the other at risk services described so far, our first step was to determine how many public IPv4 endpoints seemed to have the affected port open, 49152/TCP. A quick zmap scan showed nearly 8 million hosts with this port open. With our limited knowledge of the protocol service, we looked for any wproxy-like responses, which quickly whittled down the list to approximately 42,000 IPv4 hosts exposing the wproxy service.

We had hoped that a quick application of geo-IP and we’d be done, but it wasn’t quite that simple. Using the same techniques as with other services, we grouped by common locations until something caught our eye, and immediately we knew something was up. Up until this point, all of this had landed squarely in AT&T land, clustering around Texas and California, but several different lenses into the 49152/TCP data pointed to one region—Connecticut:

Sure, there are a few AT&T mentions and even 5 oddly belonging to Arris in Georgia, but otherwise this particular service seemed off. Why all Texas/California AT&T previously, but now Frontier in Connecticut? Guesses of bad geo-IP data wouldn’t be too far off, but in reality, Frontier acquired all of AT&T’s broadband business in Connecticut 3 years ago.

This means that AT&T broadband customers who were at risk of having their internal networks swiss-cheesed by determined attackers with a penchant for packets for at least 3 years are now actually Frontier customers using AT&T hardware, almost certainly further complicating the supply chain problem and definitely putting customers at risk because of a service that should have never seen the public internet in the first place.

Examining Project Heisenberg’s logs for any traffic appearing on 49152/TCP and there is largely just suspected background noise in 2017, albeit a little higher than port 49955/TCP and 61001/TCP. There are a few slight spikes back in February 2017, perhaps indicating some early scouting, but it is just as likely to have been background noise or probes for entirely different issues. Some high level investigation shows a deluge of blindly lobbed HTTP exploits at this port.

Conclusions

The issues disclosed by NoMotion are certainly attention-grabbing, since the initial analysis implies that AT&T U-Verse, a national internet service provider with millions of customers, is powered by dangerously vulnerable home routers. However, our analysis of what’s actually matching the described SharknAT&To indicators seems to point to a more localized phenomenon; Texas and other Southern areas are primarily indicated, with flare ups in California, Chicago, and Connecticut, with significantly lower populations in other regions of the U.S.

These results seem to imply which vendor is in the best position to fix these bugs, but the supply chain problems detailed above add a level of complication that will inevitably leave some customers at risk unnecessarily.

Armed with these Sonar results, we can say with confidence that these vulnerabilities are almost wholly contained in the AT&T U-Verse and associated networks, and not part of the wider Arris ecosystem of hardware. This, in turn, implies that the software was produced or implemented by the ISP, and not natively shipped by the hardware manufacturer. This knowledge will hopefully speed up remediation.

Interested in further collaboration on this? Have additional information? Questions? Comments? Leave them here or reach out to research@rapid7.com!