Last updated at Thu, 08 Aug 2019 17:33:30 GMT

This research was produced with immense help from Vasudha Shivamoggi, Kwan Lin, Bob Rudis, and Jon Hart. Their work to gather, analyze, and interpret the data presented here is deeply appreciated.

In the spirit of HaXmas, we at Rapid7 Labs are digging up some of the research nuggets that got buried during the rest of the year. Thus we give you a haunting story to warm you by the fire this winter. Join us as we tell the tale of the disappearing port 81 botnet this HaXmas—otherwise known as “When GoAhead Was Left Behind.”

For a tumultuous 11 days in April (2017-04-16 through 2017-04-26) Rapid7 Labs observed a botnet with roughly 18,000 distinct IP addresses marauding across the public internet in search of prized quarry laying in wait on TCP port 81. The emergence of this new cadre of corrupted computing devices was first documented by the fine sleuths over at the Network Security Research Lab 360. They noted that the primary weapon of choice for these attackers were HTTP GET requests targeting login.cgi and the botnet appeared to have its eyes fixed on finding vulnerable GoAhead servers, which are embedded web servers present in over 1,000 different types of Internet of Things (IoT) devices.

These fiends disappeared almost as quickly as they arrived: botnet traffic slowed to a crawl after 2017-04-26. While the danger seems to have passed, we at Rapid7 Labs urge vigilance lest ye be DDoS'd away in a resurgence of activity.

Come with us as we don our data science deerstalkers and explore the case of the disappearing port 81 botnet.

Timeline Review

I don't know what of our tools you, our readers, use on a regular basis, but one of the things, I like to look at first when I login to isc.sans.edu is the Top 10 Ports by Unique Sources chart. This suggests coordinated (think botnets) scanning. So, I was really shocked to see port 81 had jumped up to 2nd position just behind all the Mirai-ish port 23 scanning. Take a look at the port 81 chart. If any of our readers have any insight into what is going on here since 16 Apr, plase [sic] let us know.

  • On 2017-04-24, the 360 Network Security Research Lab posted a new threat report about the botnet on their blog which provides a very detailed analysis of the traffic to port 81 that they had seen up to that point.
  • On 2017-04-26, the heightened scanning activity dropped as the botnet appeared to go quiet.

Digging In

On 2017-04-26, having seen several write-ups about a botnet spreading through port 81, we decided to use the data we have collected with Project Heisenberg and Project Sonar to analyze the botnet’s activity. Project Heisenberg detected the attempted connections on port 81 and queries for login.cgi. Project Sonar subsequently performed an HTTP study of TCP port 81 on the public IPv4 Internet. We examined data from before the dramatic increase in activity, as well as data from after the increase in activity.

Storming A Very Particular Port

There's no particular standard around port 81 usage, though it does include a mixed range of web servers, connected cameras, and various IoT devices.
Figure 1 below shows that when aggregating unique IP connection attempts per hour on port 81 to our Heisenberg honeypots, we can see a distinct surge in activity between 2017-04-16 and 2017-04-26.

Figure 1

Dissecting The GoAhead Botnet

The 360 Network Security Research Lab report stated that members of the botnet attempted to spread by initiating HTTP connections to IP addresses on port 81, and then requesting login.cgi. For the month of April 2017, and almost exclusively between 2017-04-16 and 2017-04-26, Project Heisenberg recorded 18,000 distinct IP addresses making requests for login.cgi.
Breaking those connections down by hour, we see some periodicity in the number of unique IPs:

Figure 2

Figure 2 shows that the population of IP addresses that scans for login.cgi is almost entirely inactive during the rest of the month of April.

Figure 3 below shows their scanning behavior for the entire month of April, which highlights the increase in activity between 2017-04-16 and 2017-04-26:

Figure 3

On either side of those dates, we saw at most six out of those 18,000 IPs performing any sort of scanning activity in any given hour in the month.

We noted that the botnets are fairly indiscriminate in their scanning. Project Heisenberg's honeypots are scattered across the internet in a variety of places, and when we separated our analysis by those places, Figure 4 shows that we observed a fairly consistent pattern.

Figure 4

HTTP Requests with a Relative Path

The botnet requests took advantage of poor design in GoAhead servers that additionally complicated efforts to study the botnet within legal constraints.
Astute readers may have wondered why we continually make mention of HTTP GET requests for the relative path login.cgi when a properly formed request would be for the absolute path of /login.cgi as required by the the HTTP RFC. It is indeed curious why anything would make requests for login.cgi, since most every reasonably well-behaved HTTP server out there is going to return an error, likely an HTTP 400 indicating a bad request.

As it happens, GoAhead isn't exactly well-behaved. In addition to responding positively to requests for resources with relative paths, GoAhead's configuration is (was?) also such that requests for resources with relative paths bypassed any required authentication AND disclosed the source code for the requested resource, conveniently including the username and password (https://blogs.securiteam.com/index.php/archives/3043).

Many HTTP configurations that require authentication do so for large swaths of resources. In the case of many of these IoT devices, they are likely requiring authentication for anything from / "down". Furthermore, the majority of these configurations should prompt for authentication before serving any resource, valid or otherwise.

A vulnerability like this presents an interesting challenge from a security research perspective.

If you were to attempt to assess the potential exposure of assets on the public Internet to this particular rash of vulnerabilities, one reasonably accurate way is to perform the very request that the botnet is performing—that is, making an HTTP GET request to login.cgi. If the source code is returned, the endpoint is vulnerable; however, performing a test like this is prohibited according to various regulations. On the other hand, an assessment that made HTTP GET requests to /login.cgi would suffer from the fact that a proper HTTP configuration should require authentication and return an HTTP 401 error regardless of whether or not the resource exists. Contrast this with what would happen when valid credentials are provided for a resource that doesn't exist: it would (hopefully) return a 404.

Scanning Locations

By backtracing the IP addresses of the machines pinging our honeypots, we can explore where the requests distributing the payload are coming from, along with information about the networks they live on. The set of IP addresses that we examined was not reduced by any exclusion lists, such as for known research devices.

The first component—geographic data—shows that the majority of requests are sourced from China, followed by the United States, with a smaller number coming from France and the United Kingdom.

Figure 5

Digging into those IP addresses' metadata, we can look at which networks (broadly construed) the requests came from. Doing so revealed that the IP addresses largely belonged to generic consumer internet providers. In combination with information on where GoAhead cameras are sold and prominent, this strongly suggested that the payloads were being distributed by already-infected devices.

Looking For Clues Across The Globe

Using Project Sonar data, we were able to identify a non-trivial population of devices that looked like potential targets for this botnet. Several days after the botnet’s scanning activity stopped, there were tens of thousands of devices from which we did not see scanning activity, but that fit the profile for targets (for which the botnet was scanning).

Of the 18,000 IPs that Project Heisenberg saw scanning for login.cgi, 3,900 of them were picked up in a subsequent Project Sonar scan; of those, 98.95% provided a 401 Unauthorized HTTP status code when they received a request for /login.cgi, suggesting that most devices were properly configured.

So What Did Happen to the Port 81 Botnet?

As mentioned in the beginning, we logged a drop-off in scanning activity on port 81 after 2017-04-26. However, the decline in activity did not persist; our monitoring activities have revealed periodic increases in activity on port 81.

Figure 6

While few of the spurts of activity have matched the initial onslaught in terms of sustained duration or scale, there have been exceptions. On 2017-05-12, there was a dramatic but momentary spike in activity that was more than twice as large as the highest point in the 2017-04-16 through 2017-04-26 range; further investigation revealed the spike originated from a single source.

A similar but smaller event occurred on 2017-10-27, due mainly to an uptick in connections from a single source. By contrast, a spike observed on 2017-10-23 was associated with many different sources, each making many connections per hour. This broad source base suggests possible ramping-up of the botnet—however, like before, the activity wasn’t sustained. Further observation indicated that there were still occasional but non-cyclical spikes in activity on port 81 that exceeded the original uptick in scale.

As mentioned in the beginning, we logged a drop-off in scanning activity on port 81 after 2017-04-26. The devices we had seen participating in botnet scanning activity went silent, but we saw a population of devices that could still have been infected. So why did the botnet stop scanning?

A few theories in question form:

  • Did the attackers believe they had identified a sufficient number of devices on fairly-fixed addresses and leave those devices dormant only to be used at a more precise or opportune time?
  • Could the botnet controllers — possibly — have put their initial plans in motion too soon? Were they unprepared to take advantage of their newly acquired drone arsenal?
  • Perhaps the botnet operator(s) read the reports about researchers tracking the scanning activity and decided to shut down their activities?
  • Could a rival botnet have taken over?
  • Did ISPs actively work to mitigate the impacts of the botnet?

Whatever the reason, the curious case of the disappearing port 81 botnet fell into the bucket of mysteries hidden away until HaXmas time, when we at Rapid7 Labs gladly dusted off this gift from our telemetry trove to accompany your figgy pudding. Botnet research is a gift that keeps on giving, so keep an eye on this space (particularly our Heisenberg and Sonar blog tags) to see what treasures we mine in 2018!