Project Sonar Study of LDAP on the Internet

Last updated at Wed, 16 Aug 2017 15:51:58 GMT

The topic of today's post is a Rapid7 Project Sonar study of publicly accessible LDAP services on the Internet. This research effort was started in July of this year and various portions of it continue today.  In light of the Shadowserver Foundations's recent announcement regarding the availability relevant reports we thought it would be a good time to make some of our results public. The study was originally intended to be an Internet scan for Connectionless LDAP (CLDAP) which was thought to only be used by Microsoft Active Directory. This protocol uses LDAP over UDP instead of the typical TCP. CLDAP was proposed in RFC 1798 in June of 1995 but its standard was marked as abandoned per RFC 3352 in March of 2003 due to lack of support and implementation. Microsoft uses CLDAP as part of the client's discovery mechanism via LDAP Ping (additional reference).

In the early phases of the study it was determined that the toolset could be used to survey LDAP over TCP and so the scope was expanded. In this document the term LDAP will used to interchangeably for both LDAP (TCP) and CLDAP (UDP).

The intent of the study was to attempt to answer multiple questions:

1. How prevalent is publicly accessible LDAP on the Internet? The term "publicly accessible" in this case means services that allow connections from and communications with our various scanners. In no case did we test with any form of authentication other than that of an anonymous LDAP bind.
2. Can we identify the remote service and, if so, determine if Microsoft Active Directory is the only product using CLDAP protocol?
3. Would CLDAP be an effective service in DRDoS attacks?

Study implementation

Like most other UDP services CLDAP will only respond to correctly formatted probes. In our study we performed service discovery using Zmap with a probe file containing a rootDSE request. This request, defined in RFC 4512 Section 5.1, is part of the LDAP protocol standard and is essentially a request for information about the server and its capabilities. The rootDSE request was chosen as the functionality should be present on all LDAP servers and, importantly, it does not require authentication by RFC definition. This reduces the legal risk of the study as no authentication mechanisms needed to be bypassed.

During the study it was found that most of the LDAP responses were larger than a single packet and were fragmented in order to traverse the Internet.  Fragment reassembly functionality is not part of the Zmap use case at this time so we followed the scan with another scan against open services using a custom tool that performs a more comprehensive capture of the response.

Since the protocol for communicating with Microsoft's LDAP on UDP is pretty much per RFC standard, it would make sense that any tooling that works with LDAP over UDP should also work with LDAP over TCP with minor changes. This turned out to be the case and the study was expanded to the following ports and services:

We've performed multiple Internet wide scans on these ports as part of the study. Where possible we negotiated TLS or STARTTLS.

The results published in this post are based on Internet wide IPv4 scans performed September 14th and 15th of this year.

Study results

Prevalence

The answer to the study's first question is that there are a significant number of publicly accessible LDAP servers on the Internet.

The percentage of LDAP servers is actually slightly higher than reported. Some of the service responses that did not decode as LDAP were visually identified as fragments of legitimate LDAP responses.

Service identification

A significant number of services returned information that allowed us to successfully fingerprint them. This fingerprinting capability was added to the Rapid7 Recog tool which is discussed later in this post. In our data set we found that most of the LDAP services were Microsoft Active Directory Controllers (ADC) or Lightweight Directory Services (LDS) (formerly Active Directory Application Mode)

Here is the breakdown of ADC and LDS services in the data set. The Samba ADC implementation has been included as well.

It turns out that it is possible to not only identify the software in most cases, but we can usually identify the host operating system as well. In the case of Microsoft ADC and LDS services, there is almost always enough information to determine the more specific operating system version such as Windows Server 2012 R2. Most of the ADC and LDS services in the data set are running supported versions of the OS. Some are ahead of the curve, some are behind. As a note, Microsoft ended all support for all versions of Windows Server 2003 as of July 14, 2015.

We can also identify other products, just in smaller numbers. The table below contains a few examples selected from a much longer list.

This allows us to answer the study's second question of the ability to identify services. We can successfully do this in many cases though there is a non-trivial number of generic LDAP response returned on 389/tcp. We've also determined that Microsoft ADC and LDS aren't the only services using CLDAP as we found six instances Novel LDAP Agent for eDirectory in the data set.

DRDoS utility

The specific type of DDoS we are interested in was described by Rapid7 Labs researcher Jon Hart in a recent Rapid7 Community blog post:

A distributed, reflected denial of service (DRDoS) attack is a specialized variant of the DDoS attack that typically exploits UDP amplification vulnerabilities.  These are often referred to as volumetric DDoS attacks, a more generic type of DDoS attack that specifically attempts to consume precious network resources.

A UDP amplification vulnerability occurs when a UDP service responds with more data or packets than the initial request that elicited the response(s). Combined with IP packet spoofing/forgery, attackers send a large number of spoofed UDP datagrams to UDP endpoints known to be susceptible to UDP amplification using a source address corresponding to the IP address of the ultimate target of the DoS.  In this sense, the forged packets cause the UDP service to "reflect" back at the DoS target.

Based on service responses to probes on 389/udp it would appear CLDAP does provide a significant amplification source. The UDP datagram probe that was used in the study was roughly 51 bytes.  Most of the valid LDAP responses without the headers ranged from 1,700 to 3,500 bytes. There were a few outliers in the 5,000 to 17,700 range. The chart below shows the response size distribution with the larger outliers removed.

The services returning a 3,000 bytes in response data are providing a 58x bandwidth amplification factor. We didn't track the number of packets returned but using almost any reasonable MTU value will result in a packet amplification factor of 2x or 3x. Based on this finding CLDAP is effective as a DRDoS amplification source. Its utility as such will primarily be limited by how prevalent public access to the service remains over time.

Study success

When the study completed we were able to answer the study's primary questions. LDAP is fairly prevalent on the public Internet.  We can identify the software providing most of the LDAP services.  We were able to determine that Microsoft ADC and LDS weren't the only software providing CLDAP services though the Internet presence of others was statistically insignificant.  We were also able to determine that CLDAP could be used in DRDoS attacks if significant numbers of services remain available on the public Internet.

Community contributions

Rapid7 is a strong supporter of open source.  In addition to well known projects like Metasploit we also open source some of our tools and utilities. Two such tools were used in this study and have been enhanced as a result of it.

Recog

The initial scans provided us with a corpus of responses that we could use for generating product fingerprints. Reviewing the responses proved to be time consuming but very fruitful. The result was that Rapid7's open source fingerprinting tool Recog was updated to support LDAP fingerprints and an initial 55 fingerprints were added. Part of the update also included adding support to Recog for building fingerprints and examples with binary strings. This means that server responses can be handed directly to Recog without being processed or stripped if the fingerprint was constructed to support it. This functionality is available in the public Recog GitHub repo and gems as of version 2.0.22 which was released on August 25, 2016.

DAP

As part of our study process we often run the scan results through Rapid7's open source Data Analysis Pipeline (DAP) tool for data enrichment. This generally includes adding geoIP tagging and protocol decoding if appropriate. DAP can also leverage Recog's version detection for determining the remote service's product, version, and/or operating system. Part of this study included adding support for decoding LDAP protocol responses to DAP. It will now decode a properly formatted LDAP response into a nested JSON structure. This functionality is available in the public DAP GitHub repo and gems as of version 0.0.10 which was released on August 2, 2016.

Example

In the example below DAP is handed JSON formatted data that contains an IP, a base64 encoded server response, the port, and protocol. It returned additional data including version detection (dap.recog..), decoded LDAP response (data.SearchResultEntry…., data.SearchResultDone), and geoIP information (ip.country…, ip.latitude, ip.longitude). At the end the jq utility is used to improve the readability of the result.

Command

bash echo '{"ip": "66.209.xxx.xxx", "data": "MDACAQdkKwQAMCcwJQQLb2JqZWN0Q2xhc3MxFgQDdG9wBA9PcGVuTERBUHJvb3REU0UwDAIBB2UHCgEABAAEAA==", "port": 389, "proto": "tcp"}' | \
bin/dap json + \
transform data=base64decode + annotate data=length + \
recog data=ldap.search_result + \
decode_ldap_search_result data + \
remove data + geo_ip ip + \
geo_ip_org ip + json | jq 

Result

{  
  "ip": "66.209.xxx.xxx",  
  "port": 389,  
   "proto": "tcp",
   "data.length": 64,
   "data.recog.matched": "OpenLDAP",
   "data.recog.service.vendor": "OpenLDAP",
   "data.recog.service.product": "OpenLDAP",
   "data.SearchResultEntry": {
     "objectName": "",
     "PartialAttributes": {
       "objectClass": [
         "top",
         "OpenLDAProotDSE"
       ]
     }
   },
   "data.SearchResultDone": {
     "resultCode": 0,
     "resultDesc": "success",
     "resultMatchedDN": "",
     "resultdiagMessage": ""
   },
   "ip.country_code": "US",
   "ip.country_code3": "USA",
   "ip.country_name": "United States",
   "ip.latitude": "36.xx",
   "ip.longitude": "-115.xxxxxxxxx"
}

This is one of the simpler examples. The responses from Active Directory Controllers contain significantly more information.

Final thoughts

The results from the study were able to successfully answer the original questions but prompted new ones. Fortunately there is quite a bit of information that remains to be extracted from the data that we have. It is likely that there will be multiple future blog posts on the data, the software and versions identified in it, and information about the individual hosts that can be extracted from it.

Unlike other studies that Project Sonar publishes on Scans.IO, we have not yet made these data sets available. For those of you that are concerned that your network may be exposing LDAP services I recommend the following:

• If your organization is a service provider or a company with assigned ASNs you can sign up for free Shadowserver reports. The Shadowserver Foundation scans the Internet for certain services of concern, such as those that could be used in DDoS, and will provide regular reports on these to network owners. Last week they announced support for discovering and reporting on CLDAP.

• Use an external service, such as the Rapid7 Perimeter Scanning Service, or an externally hosted scan engine to perform scans of your Internet accessible IP space. Ensure that the ports listed above are included in the scan template. This will provide a more accurate picture of what your organization is exposing to the Internet than that provided by an internally hosted scanner.

• Review firewall rules to determine if any are overly permissive.

As always, if you are interesting in having Sonar perform additional relevant studies, have interests in related research, or if you have questions, we welcome your comments here as well as by reaching out to us at research@rapid7.com.