A functional evaluation is performed to understand how the device operates under normal use, which allows for functions, features, components, and communication paths within the product’s ecosystem to be discovered and mapped. Using the data obtained from this type of functional evaluation, a detailed test plan can be developed to encompass as many of the six targeted tested areas, previously discussed, in the products ecosystem. 

During the functional evaluation, we always recommend that two environments be set up and configured for testing. This allows for more comprehensive testing by providing additional opportunities to perform cross-account and cross-system attack simulations. A second environment also allows for comparisons to be made between normal and altered configurations. Also, when hardware-level testing is encountered later, destructive testing can be performed against one of the environments while still maintaining a usable test environment.

As part of the functional evaluation, once all the normal conditions of operation have been mapped out, it is also practical to then test the ecosystem against some common failure states. Examples of typical failure states include:

• Loss of internet connectivity
• Loss of Wi-Fi access point connectivity
• Failure of a local subcomponent such as a sensor, hub, or actuator
• How the ecosystems handles changing topologies overall

When these states occur, we check to ensure the overall security of the ecosystem has not degraded, which could allow unauthorized access to the system or data. 

A common example of this is often encountered with consumer IoT products. For example, when an IoT product cannot connect to its Wi-Fi access point as configured, what will normally happen? Will it fail to operate? Or, will it fall back into its initial configuration state, meaning the device starts up an internal Wi-Fi access point as it did when it was first taken out of the box? 

In most cases, the device would ideally stop working and require a factory reset before dropping back into configuration mode, but that’s not always the case. We continue to encounter products that drop back into configuration mode when they fail to connect to their configured Wi-Fi access point. This allows malicious actors to take control of the devices and modify the configurations, and in some cases, extract historical configuration data that can then be used to compromise the primary Wi-Fi network when it does come back online.

Throughout the IoT ecosystem assessment, open source intelligence (OSINT) gathering is used to discover information about the device, system components, and supporting infrastructure. This discovery and enumeration can include researching the following components:

• Product make and model
• Software/firmware components
• Web, cloud, and API components
• FCC filings and RF testing records
• Hardware component manufacturer data sheets
• Historical records on vulnerabilities in the product and comparable products
• User and administrative manuals

OSINT data helps with many phases of the assessment and can be used to identify hardware disassembly, vulnerable code, or product design flaws that impact the security of the product’s overall ecosystem.  At a minimum, OSINT helps to define further testing methodologies within the product ecosystem, which helps to ensure a more thorough evaluation of the product's security.

IoT technology typically uses various web services for remote control, data collection, storage, and product management. Web services and cloud API components can be a critical component in an IoT product ecosystem and potentially vulnerable to attack and exploitation.

By conducting a comprehensive assessment of associated cloud services, APIs, and communication between identified cloud services and components within the product ecosystem, many security issues can be quickly identified. This review includes the manual and automated testing for common web application security issues, and complex business logic flaws within the services. This testing also includes testing of the OWASP Top 10. 

As part of this phase of testing, we often configure the management application and hardware communication to be routed through a web proxy for the purpose of analysis and testing of cloud services security. If communication is found to be encrypted, this often requires the installation of a self-signed web proxy SSL/TLS certificate to intercept and decrypt all encrypted traffic.

IoT ecosystem testing also includes an assessment of the application used to manage and control the device. This can often be a mobile application (Android, iOS, etc.) or other thin or thick clients used to remotely manage or control the IoT system under test. In-depth testing and analysis is conducted against all management and control applications used to manage IoT products and systems.  

Similar to cloud-focused testing, detailed testing of functions and communications between management and control applications and associated components or cloud services in the product ecosystem is required to review the overall security posture of the product. 

As part of this testing, the applications are examined to see whether they are storing any confidential data on the local system and validate, if that data is properly encrypted. With mobile applications, various techniques are used to decrypt and decompile both Android and iOS applications for examination for security issues. This testing also includes testing of the OWASP Mobile Top 10 to identify risks or lapses in security practices.

Often, these investigations also reveal mechanisms within the application used to update the embedded device's firmware. As part of this testing process, the firmware for embedded devices can often be captured, when the management application is used to retrieve and deploy the firmware to the embedded hardware. Also, these application mechanisms can frequently be leveraged to launch malicious firmware update attacks on embedded devices to further test the device's protections against such attacks.

IoT technologies that expose services through standard network stacks such as Ethernet and Wi-Fi can have risks associated with them. Accessible network UDP and TCP ports, protocols, and services discovered within the IoT product ecosystem are reviewed and analyzed. Using this information, a thorough penetration test is conducted to identify vulnerabilities or misconfigurations that might be leveraged to compromise the system or gain access to critical or sensitive information.  

As part of this phase of testing, capturing communication from the various components in the products ecosystem is done, often using tools such as Wireshark network sniffer. This captured data will often reveal the critical communication paths, ports, and services used by the hardware, software, and internet services within the product’s ecosystems. 

For example, while capturing network traffic from the IoT product ecosystem, we often encounter the protocol MQTT.  In many cases, this protocol is found to be misconfigured, which includes encrypted communications not enabled and wildcard requests against the MQTT broker allowed. When wildcard requests are allowed, a malicious actor can extract critical data or alter the operations of the product ecosystem of all users of the MQTT broker. 

In the case of Wi-Fi communication, the open nature of the radio environment leads to the possibility of eavesdropping by malicious users and will require suitable security algorithms to ensure data privacy, data integrity, and protection against denial-of-service (DoS) attacks. Four basic security requirements here include: 

1. Confidentiality: Guarantees the contents of the message are disclosed only to authorized individuals/endpoints. 
2. Authorization: Guarantees the sender is authorized to send a message.
3. Data integrity: Guarantees the message was not modified in transit, either accidentally or deliberately. 
4. Refutability: Guarantees the message was sent by a properly identified sender and is not a replay of an earlier message. 

During the network testing phase, all of these areas of concern will be tested and validated.

Physical inspection is performed to assess the characteristics of the physical attack surfaces of the targeted device. This inspection includes a detailed examination of the embedded device's interior and exterior characteristics. Embedded devices have myriad components and characteristics, with some common attack vectors including: 

• Exterior interface ports 
• Volatile and non-volatile memory components
• Debug console access interfaces
• Serial console access interfaces 

After an inspection, physical testing and applicable attacks are performed. Though attack vectors may vary across devices, they often follow common themes. Examples of physical testing could include the following actions:

• Access or privilege escalation through available ports
• Circumvention of device protections, such as boot loader restrictions or restricted BIOS controls
• Device configuration modifications
• Access of storage components to extract sensitive or confidential information, such as authentication credentials, private keys or tokens, and personally identifiable information (PII)
• Access to firmware through hardware access that might be inaccessible in default configurations.
• Access to debugging or console logs to gather system information or other sensitive artifacts

One common issue often encountered on devices that utilize a full embedded operating system is the use of U-Boot. Many U-boot versions can be attacked to force the device to fail into a U-boot console, and from there, it is often possible to configure the device to boot into a single user mode, allowing for root-level access to the device's operating system.

In another common test example, when debugging interfaces such as JTAG, SWD, cJTAG, SPI, and ISCP are identified, specific security tests are conducted to see whether each CPU or MCU on-chip protections are properly implemented to prevent the easy extraction of the firmware from the device. Of course, whenever external flash memory is used for storage of firmware and configuration data, even if debug services are disabled, it can be easy to remove the flash chip and extract the data using a standard flash chip reader. Once the firmware is extracted, further testing can be conducted, as discussed in the next section.

One testing method that can help reveal important data about the overall end-to-end security of an IoT product ecosystem is inter-chip communication analysis. Analyzing data as it traverses the circuitry of an embedded device can reveal details often missed or unavailable when examining data flowing to the device through normal external avenues. This can include:

• Device commands
• Key strengths (Fixed, Rotating)
• Authorization and validation processes
• Encoding methods

When firmware and binaries are accessible via physical or application-level attacks, they will be reviewed. During this process, the firmware is examined for the following data:

• Configuration information
• Hardcoded keys
• Hardcoded passwords
• Undocumented commands
• Undocumented URIs 
• External API calls 

Often, machine-to-machine communication leverages hardcoded passwords and keys that are found through firmware analysis. Leveraging this data, further attacks against the product ecosystem can be conducted, allowing unauthorized access to subcomponents, cloud services and APIs, and management and control applications. 

It is also common to find that this access control data can be used across all of the products. For example, if an attacker finds the RSA private key stored within the firmware of a specific IP camera and determines it is identical across all the same brand of cameras, the attacker can use this data to decrypt all communication to and from these products to compromise not just the one camera, but everyone’s cameras.

Many embedded and IoT devices also use radio-based communication (RF) methods. Protocol-specific communication testing is also performed to identify RF security issues to help determine risk and impact. Specialized testing and analysis of RF communication is conducted to identify whether the communication:

• Conforms to expected encryption techniques
• Uses component pairing processes that cannot be subverted
• Allows unauthorized access or control
• Can be easily used to discover and enumerate underlying command and control transmissions
• Is vulnerable to replay attacks
• Uses outdated protocol standards

By leveraging software-defined radio (SDR) solutions, RF communication can be captured, analyzed, and replayed to conduct in-depth testing of the RF protocol in use. During this phase of testing, we can also leverage information from other phases to expand the testing further. This includes RF testing data from FCC records and leveraging RF communication IC chip configuration data. Such data can often be captured during inter-chip communication analysis on the physical embedded hardware.

Also, it's important to note that many IoT ecosystem devices may only use RF communication and are not directly accessible on conventional networks. Because of this, updating the firmware must be conducted over RF, which is often referred to as over-the-air (OTA) updating. For example, Bluetooth Low Energy (BLE) devices often have their firmware pushed via OTA from a nearby management and control mobile application. During security testing, the OTA updates can be leveraged to inject altered firmware into the device using this mechanism.

Using the previously discussed methodology, we conducted security testing on over 50 IoT product ecosystems across the following five industry categories:

During testing, we identified many similar issues, as shown in Figure 3, below:

Clear-Text Storage of Confidential Data

The most common issue encountered was the clear-text storage of confidential data, with the consumer and medical spaces being the most predominant industry categories affected by the issue. The type of data encountered in this category is typically authentications and configurations data, passwords, encryption keys, and control and systems management data. During testing, this data was often found to be easily accessible, and in many cases, when passwords and encryption keys were obtained, it was possible for the tester to compromise other key components of the product’s ecosystems. 

In nearly every case where this was encountered, mitigation solutions were readily available, but not implemented. For example, consumer IoT technology, which leverages mobile applications for management and control in the product’s ecosystem, was rarely found to be configured correctly. 

For Android applications, the solution is to leverage Keystore, which allows for the storage of cryptographic keys in a container to help mitigate against the extraction or unauthorized use of the credential data. If properly implemented, end users’ authentication data can be properly encrypted during storage. 

For iOS applications, the solution is the keychain service. Similar to the Android keystore, the iOS Keychain API provides a solution for applications to safely store secret information by encrypting it before it is stored on the local file system. Both of these solutions provide standard methods for major mobile operating systems and at a minimum, should be implemented on all mobile applications where critical data, such as authentication information, needs to be stored.

Weak Passwords

The second most common finding while testing IoT technology was weak passwords, which was no surprise. This category included default passwords, hardcoded passwords, no password, and weak passwords. The industrial IoT technology space had the biggest exposure in this area, with consumer IoT technology close behind.

Weak passwords have been a thorn in the flesh of IoT technology since its inception. We have seen a drastic improvement by consumer product manufacturers to correct this error over the past 12 months, and I also expect the various legal requirements such as California SB-327 and the United Kingdom's Code of Practice for Consumer IoT Security to make an impact. Hopefully this means the default password issue will be behind us in the consumer IoT technology area, but with that said, I still think we need to continue the battle against weak passwords for products in the other industry categories. 

HTTPS Disabled or Not Enforced

While testing IoT technology, we often encounter embedded web servers on IoT-embedded devices. During testing of these embedded web services, we encountered the issue of HTTPS being disabled or not enforced. This was found to mostly affect enterprise-type IoT technology. 

When HTTPS-encrypted web services are not enabled by default on embedded devices, we run the risk of critical information such as passwords being exposed in clear-text over the network internally and externally during the login process. The best way to resolve this issue is to make sure the IoT products being purchased support HTTPS, then make sure that HTTPS services are enabled when the IoT is being deployed, if used. If not used, all web services should be disabled, if possible.

In this section, we give a general breakdown and comparison of industries (Table 1), ecosystem components (Figure 1),  and severity of the issues discovered (Table 2) during IoT product testing.

Examining the critical findings table, we see that medical and industrial IoT technology were the most impacted, with the findings predominantly being found on the embedded technology itself. However, while examining medical IoT technology, an equal share of critical issues were also found on the web APIs. The severity level on the finding from medical devices appeared to have a higher occurrence of critical severity, mainly because of the impact caused by security failures on these devices, as opposed to similar issues that may have been found on consumer-grade IoT devices. Also, when we examine the “high” findings, we again make note that medical IoT also consumed the highest level within this severity class. 

Also, we can see in Figure 4 that no findings were identified related to enterprise IoT technology in the category of management and control application. This is due to the fact that testing of these devices was limited to embedded hardware, and the typical management for the devices that were tested was predominantly handled by cloud web APIs in these cases.

The limited finding related to Transportation is due to a limited number of test samples, which only included embedded hardware, and a more focused effort by manufacturers on security during the development of the tested technology. 

In conclusion, it is important to note that a defined methodology is critical to ensure an effective IoT security testing program is in place and that the methodology focuses on the entire product ecosystem. By following this methodology, manufacturers can develop an effective testing program that will improve the security of their products when they go to market.

Rapid7 (Nasdaq: RPD) is advancing security with visibility, analytics, and automation delivered through our Insight cloud. Our solutions simplify the complex, allowing security teams to work more effectively with IT and development to reduce vulnerabilities, monitor for malicious behavior, investigate and shut down attacks, and automate routine tasks. Customers around the globe rely on Rapid7 technology, services, and research to improve security outcomes and securely advance their organizations. For more information, or to get involved in our threat research, visit www.rapid7.com.