To start, you need to identify which technologies are being used in your applications. Classic HTML, PHP, or Perl applications are no challenge for traditional DAST tools because that’s what they were originally built to handle. However, dynamic HTML5, Action Message Format (AMF), AJAX-driven, and Single Page Applications (SPAs) are a different story. SPA frameworks like ReactJS and Angular start off with a minimal HTML loading framework; if your dynamic application security solution doesn’t fully understand that framework, it will have no way to effectively crawl and analyze them.

APIs are used by both mobile and web applications, but often times present weaknesses in authentication and session management. Because of this, they make enticing targets for attackers to tamper with, commit fraud, or perform man-in-the-middle attacks in order to gain access to protected personal information or intellectual company property. Modern applications further complicate matters because they often use APIs behind the scenes, as well as obfuscated multi-step workflows like shopping cart sequences, which cannot be crawled at all—this can be tricky for security teams.

Lastly, it’s important to note that attackers are increasingly targeting backend services. While APIs leveraged by mobile apps create new ways to engage customers in rich internet applications, they also create novel ways for attackers to reach backend servers.

That’s why it’s more important than ever to keep track of the technologies used in your applications, so you can find an appropriate web application security scanner and supplement your scanning efforts accordingly. A modern application security program needs to be built around the ability to understand and test apps with sophisticated interfaces, APIs, and protocols: in some cases, a “Universal Translator” feature may be leveraged to handle custom parameters and advanced authentication processes, and automatically detect vulnerabilities in complex application workflows like shopping carts. 

Below is a list of the advanced technologies that your applications may be using:

• Applications built on Single Page Frameworks (SPAs) like ReactJS and Angular
• AJAX-driven applications, such as JSON (JQuery), REST, GWT (Google WebToolkit)
• Applications with Flash remoting, like Action Message Format (AMF)
• Applications with HTML5
• Backends of mobile apps powered by JSON, REST, and other custom formats such as Google Protobuf
• Web services and RESTful APIs, such as JSON, REST, XML-RPC, and SOAP
• Complex application workflows, like shopping cart sequences or other strict processes and XSRF/CSRF tokens

Talk to your developers to get a list of technologies and versions; since this can be a time-intensive process, you can get a head start with technology profiling tools like Wappalyzer. Use this list of software and versions to ask vendors about compatibility when you evaluate DAST tools.

If your applications use (or plan to use) these technologies, consider cross referencing that list with the capabilities of the solutions you may be evaluating—this brings us to step two. 

You can’t protect what you can’t see, which is why crawling your applications is critical to ensuring you release secure applications. But not all scanners are created equal. As development techniques continue to modernize, it becomes increasingly difficult (if not impossible) for traditional or outdated scanners to keep up. This can mean missed vulnerabilities and issues simply because the scanner isn’t built to account for new frameworks and formats used to build and support your app.

Many DAST solutions will also produce a crawl map: a list of URLs that were accessed during the simulated attack. This is a good starting point to confirm that a DAST tool has crawled every URL you own, as provided by your developers. However, with modern applications, parts of the application may not be on a separate URL, such as pop-in windows in SPAs. If you’re scanning these types of applications, be sure your DAST tool also provides a list of document object model (DOM) locations accessed.

To ensure you are using the best scanner for your applications, one that will work for your current applications and that continuously keeps up with new application development techniques down the road, we have included criteria for you to use below.

How to evaluate DAST engine coverage

AJAX

AJAX is an older, but still common asynchronous communication format based on XML. You can test for the security scanning coverage of an AJAX application by analyzing the request/response traffic. To do so, simply enable the DAST tool’s detailed logging feature, run it through a proxy like Paros, Burp, or WebScarab, and save the logs for manual review.

JSON

RESTful APIs submit their requests using JSON, where attackers can hide malicious code. It’s important to be able to simulate these attacks to ensure your web application is sanitizing incoming JSON, especially since it can be nested. By reviewing traffic logs from a scan, you can indicate whether the DAST tool is fully capable of scanning dynamic applications like yours. Check to make sure it is attempting to inject into JSON, hopefully on different key/value pairs and on multiple nested levels. However, not all solutions provide detailed logging—if this is the case, you will need to set up a proxy to capture traffic during the scan. Begin by limiting your scanning scope to a page that uses JSON, then check to see if the DAST tool requests include the JSON traffic.

SPAs

Single Page Applications (SPAs) are the latest evolution in web applications and dynamically load content based on JavaScript events. SPAs are difficult for most DAST engines to scan, since the engines must emulate all of the JavaScript to fully crawl the fields of a page—if you can’t crawl it, you can’t attack it. SPAs pose yet another challenge to scanners because they use proprietary event names. During your evaluation, ensure that the solution can understand and adapt to the proprietary events for the framework you use. A proper, modern DAST solution should be able to interface with the dynamic libraries that power SPAs and interrogate them directly; if these Javascript libraries are not supported by the solution, it will be blind to most of the application, especially where interactions with the app are abstracted from the browser (i.e. a virtual DOM). Be sure to get a list of the versions of JavaScript libraries and frameworks used, and confirm that the DAST tool supports those versions.

Parameters

Parameters are the oldest form of passing variable information to a web application, as well as the most broadly supported. They pass information from the client to the server through the URL itself, or through the message body of the HTTP request. You can leverage proxies and traffic logs to see if a DAST tool is equipped to see and test your parameters. This is especially important with regards to more modern formats, such as JSON, where the parameters may be inside of arrays—both traditional and one line. Verify that your DAST solution can identify and interact with all of your parameters and that it's not simply applying old scanning techniques, hoping they can catch vulnerabilities and issues that only new scanning methods truly can. APIs are also crucial to consider, as a DAST tool should also know the correct types of values to interact with the parameters: You can’t accurately test your API parameters if your web service is sending invalid data to them.

Emerging Technologies

Microservices

Large, complex web applications are difficult to scale and maintain, so developers are beginning to split different components of an app into multiple APIs behind the scenes. A single front-end user interface (UI) may actually be calling on several (to several dozen) RESTful APIs spread across many Docker containers. Plus, different development teams are often responsible for different components of a large application, meaning any given dev team may only be concerned with DAST results for their part of the application. For these scenarios, ensure that your DAST tool can scan each API and support their particular authentication method.

Serverless Applications

Some development shops are starting to write code for serverless applications in the cloud using computational services like Amazon Lambda or Azure Functions. These applications don’t have a UI, direct access to ports like HTTP, nor an underlying operating system. In short, scanning these applications is still mostly uncharted territory, and pen tests are currently the only dependable way to assess them for risk.

Most DAST solutions first perform a "crawl" of the client interface to understand the application and generate a list of targets, then conduct an "attack" or "audit" of that list to find vulnerabilities. Legacy DAST solutions communicate with applications through the frontend to identify potential security vulnerabilities in the web application, as well as architectural weaknesses. However, newer applications have rich clients and RESTful APIs, making less and less of the applications crawlable, and therefore making it more challenging for you to simulate attacks against them to uncover dangerous vulns.

There are several key capabilities a DAST solution should have to sufficiently simulate attacks on modern applications like SPAs, mobile, and rich internet apps: These include automated testing scripts (like Selenium), API definitions (Swagger and RAML), macros, and traffic recordings.

Ingestion and normalization of data

Many DAST solutions have to write unique vuln checks for each technology type. For example, if you were scanning for SQLi, it would have one check for PHP, another check for .NET, another check for Java, and so on. Having a solution that can properly normalize incoming data prevents rewriting the same checks over and over again, and ensures that the tool can recursively scan deeper into nested structures like JSON (REST) or XML (SOAP), rather than just the top layer.

Analysis and attack of API formats

It's great if a DAST solution can consume data from a proxy, but if it's not familiar with JSON and REST, it will not be able to create attack variations, even when fed recorded traffic. (Note: Recorded traffic should be a last resort—not only is it time intensive for your dev team, but also requires re-recording whenever features are added or changed.) These API formats pose a unique challenge to DAST tools because they must be able to decipher the new formats and insert attacks to test application interfaces. Ideally, a DAST solution is actually able to attack the recorded traffic, and add new parameters or modify expected values for those parameters. By doing so, the tool can both analyze your API and interact with it so as to attack it effectively.

Automatic testing of REST APIs

In recent years, new frameworks like Swagger (OpenAPI specification), API Blueprint, and RAML have improved API testing, but in many ways, it's still a manual process fraught with errors. However, Swagger has revolutionized this with its machine-readable API definition files. Now, DAST solutions can discover Swagger definition files and automatically test APIs end-to-end with a simple URL, just like a web application. These Swagger files take the place of the crawl, ensure full coverage of the API, and are often automatically updated on a URL with every new build.

If you have APIs that are documented in Swagger, you can begin taking advantage of this today. If not, it's worth exploring as an option for securing your APIs with your developers and DAST vendor—by combining this type of automation with API parameter training, testing REST APIs becomes notably more efficient and effective.

Since Swagger is still a new format, many developers run into roadblocks generating files understood by DAST tools; work with your dev team to make sure that your Swagger files match the specifications and versions supported by your solution. For validation help, you can turn to the Swagger Editor during the configuration process.

Since Swagger is still a new format, many developers run into roadblocks generating files understood by DAST tools; work with your dev team to make sure that your Swagger files match the specifications and versions supported by your solution. For validation help, you can turn to the Swagger Editor during the configuration process.

Testing of SPA frameworks

Ever played an old video game through an emulator on your PC? (We're looking at you, Nintendo fans.) Chances are, the emulated version was a bit buggy and didn't function quite the same way as the original game. The same thing is true for automated tools trying to interact with websites meant to be rendered in a web browser for humans.

SPAs are the umbrella category for all Javascript-driven (Angular, React, Knockout, etc.) applications we see today. One of the primary reasons these applications trip up scanners is that they leverage proprietary browser events. Traditional DAST tools were built on the understanding of standard event browsers, so in order for your DAST solution to work on a SPA, it must first be able to accommodate for these proprietary events—keep this in mind when evaluating any potential DAST providers.

Every developer is guilty of Frankenstein-ing code leveraging older and newer technologies and frameworks. While this may be a quick workaround to get things up and running faster, non-standard code can also trip up DAST tools. Viewing a crawl map or inspecting the DOM elements can reveal where the solution is struggling to emulate the site. From there, you can search the traffic logs for that URL to troubleshoot the issue(s).

Ability to understand modern authentication techniques

Much like legacy DAST tools, traditional authentication methods fail to deliver the protection they once promised. While they were historically used to protect server-side mobile applications from SQL injection and cross-site scripting (XSS) attacks, many of today’s authentication methods simply aren’t sophisticated enough to provide adequate security. For example, attackers can still exploit weak passwords when a scheme authenticates only the user and not the application. So, how can you avoid this?

Client-side certificates can be used to identify the application, but this isn’t feasible for all apps (especially customer-facing, mobile ones). Thus, companies have been using more sophisticated mechanisms for authentication such as single sign-on (SSO), multifactor, CAPTCHA, or the use of security questions that change each time you log in.

Review documentation, or ask potential DAST providers which techniques they support and how they map to the ones your development team utilizes. Otherwise, it may prove challenging for the scanner to crawl every part of your app, leaving you vulnerable to a blind attack for applications in production.

Better, safer applications for tomorrow start with more secure development practices today. One of the most impactful steps you can take to future-proof your applications is to work toward a DevSecOps approach. This is the concept of adopting tools and processes that allow software development, security, and operations teams to work together to integrate security into every phase of the software development lifecycle (SDLC). This ensures that vulnerabilities are detected before application builds are promoted to the next level or put into production.

In order to achieve this, a few key processes should be in place:

• Embed vulnerability assessment agents into your virtual environment instances so that vulnerabilities and misconfigurations can be identified as soon as an instance is spun up.
• Integrate vulnerability assessment tools with container registries so that images can be assessed before they are deployed.
• Apply automation to the continuous integration (CI) and continuous deployment (CD) processes so that automated security tasks are included in the build pipeline, such as container assessment and DAST scans.
• Automatically send detected issues to your ticketing or incident tracking system for immediate resolution by the application development team.

These workflows may require some time and investment up front, but the long term result is end-to-end application security that grows as your infrastructure does.

Equally as important is training developers on proper input validation, as well as other secure coding concepts. Developers need to know that each of the new formats carries the exact same risks as web applications; where we have parameters passed over HTTP, we have risk.

When developing dynamic applications and SPAs, developers should keep the following tips in mind:

• Data provided by the client should never be trusted.
• A device’s mobile equipment identifier should never be used to authenticate a mobile application. Instead, it should use multiple techniques to verify that requests are from the intended user.
• Since session tokens for mobile apps rarely expire, attackers can use them for a very long time.
• Credentials should not be stored in the application’s data store local to the device.
• When requiring SSL, a valid certificate should be required.

Additionally, you should ensure your security investments are also future-proof. Talk to your vendors about how their solutions will be able to handle new technologies as they arrive and are adopted.

Guaranteeing reliable web application security for mobile, dynamic, and SPAs can be tricky business. However, conducting the proper research, choosing the right DAST solution, and performing an ample amount of testing will help detect vulnerabilities and prevent new attacks, allowing your applications to thrive both within your walls and when presented to the end user.

Meet the DAST solution built for modern apps

Listen, you know we have to end with a quick plug—Rapid7 offers web application security solutions that can test the application logic mentioned throughout this guide. Our solutions were designed to “understand” web applications with sophisticated interfaces, APIs, and protocols and can assess them comprehensively, even if they utilize custom parameters, non-traditional authentication processes, and complex workflows. By integrating into the early phases of the software development lifecycle (SDLC), InsightAppSec can help security teams adopt a true DevSecOps approach and keep pace with rapid application development cycles. With both cloud and on-premises scan engine deployment options, InsightAppSec effortlessly scales to your application security testing needs.

We’d be remiss if we didn’t mention that InsightAppSec is well-known for providing the most automated coverage, with sophisticated crawling technology and a Universal Translator that can interpret the newest technologies being used in today’s web, mobile, and single page applications. If that wasn’t enough, it also includes interactive and actionable reports that prioritize the highest risk and streamline remediation efforts by enabling users to access and analyze the data that matters most (you can even replay attacks in real time).

Bound by regulatory or corporate governance to store data locally? Rapid7 also offers AppSpider for those needing to store their data on-premises.

Don’t think application security is a reality with your limited resources? Talk to our experts about Managed AppSec, a fully-hosted application security program.

Learn more about all of our application security solutions here.