Hi !
Yuri Shabalin, Chief Architect at Swordfish Security, is back on the air. We've been consulting on building secure development processes for our clients for quite some time now. In this process, we constantly encounter various nuances and challenges when implementing DevSecOps practices. We've previously discussed SAST and OSA, and today I'd like to discuss the practice of automated dynamic application analysis (DAST). Specifically, I'll discuss the pitfalls we face when implementing this practice, what special considerations should be taken, and how to avoid problems.
Well then, let's get started.
First, it's a "spider" for mapping the application. It crawls every link on every page it can reach (a process called crawling), clicks buttons, examines the contents of files that might provide additional links, and uses a dictionary to search for potential page and directory names. This allows it to assess the attack surface, including all possible points of interaction with the application—forms, open and hidden input fields, request parameters, dynamic elements, HTTP header details, and so on.
The difficulty with spiders is that a page may contain dynamically generated content (such as AJAX elements that are understandable to the user, such as a link or input field) that spiders cannot see. Consequently, if the application crawl is incomplete and such elements are not included, the number of vulnerabilities that could otherwise be detected is greatly reduced.
The second component of the scanner is the "analyzer." This component typically operates in either passive or active mode and directly analyzes the application. In the passive mode, the analyzer works only with the data received from the "spider," analyzing server responses to determine what information and headers are present, which of the detected links could potentially contain sensitive data (or access restricted areas of the application), and so on. The tool itself does not send any requests and does not interact with the application in any way.
In active mode, the analyzer begins sending requests with incorrect data for the interaction points detected by the spider, as well as for others, such as HTTP headers or parameters that do not appear anywhere on the pages but may be used in the application (for example, the GET request parameter &isAdmin=true).
Based on the server's responses and reactions, the analyzer then draws a conclusion about the presence of a particular vulnerability. This typically involves either evaluating the server's response time for "correct" and "incorrect" data, or searching for a specific string in the response text using a pattern.
As mentioned earlier, DAST is a practice that implements automated, dynamic analysis of an application. This is accomplished using various tools, the selection of which should be based on very specific parameters. Let's highlight some of these and discuss them in more detail. Some are common to all tools, while others are applicable only to dynamic scanners.
But a logical question arises: how to conduct such a test. In the case of false positives, it's quite simple: analyze the report, sort through the alerts, determine which portion are actually false positives and not vulnerabilities, and based on this, we have a rough idea of the tool's "fault" rate. However, to understand which security issues the scanner missed, we need to have an idea of what vulnerabilities the application actually contains. This isn't known in advance. There are several ways to test the scanner, and which one to choose is up to you.
Let me give you a few possible cases:
So, ideally, to compare the scanning speed of different tools, you should run them against the same application under roughly the same conditions. If you use virtualization or containerization, you can spin up a new instance with the same number of elements within the system to measure speed before each new tool, but it's important that the scanners are under the same initial conditions. In my experience, we tested tools one after another, and each subsequent one showed progressively worse speed results. It turned out that during previous scans, a huge number of new entities had been created within the application, causing it to respond significantly more slowly to requests. Yes, this wasn't noticeable during normal use, but when there were tens of thousands of requests, every second of delay added to the final result, significantly slowing down the scan.
Situations vary. Sometimes you need to scan an application in several ways, from a full, comprehensive analysis to a light, superficial check, and the tool's ability to fine-tune its settings can be very helpful.
It's worth paying attention to the number of parameters a tool allows you to customize, how easy it is to adjust them, and how scan time varies depending on specific settings. I would recommend creating several scanning profiles: quick and superficial for an initial app analysis, and full or maximum for a more comprehensive analysis. Then, I would compare these modes across different vendors. Believe me, the results of testing multiple tools can be surprising.
The first is how many technologies and frameworks your tool covers. For example, we use Angular, Vue.js, and React for the frontend and Python, PHP, and .Net for the backend across all our services. The tool we're testing only supports Angular, React, Python, and PHP (that is, four of the six technologies/languages).
The second (and indeed decisive) parameter may be support for key technologies or technologies used in your most critical and important services. For example, your main application may be written in Angular+Java, while a ticket portal or landing page is built on .Net. In this case, the focus shifts to supporting the technologies for the most important service. So, identify a list of such services and test how the dynamic analysis tool performs on them or similar projects.
I've encountered several cases where a tool had its own Jenkins plugins, but they weren't very user-friendly. I wanted more freedom and customization, so I had to write my own API wrapper. I wish I hadn't, as I've definitely grown more gray hair trying to understand how the tool works, why the documentation examples don't work, and why the same token is returned and accepted in different formats (with and without delimiters, lowercase, uppercase, etc.) in three different APIs. After this experiment, I wanted to find the author of this API and have something emotional to say to them, as working with such an API is very difficult.
So, it's important to evaluate it holistically. How well the tool integrates with your development tools and how easy it is to use its API, because sooner or later you'll need it.
Here, too, you'll have to put in some effort to gather all possible login options across all the services. It might seem like there's nothing complicated about it. Just enter the correct login and password into the form, get a session, and off you go, analyze. But as usual, the devil is in the details. We've seen all sorts of things under the hood of authentication: password hashing before sending, encryption with a shared key on the front end, certificate authentication, Kerberos, and what not. Whether the tool supports all this is a good question. It would be extremely frustrating and unpleasant to choose a solution that can't authenticate with some services because they weren't included in the testing area. Therefore, gather as many different applications as possible for testing and test whether you can pass the login step on each one.
A quick tip, slightly related to the previous point about APIs and integration options. Some tools support upstream proxies during scanning. This approach can be helpful if your application uses something unusual during authentication. For example, in one case, we had to implement a Mitm Proxy, write our own plugin to handle the login request, perform some parameter manipulation, and pass it on to the application, since the dynamic analysis tool couldn't independently replicate the user authentication process due to the specifics of the application. Yes, we had to think about it, do some coding, and slightly rework the scan initiation process, but ultimately, we successfully covered some services with more complex authentication. However, this wouldn't have been possible without the integration options we'd previously explored.
This also includes a point that's often forgotten (I forgot about it myself and only added it when I reread the article). It's about logging out. During scanning, many different requests are sent, and the server may simply "kick" the user out of the system in response to some of them. A dynamic analysis tool must detect this and re-run the "login" process, as further scanning is pointless. It's important to understand how flexible the conditions for this process are and how accurately the tool can detect when requests are no longer successful (this could be a server response code, a message in the response body, a header or the absence of one, and much more).
First, consider how frequently the tool updates, signatures/patterns, or analysis rules are updated. In some cases, these processes are separate, meaning that product functionality updates may be released every three to four months, but updates to the analysis rules and patterns for detecting known vulnerabilities may appear much more frequently. Check the tool's website or request this information from the vendor. This will clearly demonstrate how closely the developer follows trends and how up-to-date your test database is. Alternatively, evaluate how quickly each vendor added patterns and methods for detecting Log4Shell vulnerabilities to the compared tools, and, more interestingly, for less high-profile vulnerabilities that emerged after the Log4j library attracted the attention of researchers.
Secondly, clarify the extent to which you will be able to influence the tool's development and how feature requests are handled. This will show how quickly the functionality you need will be implemented in the product. Taking the example above (from the previous point about authentication), how quickly can the vendor refine the functionality so that the product can authenticate out-of-the-box in an application with slightly more complex logic than submitting a form to the server? And how convenient is this process, and whether you'll have to spend months corresponding with the vendor, constantly reminding them of your request.
Another good feature of this functionality is the process of enriching application information, meaning it's possible to specify which APIs are being used before scanning. This could include downloading a json file from Swagger or parsing the HTTP Archive (har) format, among many others, which can help the tool cover the application as fully as possible. I'd recommend compiling a list of what each tool can import and exploring how this can be integrated into the process. Some of our colleagues set up a special proxy server and asked testers to run manual tests through it, thereby collecting all the links and functionality being tested. Another process, as part of automated testing, compiled a har archive that was automatically downloaded before scanning, which also greatly helped increase application coverage. So, consider consulting with your testing department—they might also be able to help you decide which integrations to prioritize.
According to the new BSIMM paradigm – "Shift Everywhere," all tools should be used where they can provide the greatest benefit. Based on this, let's look at which stages of the development process we can integrate dynamic analysis.
In this case, the result can be presented as a report to the developer who ran the tests locally. If the scan is performed as part of a pull request, a comment with newly identified warnings can be directly written to the code review system. While this probably won't work out of the box in any tool, with a little effort and a few custom wrappers around the scan process, it can be quite convenient.
In any case, remember one rule: if a resource has an admin section, it should only be tested manually! I once experienced a situation where we didn't block the tool from accessing the admin panel, and it deleted all the data needed for testing, created a bunch of unnecessary data, and then simply crashed the setup. It was an unpleasant situation, but I learned well that testing the admin section should only be done manually!
But it's important to choose the right time and coordinate with everyone who will be using the testbed to avoid any overlaps or interference. Firstly, a full scan can take quite a while and can be run, for example, in the evening, when no one is actively using the testbed. Secondly, dynamic analysis creates a load on the application, which can impact the load testing results if they are run simultaneously. And of course, don't forget about functional testing, which dynamic analysis can significantly interfere with, creating additional complications or incorrect test results.
The third option is full-fledged testing within the application, systematically traversing all available APIs and trying all parameters. This is the most difficult and dangerous approach, and I wouldn't recommend it. It can trigger problems at the most inopportune moments and in unexpected places. Like what happened to me when I scanned a website on an industrial network from the corporate WiFi, and the IP address I was scanning from was simply banned. Meanwhile, a demonstration of the new website was taking place for top managers. When everyone tried to access the website from corporate devices on our WiFi, they saw a "403 - Access Denied" message instead of the beautiful new website. This was an extremely unpleasant experience, which taught me to always coordinate the timing of such events not only on test servers but also on production servers (and especially on production servers). So please don't repeat my mistakes and conduct testing only during designated times.
However, in light of recent events, the choice of dynamic analysis tools has significantly narrowed, and such a wide variety is no longer available. However, some good solutions remain. Some of the tools presented below are open source, while others are paid, commercial tools. All of them can be extended and customized to suit your specific process and needs.
OWASP ZAP
OWASP ZAP (Zed Attack Proxy) is one of the most popular security tools in the world. It's part of the OWASP community, meaning it's completely free. It has various SDKs and APIs for various programming languages, and you can use both in-house and community-developed plugins. It can be run in various modes and managed programmatically, allowing for integration into the development process. It also has plugins for various CI/CD tools. However, there's a downside: since it's open source, the scanning quality, in my opinion, is inferior to enterprise solutions. And the out-of-the-box functionality isn't particularly impressive, although it can be expanded and enhanced as needed.
BurpSuite Enterprise
This product from PortSwigger is theoretically aimed at the enterprise level. It offers a portal for configuring scans, a full-fledged REST API for interacting with and managing scan launches, receiving reports, and much more. The scanning agent is the classic and familiar BurpSuite, running in headless mode, but with some limitations. For example, you can't upload custom plugins or interact with it in any way other than through control commands from the main portal. When we tested it, its functionality was quite limited, especially when it came to setting up logins in the application. In my opinion, it's much more cost-effective to use the classic BurpSuite Pro, running in headless mode, with its plugins, extensions, and a module that implements a REST API for scanning management. Of course, this configuration requires some effort, debugging, and customization, but the results can be quite impressive, given the wide variety of plugins and capabilities of this tool.
PT BlackBox
Finally, a new product from Positive Technologies, aimed at filling a gap in dynamic analysis. The company has long had a cloud solution, PT BlackBox Scanner . It features a full-fledged REST API for managing scans, has all the makings of a detailed analysis parameter setup, and promises other exciting features. The vendor will officially release the boxed version of the product in the last week of August. We've already tested the beta version before launch, and we can say it's a very good alternative to other foreign vendors. Once the official release is released, we'll conduct further testing and a full comparison with other solutions and share the results with you.
I hope that our experience and some tips will help those who face the throes of selecting and testing tools, and will help them make the right choice and fully analyze the applicability of the tools and the practice as a whole.
Thank you all, see you in new articles!
Yuri Shabalin, Chief Architect at Swordfish Security, is back on the air. We've been consulting on building secure development processes for our clients for quite some time now. In this process, we constantly encounter various nuances and challenges when implementing DevSecOps practices. We've previously discussed SAST and OSA, and today I'd like to discuss the practice of automated dynamic application analysis (DAST). Specifically, I'll discuss the pitfalls we face when implementing this practice, what special considerations should be taken, and how to avoid problems.
Well then, let's get started.
Table of contents
- Introduction
- Key parameters when choosing a tool
- Methods of integration into the process
- What tools to look at
- Conclusions
Introduction
DAST (Dynamic Application Security Testing) is a secure development practice that involves automated scanning that simulates malicious external attacks that attempt to exploit common vulnerabilities. To detect security issues, DAST tools check all HTTP access points and simulate random user actions and interactions. This practice analyzes a deployed and functioning application. The tool simulates the actions of a regular user: it navigates the web application, determines which APIs the service has, and sends specially crafted requests. For example, it inserts various invalid data (special characters, quotes, parentheses, separators, line breaks, etc.) into various places in the request. Vulnerabilities are detected by analyzing the sent request and the received server response and comparing them with a normal, legitimate request. The dynamic analysis tool sends and analyzes a huge number of requests and can find a variety of security issues, from XSS to RCE.Key parameters when choosing a tool
Most scanners have similar capabilities and operating principles. The key components of a decent scanner are listed below.First, it's a "spider" for mapping the application. It crawls every link on every page it can reach (a process called crawling), clicks buttons, examines the contents of files that might provide additional links, and uses a dictionary to search for potential page and directory names. This allows it to assess the attack surface, including all possible points of interaction with the application—forms, open and hidden input fields, request parameters, dynamic elements, HTTP header details, and so on.
The difficulty with spiders is that a page may contain dynamically generated content (such as AJAX elements that are understandable to the user, such as a link or input field) that spiders cannot see. Consequently, if the application crawl is incomplete and such elements are not included, the number of vulnerabilities that could otherwise be detected is greatly reduced.
The second component of the scanner is the "analyzer." This component typically operates in either passive or active mode and directly analyzes the application. In the passive mode, the analyzer works only with the data received from the "spider," analyzing server responses to determine what information and headers are present, which of the detected links could potentially contain sensitive data (or access restricted areas of the application), and so on. The tool itself does not send any requests and does not interact with the application in any way.
In active mode, the analyzer begins sending requests with incorrect data for the interaction points detected by the spider, as well as for others, such as HTTP headers or parameters that do not appear anywhere on the pages but may be used in the application (for example, the GET request parameter &isAdmin=true).
Based on the server's responses and reactions, the analyzer then draws a conclusion about the presence of a particular vulnerability. This typically involves either evaluating the server's response time for "correct" and "incorrect" data, or searching for a specific string in the response text using a pattern.
As mentioned earlier, DAST is a practice that implements automated, dynamic analysis of an application. This is accomplished using various tools, the selection of which should be based on very specific parameters. Let's highlight some of these and discuss them in more detail. Some are common to all tools, while others are applicable only to dynamic scanners.
Scan quality
The primary characteristic of any tool is scan quality. In other words, it's the ratio of detected vulnerabilities to missed ones. False positives are also a significant factor, as their presence significantly complicates analysis and clutters the results. If there are many of them, real vulnerabilities can get lost among them.But a logical question arises: how to conduct such a test. In the case of false positives, it's quite simple: analyze the report, sort through the alerts, determine which portion are actually false positives and not vulnerabilities, and based on this, we have a rough idea of the tool's "fault" rate. However, to understand which security issues the scanner missed, we need to have an idea of what vulnerabilities the application actually contains. This isn't known in advance. There are several ways to test the scanner, and which one to choose is up to you.
Let me give you a few possible cases:
- In the first case, we have our application, in which a number of vulnerabilities have been discovered (through bug bounties, penetration testing reports, etc.). The key is to understand that the problems exist and know where they are. By running a scanner against this application, we try to find these detections in its results. This is a very good practice, which perfectly demonstrates the tool's scanning quality for your specific applications and its support for the technologies used in this service. And during this experiment, it becomes clear that using an automated tool, these problems could have been identified earlier, before the system went into production. Unfortunately, not everyone has such examples and setups (and even if they do, it's not guaranteed that they are still working due to API changes or other reasons).
- In the second case, we don't have a vulnerable application, but we still need to test the tool's functionality. This is where pre-existing vulnerable applications, typically created for training purposes, can come in handy. You can use them to experiment with identifying and exploiting issues. A little advice: it's best to choose something more interesting and exotic (ones the vendor doesn't know about) rather than the most popular applications (like WebGoat). Sometimes it seems like some tools have checks built right into the code. if page.title contains "WebGoat"...
In short, find an application that's closest to your developments in terms of the technology stack it uses. At the end of this article, there are several links to possible examples of vulnerable web applications that you can review and use for testing.
Scanning speed
Scan speed is always crucial, especially if our checks are integrated into the development process. It's not uncommon for a tool to perform testing, while the development pipeline waits for the scan to complete. This wastes not only time but also money. Of course, this is all regulated and managed by the integration method and processes within teams and the stage at which the scan is performed, but it is nonetheless a significant factor. In fact, scan speed depends heavily on the application being analyzed, how quickly it responds to requests, how many concurrent connections it can handle, and so on.So, ideally, to compare the scanning speed of different tools, you should run them against the same application under roughly the same conditions. If you use virtualization or containerization, you can spin up a new instance with the same number of elements within the system to measure speed before each new tool, but it's important that the scanners are under the same initial conditions. In my experience, we tested tools one after another, and each subsequent one showed progressively worse speed results. It turned out that during previous scans, a huge number of new entities had been created within the application, causing it to respond significantly more slowly to requests. Yes, this wasn't noticeable during normal use, but when there were tens of thousands of requests, every second of delay added to the final result, significantly slowing down the scan.
Possibility of detailed customization
Every automated analysis tool should have detailed scanning settings. These not only improve the quality of the process (by telling the tool which technologies are used and which checks are relevant), but also increase the speed of analysis by removing unnecessary queries and limiting the scope of analysis. Moreover, the more detailed the settings and the more precisely we can tell the tool what and how to scan, the better. Of course, there are "smart" and advanced solutions that can automatically adjust and detect various application parameters. This is certainly good, but in my opinion, it shouldn't eliminate the option of detailed manual scanning settings.Situations vary. Sometimes you need to scan an application in several ways, from a full, comprehensive analysis to a light, superficial check, and the tool's ability to fine-tune its settings can be very helpful.
It's worth paying attention to the number of parameters a tool allows you to customize, how easy it is to adjust them, and how scan time varies depending on specific settings. I would recommend creating several scanning profiles: quick and superficial for an initial app analysis, and full or maximum for a more comprehensive analysis. Then, I would compare these modes across different vendors. Believe me, the results of testing multiple tools can be surprising.
Supported technologies
Of course, you should choose a tool based solely on the technologies used in your company's development. To do this, it's best to conduct a brief analysis of such products and compile a list of relevant technologies, frameworks, and languages. This list can be quite extensive, especially for large companies, so I'd use a few parameters as some evaluation criteria.The first is how many technologies and frameworks your tool covers. For example, we use Angular, Vue.js, and React for the frontend and Python, PHP, and .Net for the backend across all our services. The tool we're testing only supports Angular, React, Python, and PHP (that is, four of the six technologies/languages).
The second (and indeed decisive) parameter may be support for key technologies or technologies used in your most critical and important services. For example, your main application may be written in Angular+Java, while a ticket portal or landing page is built on .Net. In this case, the focus shifts to supporting the technologies for the most important service. So, identify a list of such services and test how the dynamic analysis tool performs on them or similar projects.
Supported integrations
A tool that's separate from the process and not integrated into it is practically useless. While you can run periodic checks and process the results, this won't yield the desired results. For dynamic analysis (or any other practice, really) to yield maximum benefit and results, it needs to be integrated into the development process and run at specific points during the build. To understand how easily a tool will integrate into your infrastructure, compile a list of what's used in the CI/CD process in advance and create a rough plan for how and when the tool will be run. This will allow you to evaluate the tool from an integration perspective and determine its suitability. Are there plugins or ready-made integration examples for your CI/CD tools? Does the tool have a CLI? How easily does it integrate with various defect collection or orchestration systems (DefectDojo, AppSec.Hub, etc.)? How well is the tool's API described? And in fact, the API is probably one of the most important things, since sooner or later you'll still have to look into it and automate something.I've encountered several cases where a tool had its own Jenkins plugins, but they weren't very user-friendly. I wanted more freedom and customization, so I had to write my own API wrapper. I wish I hadn't, as I've definitely grown more gray hair trying to understand how the tool works, why the documentation examples don't work, and why the same token is returned and accepted in different formats (with and without delimiters, lowercase, uppercase, etc.) in three different APIs. After this experiment, I wanted to find the author of this API and have something emotional to say to them, as working with such an API is very difficult.
So, it's important to evaluate it holistically. How well the tool integrates with your development tools and how easy it is to use its API, because sooner or later you'll need it.
Types and technologies for recording login sequences
For dynamic scanners, the degree of their technology for recording login sequences is crucial. After all, the real fun is always hidden behind authentication, which must be passed to access the application.Here, too, you'll have to put in some effort to gather all possible login options across all the services. It might seem like there's nothing complicated about it. Just enter the correct login and password into the form, get a session, and off you go, analyze. But as usual, the devil is in the details. We've seen all sorts of things under the hood of authentication: password hashing before sending, encryption with a shared key on the front end, certificate authentication, Kerberos, and what not. Whether the tool supports all this is a good question. It would be extremely frustrating and unpleasant to choose a solution that can't authenticate with some services because they weren't included in the testing area. Therefore, gather as many different applications as possible for testing and test whether you can pass the login step on each one.
A quick tip, slightly related to the previous point about APIs and integration options. Some tools support upstream proxies during scanning. This approach can be helpful if your application uses something unusual during authentication. For example, in one case, we had to implement a Mitm Proxy, write our own plugin to handle the login request, perform some parameter manipulation, and pass it on to the application, since the dynamic analysis tool couldn't independently replicate the user authentication process due to the specifics of the application. Yes, we had to think about it, do some coding, and slightly rework the scan initiation process, but ultimately, we successfully covered some services with more complex authentication. However, this wouldn't have been possible without the integration options we'd previously explored.
This also includes a point that's often forgotten (I forgot about it myself and only added it when I reread the article). It's about logging out. During scanning, many different requests are sent, and the server may simply "kick" the user out of the system in response to some of them. A dynamic analysis tool must detect this and re-run the "login" process, as further scanning is pointless. It's important to understand how flexible the conditions for this process are and how accurately the tool can detect when requests are no longer successful (this could be a server response code, a message in the response body, a header or the absence of one, and much more).
Tool development roadmap
Technology never stands still: every day, a new JavaScript framework appears somewhere in the world, and developers are eager to rewrite everything using it. Therefore, the speed of support for new technologies and the speed of response to new attack vectors or vulnerability patterns are important factors when choosing a tool. Here are a few things to consider, as described below.First, consider how frequently the tool updates, signatures/patterns, or analysis rules are updated. In some cases, these processes are separate, meaning that product functionality updates may be released every three to four months, but updates to the analysis rules and patterns for detecting known vulnerabilities may appear much more frequently. Check the tool's website or request this information from the vendor. This will clearly demonstrate how closely the developer follows trends and how up-to-date your test database is. Alternatively, evaluate how quickly each vendor added patterns and methods for detecting Log4Shell vulnerabilities to the compared tools, and, more interestingly, for less high-profile vulnerabilities that emerged after the Log4j library attracted the attention of researchers.
Secondly, clarify the extent to which you will be able to influence the tool's development and how feature requests are handled. This will show how quickly the functionality you need will be implemented in the product. Taking the example above (from the previous point about authentication), how quickly can the vendor refine the functionality so that the product can authenticate out-of-the-box in an application with slightly more complex logic than submitting a form to the server? And how convenient is this process, and whether you'll have to spend months corresponding with the vendor, constantly reminding them of your request.
Quality of application mapping or crawling
If we're analyzing an application from scratch, without any additional information, we need to understand how feasible it is to collect all the application paths and transitions, how accurate the crawling will be, and what it will cover. I'd highlight this parameter as one of the most important, as the quality of the scan directly depends on it. Here, we can pay attention to the settings and methods for obtaining information. Can the application not only monitor requests from the frontend to the backend but also parse, for example, the application's swagger or WSDL? Can it attempt to find links in HTML or JS, and how well is its SPA support implemented?Another good feature of this functionality is the process of enriching application information, meaning it's possible to specify which APIs are being used before scanning. This could include downloading a json file from Swagger or parsing the HTTP Archive (har) format, among many others, which can help the tool cover the application as fully as possible. I'd recommend compiling a list of what each tool can import and exploring how this can be integrated into the process. Some of our colleagues set up a special proxy server and asked testers to run manual tests through it, thereby collecting all the links and functionality being tested. Another process, as part of automated testing, compiled a har archive that was automatically downloaded before scanning, which also greatly helped increase application coverage. So, consider consulting with your testing department—they might also be able to help you decide which integrations to prioritize.
Methods of integration into the process
So, we spent a lot of time testing the tools and selected the most suitable ones based on a number of parameters. The next step (or rather, the previous one, during the planning and selection stage, but that's not so important) is to understand how it can be integrated, into which processes, and at what stages.According to the new BSIMM paradigm – "Shift Everywhere," all tools should be used where they can provide the greatest benefit. Based on this, let's look at which stages of the development process we can integrate dynamic analysis.
Quick scan with tests before Merge Request
In some teams, I've encountered a rather convenient practice where, upon creation of a Pull Request (or at the developer's request), a separate testbed is created from the branch. This allows for fairly rapid testing of changes, ensuring that all task requirements are met, running tests before merging with the main branch, and many other useful features. In this process, in addition to running regular tests, you can also include a dynamic analysis tool, which, upon completion, will provide the developer with a report describing the vulnerabilities identified. And, by looking at new detections, you can immediately fix them quickly, without waiting for scanning at subsequent stages. I already mentioned the possibility of creating a "light" version of scanning, which either checks specific API points specified during startup, or simply performs a more superficial analysis. This can yield very good results, especially if it is made convenient for the developer and, for example, dynamic analysis is natively included in the automated test process.In this case, the result can be presented as a report to the developer who ran the tests locally. If the scan is performed as part of a pull request, a comment with newly identified warnings can be directly written to the code review system. While this probably won't work out of the box in any tool, with a little effort and a few custom wrappers around the scan process, it can be quite convenient.
After deployment
One of the classic examples of using dynamic analysis tools is this: after the code is built, uploaded to the version control system, and deployed to the test rigs, the automated testing process begins. Once these tests pass (either successfully or not, depending on the tests), the dynamic analysis of the application is launched. There are several nuances to consider. For example, ideally, it's best to have a separate rig for security checks. A limited set of tests is also run against it after deployment to ensure that the server is still operational, responding to standard requests, and waiting to be tested. This approach has its pros and cons. The advantages include an independent environment that can be safely broken or that can impact someone else's test results; separate resources dedicated solely to the dynamic analysis tool; separate user accounts that won't be changed unexpectedly; and much more. But on the other hand, this means unnecessary costs in capacity and time to maintain everything in working order, and sometimes the systems are quite complex, difficult to deploy separately and in isolation from other services, and they only work when integrated with each other.In any case, remember one rule: if a resource has an admin section, it should only be tested manually! I once experienced a situation where we didn't block the tool from accessing the admin panel, and it deleted all the data needed for testing, created a bunch of unnecessary data, and then simply crashed the setup. It was an unpleasant situation, but I learned well that testing the admin section should only be done manually!
Before moving on to preproduction
Some teams have a dedicated environment that closely replicates the production environment. This includes the same services, similar data, mandatory work not with stubs (pre-prepared server responses) but with full integration, and other features that allow for catching various errors that were difficult to detect in previous stages. In terms of security, it's also essential to utilize this opportunity and thoroughly (but non-destructively) scan the system with a dynamic analyzer. Yes, this can take significantly longer, as the scan is full-scale, with no compromises or concessions, but it's usually worth it, as the results will help you understand that everything fixed in previous stages has been corrected, that the actual integrations haven't yielded anything new, and, overall, ensure that the system is ready for production in terms of security.But it's important to choose the right time and coordinate with everyone who will be using the testbed to avoid any overlaps or interference. Firstly, a full scan can take quite a while and can be run, for example, in the evening, when no one is actively using the testbed. Secondly, dynamic analysis creates a load on the application, which can impact the load testing results if they are run simultaneously. And of course, don't forget about functional testing, which dynamic analysis can significantly interfere with, creating additional complications or incorrect test results.
Industrial environment
There's a lot of debate about whether and how to test a production environment. In my opinion, scanning an application already in production is a must, but how deeply is a good question. There are several strategies here. The first is a superficial scan without authentication to determine whether servers are configured correctly, including headers and other frequently overlooked items. This is a mandatory, "light" scan option that should always be performed. The second strategy is a kind of intermediate scan mode, where we enable all checks, including trying various identifiers, brute-forcing accessible directories, etc., but without authenticating to the application. In other words, we look at what an attacker would be able to access from the outside, without accessing the system internally. This is also a very good option for periodic testing. When running this scan, we discovered that personal user data (scanned documents, contracts, etc.) was posted on the server for one client, accessible without authorization, simply via a direct link. Of course, this didn't happen on the test benches, because this data was only in production, but without running such a check, we would never have known about it.The third option is full-fledged testing within the application, systematically traversing all available APIs and trying all parameters. This is the most difficult and dangerous approach, and I wouldn't recommend it. It can trigger problems at the most inopportune moments and in unexpected places. Like what happened to me when I scanned a website on an industrial network from the corporate WiFi, and the IP address I was scanning from was simply banned. Meanwhile, a demonstration of the new website was taking place for top managers. When everyone tried to access the website from corporate devices on our WiFi, they saw a "403 - Access Denied" message instead of the beautiful new website. This was an extremely unpleasant experience, which taught me to always coordinate the timing of such events not only on test servers but also on production servers (and especially on production servers). So please don't repeat my mistakes and conduct testing only during designated times.
What tools to look at
Recently, the choice of tools was quite extensive, including Acunetix, Netsparker, Rapid7, Nessus, AppScan, WebInspect, and other vendors that released very good solutions and could boast of very high-quality results.However, in light of recent events, the choice of dynamic analysis tools has significantly narrowed, and such a wide variety is no longer available. However, some good solutions remain. Some of the tools presented below are open source, while others are paid, commercial tools. All of them can be extended and customized to suit your specific process and needs.
OWASP ZAP
OWASP ZAP (Zed Attack Proxy) is one of the most popular security tools in the world. It's part of the OWASP community, meaning it's completely free. It has various SDKs and APIs for various programming languages, and you can use both in-house and community-developed plugins. It can be run in various modes and managed programmatically, allowing for integration into the development process. It also has plugins for various CI/CD tools. However, there's a downside: since it's open source, the scanning quality, in my opinion, is inferior to enterprise solutions. And the out-of-the-box functionality isn't particularly impressive, although it can be expanded and enhanced as needed.
BurpSuite Enterprise
This product from PortSwigger is theoretically aimed at the enterprise level. It offers a portal for configuring scans, a full-fledged REST API for interacting with and managing scan launches, receiving reports, and much more. The scanning agent is the classic and familiar BurpSuite, running in headless mode, but with some limitations. For example, you can't upload custom plugins or interact with it in any way other than through control commands from the main portal. When we tested it, its functionality was quite limited, especially when it came to setting up logins in the application. In my opinion, it's much more cost-effective to use the classic BurpSuite Pro, running in headless mode, with its plugins, extensions, and a module that implements a REST API for scanning management. Of course, this configuration requires some effort, debugging, and customization, but the results can be quite impressive, given the wide variety of plugins and capabilities of this tool.
PT BlackBox
Finally, a new product from Positive Technologies, aimed at filling a gap in dynamic analysis. The company has long had a cloud solution, PT BlackBox Scanner . It features a full-fledged REST API for managing scans, has all the makings of a detailed analysis parameter setup, and promises other exciting features. The vendor will officially release the boxed version of the product in the last week of August. We've already tested the beta version before launch, and we can say it's a very good alternative to other foreign vendors. Once the official release is released, we'll conduct further testing and a full comparison with other solutions and share the results with you.
Conclusions
When implementing any DevSecOps practice, it's important to consider the specific characteristics and nuances of your company and select a tool based on your needs. Every product has its strengths and weaknesses, but only thorough testing and piloting of various options (of which, unfortunately, there aren't many) will determine which one is right for you.I hope that our experience and some tips will help those who face the throes of selecting and testing tools, and will help them make the right choice and fully analyze the applicability of the tools and the practice as a whole.
Thank you all, see you in new articles!
