Last Updated on February 17, 2020
Recently I spoke with one of our highly regulated clients in the financial industry, who was getting “beat up” in an audit because we had jointly chosen to run the web application penetration tests in a “near production” environment rather than in their production environment. Our thought process was as follows:
- The web application we were testing processes highly sensitive data, where integrity is critical.
- Web application penetration testing tools attempt to identify persistent injection vulnerabilities that may result in data being written to the database, thus impacting data integrity.
- Reversing successful injections in a production application/database may not be possible and/or would be more challenging if inserted data “triggers” additional actions (e.g., a stored procedure that processes our injected data).
- Testing in a near production environment that is identically configured (e.g., code base, web server, app server, database tier) provides equivalent value at virtually zero risk.
- The client would make any required changes (dictated by the test results) in production after those changes were validated in near production.
- Pivot Point Security would perform a network vulnerability assessment and penetration test in the production environment to ensure the security of the underlying infrastructure.
I was surprised when the auditor disagreed with this approach, especially, as it is the approach I see leveraged by the vast majority of our client base. The auditor was insistent (despite the fact that our client could document the identical nature of the near-production and production setups) that the testing should have been done in production. When it was explained that the risk associated with a successful injection was deemed too high, he cited that the web application vulnerability scanner should have been used in a “safe” mode.
Needless to say, running in a safe mode means that it is not performing those tests that are not deemed safe. So by running in a safe mode it may miss critical vulnerabilities – essentially a false negative – which is the most dangerous result you can have in a pen test.
So he asked the auditor “ Do you want me to run:”
- A “safe” test in production that may miss things
- A “comprehensive” test in near-production whose results are applied to production
- A “comprehensive” test in production that may impact the integrity of client data
To my surprise he didn’t care whether it was ‘1’ or ‘3’ – as long as it was run in production. I find the answer remarkable. Do you?
I found one other thing suprising — that I could not find any “standard” guidance on this from resources like OWASP or Microsoft.
If you are just learning about OWASP’s testing standard or are considering the best way to prove the security of an application, this guide is meant for you!
Hello and appreciate your article! I also have clients who are apprehensive about pen testing apps in production but required to do so by audit. Has there any further discussion on the case you have presented, even how to ensure the non-prod environment is (or nearly) a mirror image of prod?
thanks for your help,
Gerry Asche
Internally, we’ve discussed this at length. The “right” answer really comes down to what the client wants. We’ve performed several penetration tests against applications that are within the client’s PCI CDE (which states testing should be performed against production). How the testing was conducted, usually came down to what the auditor wanted to see. We had some who were fine with us testing the application at the client’s DR site, or a production equivalent staging environment. However, we’ve also worked with auditors that wanted the application tested in the live, production environment. In those cases, we documented exactly what we would be attempting, with potential results, and had both the client and auditor acknowledge the risks. We then tested in production, in off-hours after confirming they had a valid, full DB backup (or any method to roll back changes).
The bottom line is that some standards (and/or auditors) dictate that an application must be tested in the production environment. The best thing you can do educate them to the potential risks, ensure they have some sort of working disaster recovery method, and agree upon procedures should your testing cause an issue in production.
Hi John
I would only accept production environment too!
You are talking about web apps. They have to be best protected. I can tell you I never saw a perfect cloned environment. Even using full auto scripts to roll out whole IaaS & PaaS components and configurations. You will find something different anytime. I thought like you but after running several tests – my advice: take the prod env.
If you do not trust in the setup I would run the test against test- or preprod-env first. Fix (or let fix) the system and run it again against prod env.
In the end: nobody needs information about other env and how they are configured. Only results from production environment counts.
BR
Eduard
Eduard,
Good feedback and I agree with your thought process. If someone is willing to test first in a non-prod environment to minimize the likelihood that there is a problem, most notably the database being impacted, and then test in prod, that’s a great approach. The problem is most clients will not want to pay for two tests. Prod also has some other challenges in that there are often elements of an application (e.g., form fills) that can overwhelm a help desk or mail queue.
Thanks for your thoughts —
Just to update if anyone looking at this matter:
Under the PCI DSS Information Supplement Penetration Testing Guidance dated March 2015 prescribes:
Section 2.3.4 Separate Testing Environment
Because of the nature and the intent of penetration testing, such testing in a production environment during normal business hours may impact business operations, and attempts to avoid disruption may increase the time, resources and complexity of the testing. This is especially important for high availability systems that may be impacted by penetration testing in a production environment.
To avoid disruptions and to speed up testing, a separate environment that is identical to the production environment may be used for testing instead of the production environment.
The penetration tester would need to ensure the same application and network-layer controls as production exist in the testing environment. This may be
accomplished through methods to map out the production environment to verify it matches the testing environment.
This should be included in the rules of engagement. All exploitable vulnerabilities identified during the testing must be corrected on production systems and testing repeated to verify that security weaknesses have been addressed.
I’m wondering: what would be those “methods to map out the production environment to verify it matches the testing environment”.
You would have to show sufficient likeness. so application versioning… infrastructure and middleware versioning, interfacing to upstream and downstream assets? Are there comprehensive lists that auditors would find good enough?