Last Updated on January 31, 2019

Recently I spoke with one of our highly regulated clients in the financial industry, who was getting “beat up” in an audit because we had jointly chosen to run the web application penetration tests in a “near production” environment rather than in their production environment.  Our thought process was as follows:

• The web application we were testing processes highly sensitive data, where integrity is critical.
• Web application penetration testing tools attempt to identify persistent injection vulnerabilities that may result in data being written to the database, thus impacting data integrity.
• Reversing successful injections in a production application/database may not be possible and/or would be more challenging if inserted data “triggers” additional actions (e.g., a stored procedure that processes our injected data).
• Testing in a near production environment that is identically configured (e.g., code base, web server, app server, database tier) provides equivalent value at virtually zero risk.
• The client would make any required changes (dictated by the test results) in production after those changes were validated in near production.
• Pivot Point Security would perform a network vulnerability assessment and penetration test in the production environment to ensure the security of the underlying infrastructure.

I was surprised when the auditor disagreed with this approach, especially, as it is the approach I see leveraged by the vast majority of our client base. The auditor was insistent (despite the fact that our client could document the identical nature of the near-production and production setups) that the testing should have been done in production.  When it was explained that the risk associated with a successful injection was deemed too high, he cited that the web application vulnerability scanner should have been used in a “safe” mode.

Needless to say, running in a safe mode means that it is not performing those tests that are not deemed safe. So by running in a safe mode it may miss critical vulnerabilities – essentially a false negative – which is the most dangerous result you can have in a pen test.

So he asked the auditor “ Do you want me to run:”

1. A “safe”  test in production that may miss things
2. A “comprehensive” test in near-production whose results are applied to production
3. A “comprehensive” test in production that may impact the integrity of client data

To my surprise he didn’t care whether it was ‘1’ or ‘3’ – as long as it was run in production.  I find the answer remarkable. Do you?

I found one other thing suprising — that I could not find any “standard” guidance on this from resources like OWASP or Microsoft.