Information Security Blog

Catch 22: Outsourced Incident Response (Plan)

Catch 22: Outsourced Incident Response (Plan)

Unfortunately, here at Pivot Point we’re all too familiar with Incident Response “Catch 22″ situations. Until you have experienced a situation you often can’t (or, more frequently, don’t believe there is a true need to) develop a “comprehensive” Incident Response Plan.

This scenario is on my mind this morning because we spent the weekend helping an eCommerce client who does not have a sufficient Incident Response Plan recover from a very concerning incident. 

  • On Friday I got a call from a very good client saying: “We think we were hacked. All of the passwords on a critical, client-facing application were reset.” Here is what happened:
  • The “breach” was identified by customer service, which was “flooded” with calls from customers unable to access the application.
  • Logs were insufficient to fully determine what happened and who perpetrated it. The IIS and MS SQL logs were reasonably complete — but the underlying operating system logs were not turned on. The firewalls were only logging rejects, so they were not as helpful as they should be.  Common “time” was not being used, which means that logs for each device were offset by a few minutes, which makes investigation a lot less fun and notably reduces response time.
  • The IIS logging allowed us to determine that the application was directly taking user input and passing it to the back-end database in a Select statement. By definition this is an SQL Injection flaw. From the logs we could not directly determine if this could have resulted in the issue, so we bounced the concern and page/SQL reference to their development team. Ironically, our client had performed a penetration test against this application this past summer. A recent (insufficiently tested) functionality change caused the problem.
  • One of our consultants went on-site with appropriate tools to confirm the analysis done by the client security team and to be available to test any fixes.
  • A credentialed vulnerability assessment of the server was conducted to ensure that OS level access was not gained. This included user account review.
  • There was no formal definition of an insourced or outsourced Incident Response team. Fortunately, we were able to jump in without delay. If having an external party is critical to your response capability – then it is critical that it is contracted with an SLA. Had it been 3AM instead of 10AM, we may not have been prepared to assist them.  Several hours later the client confirmed that the SQL Injection attack vulnerability we inferred from the IIS Logs was indeed a vulnerability via a Burp Suite scan. So a critical revenue generating application remained “down” — at great financial and reputational cost. 
  • By Saturday midday, a fix was deployed for “known -bad” pages in the application and confirmed by the internal security team. Prior to re-deployment we conducted spot checks on the pages in question. On confirmation, the application was re-enabled. 
  • The development team was not confident that the “spot” fixes would not have any unintended effects – so we initiated a  full vulnerability assessment of the application.
  • A full Incident Response Report was developed and a meeting was scheduled to do a debriefing and update the existing Incident Response Plan.

Needless to say the total “cost” of this incident was significant: lost revenue, reputational impact, legal support, third-party incident response costs, etc. In this client’s case there were multiple lessons learned – most notably around improvements to their pre-deployment security testing practices. 

Five good practices that are likely to reduce the likelihood you spend your next weekend responding to an incident are:

  1. Ensure that critical applications/systems undergo appropriate “security certification” prior to being deployed.
  2. Ensure that logging is sufficient to detect and respond to security incidents. Be specific about the log events, amount of time, and log sources (OS/firewall/database/web server/app server) required to respond.
  3. Put a simple log monitoring tool in place, ideally with alerting on a few events that would be indicators of being under attack and/or that a breach has occurred.
  4. Make sure that anyone required to support incident response is aware of their role, contact information is readily distributed, and if necessary, an SLA on their response is in place.
  5. If legal action may be considered, make sure evidentiary protocols are established ahead of time and followed.


Free Download: ISO 27001 Implementation Roadmap

ISO 27001 RoadmapHave no fear – our “roadmap” will guide you, step by step, through the entire ISO 27001 process.

Getting to ISO 27001 certification is a process made up of things you already know – and things you may already be doing!

Is ISO 27001 Right for (Y)our Organization?


Thinking about ISO 27001 Certification? View our free On-Demand ISO 27001 Webinar

  • How to deal with increasing threats
  • How to manage multiple regulatory requirements
  • How to handle client requests for attestation
  • To validate that significant changes did not have unanticipated results

Download: Information Security Attestation Guide

Information Security GuideA Best-Practices Guide to Information Security Attestation

Download our proven Information Security Guide to simplify the process of protecting your data, proving you’re secure and growing your business.

Free Download: ISO 27001 Vendor Selection Toolkit

“ISOOur ISO 27001 Toolkit will help you to select an ISO 27001 consulting firm.
  • Review the Issues Critical to Your Environment
  • "Vet" Vendor Qualifications
  • Compare the Top 3 Vendors
  • Sample RFP Included

Free Download: A Best Practices Guide to Database Security

database security roadmap

Because data is only as secure as the systems & processes it relies on – a holistic approach to data security is essential. This roadmap is not meant to be exhaustive but rather to stimulate the necessary thought process to put you on the path to good data security.

Best Practices for Firing A Network Security Administrator

Firing A Network Security AdministratorWant to know how to fire a Network Admin? Need to know what precautions to take? Firing any employee can be a stressful event. Firing one who has significant knowledge of and privileged access to your Information Technology/Security infrastructure is even more stressful, as the risks are so notable.

Free Whitepaper: Five Best Practices for SIEM


The promise of SIEM is the consolidation of all relevant Security Event Logs from disparate sources into a single unified and normalized data store.

Free Whitepaper: Stop Wasting Money on Penetration Testing


Penetration Testing is most frequently performed to:

  • Substantiate the net effectiveness of a mature control environment
  • Prove to a third party that an environment is secure/trustworthy
  • Quickly assess the security of a less mature control environment (in a sense a technical risk assessment)
  • To validate that significant changes did not have unanticipated results

About the Author:

John Verry (CISA, 27001 Certified Lead Auditor, CCSE, CRISC) is Pivot Point's resident "Security Sherpa". He is lucky enough to spend most of his day helping clients develop a road map to address security, compliance, and attestation requirements.

Add a Comment

Share This