The "RISKY BUSINESS" Blog

My mother always used to say “you should never discuss religion or politics with others”.  As I’m not very knowledgeable in either, nor do they appeal to me very much,  it’s been pretty easy to comply with mom’s guidance. 

Over the last few weeks I’ve learned that there is one more item to add to that list – “Penetration Testing”.   I wrote a blog on Penetration Testing that was intended to stimulate discussion.  The hope was that it would move the conversation forward on an industry subject that sorely needs open and candid conversation that can inch us towards a more standard definition of the same.  Instead, what I got was highly negative feedback that was delivered with a fervor reminiscent of a religious zealot.  The more rationally I attempted to explain my position the more irrational the response – finally I gave up.  My argument was pretty simple – scale the test to ensure that the testing activities are  proportional to the risks the client is looking to validate; that is, controlled to an acceptable level.

 

While I understand the value of a black-box penetration test, ongoing vulnerability research, and writing custom exploit code,  I find it remarkable that there are practitioners that insist that unless a test includes the same – that it is not a penetration test.  To suggest that the right penetration test for the CIA is the same as the right penetration test for a widget manufacturer, ignores basic risk assessment principles.  The cost of the control should not exceed the cost of the risk it mitigates.  Where a compromised server at a widget manufacturer may be a mildly business impacting  nuisance - a compromised server at the CIA may result in thousands of lost lives.  Clearly, the extent and rigor of the testing for the CIA should exceed that of the widget manufacturer.  I have yet to meet the widget manufacturer who wants to protect himself from custom written exploit code – it’s a risk that they are simply willing to accept. 

 

I have been following a similar debate on another blog this week that I think is interesting and illustrates my point.  And no …. I am not either of the folks in the conversation :>)

Had an interesting (and familiar) call with a potential client yesterday regarding a Penetration Testing proposal. 

"We really like your proposal, but honestly, we are trying to figure out why you are so much cheaper than the other firm that we are looking at ... "    

While it is not always true, I generally believe that there is some credence to the axiom, "you get what you pay for".  It was obvious the client was of the same mind - so our notably lower price put us at a disadvantage  We had a rather lengthy discussion about the issue, and as it is a conversation that I have had many times prior (as the higher priced vendor as well), and I have to assume others are experiencing the same issue, I figured it's a conversation worth sharing.

We agreed that there are 3 probable reasons for pricing to be "significantly" different across different vendors:

 

1. Project Approach - Unfortunately, there is really no standard definition for a Penetration Test or Vulnerability Assessment. Very frequently we find notable differences in approach that have a very significant impact on the time required to perform a test.
2. Personnel Cost - The hourly rate of the personnel conducting the testing are dependent upon a number of factors including organizational size, sales/marketing/project management costs, cost of the personnel.
3. Equipment/Materials Cost - Every pen testing organization incurs costs for the tools and equipment (vulnerability assessment tools, laptops, pen testing tools, etc.) that they need to "recapture" across projects.

Project Approach

As we don't want to compare Apples & Pomegranates, it is critical that you really understand how each vendor defines a Vulnerability Assessment/Penetration Test. To help clients understand our definition, we have published very specific details of what each class of testing includes.  In the conversation above we found that both our Vulnerability Assessment and Penetration Test were more comprehensive than our competitors.  However, as our methodology employs statistical sampling (validated by 9 years of testing experience) we were going to actively attack less systems than our competitor.  When we explained our rationale, the client agreed, and was satisfied that our approach would fully achieve their objectives.

Personnel Cost

We run a pretty lean organization.  We don't employ a sales team; we prefer to have our consultants who actually perform or manage the work, work with clients to find the best approach.  We are also lucky that the content on our website, recurring clients, and client referrals provide enough leads that we don't need to spend a lot of money on expensive marketing mechanisms (brochures, trade-shows, etc.).  Because we don't need to pay for salespeople and trade-show exhibits the hourly rate we charge for our consultants' time is often a bit lower than our competitors.

Materials  Cost

We don't use "junior" consultants; the least amount of experience that any of our current consultants has is 11 years. So, we don't need to employ expensive "automated" tools (e.g., Core Impact) to compensate for a lack of experience.  Accordingly, we don't need to pass the ~$40,000 per year per consultant in license costs along to our clients. 

 

So what do patios have in common with penetration testing?  We recently had a patio built in our backyard.  When I collected three proposals, two were very close in price, and the third was almost 40% lower.  I was tempted to dismiss the lowest proposal, but instead asked the contractor to explain the price difference:

 

1. Project Approach: He had structured the proposal assuming that we could make a slight change to the skirting on our current deck that would allow him to use a less expensive retaining wall in one part of the design as it would no longer be visible.
2. Personnel Cost: He explained that as a smaller company they didn't employ separate designers and project managers. This means less "lost" time.  Further, the designer who laid out the project would actually be the person onsite which also ensured that "nothing would be lost in translation".
3. Materials Cost: They were using pavers that were sourced locally (NJ) where both competitors were sourcing them from Canada. The pavers were indistinguishable from each other and we wouldn't have to pay for the cost to ship thousands of pounds of pavers all the way from Canada.

 

Once I was comfortable that I was comparing  "apples and apples" , it was time to "Trust, but Verify".  I met with three references where the work looked great and the owners were all extremely satisfied.

Right now I'm a happy camper - adult beverage in hand, notebook on lap, feet resting on a patio that we are absolutely thrilled with (built by the lowest bidder!).  I guess sometimes you can get more than you pay for!

 

In one of the most remarkable stories involving Information Security I have read in a while ... insurgents used $29 software to monitor the video feeds coming from our drones (and we wonder how Bin Laden is still standing upright?)

To be clear ... drones which cost as much as $12M apiece don't encrypt their video feeds.

So think about this: my son's XBox Online "Modern Warfare" traffic is encrypted but our military's "real warfare" traffic  isn't?  Now that's scary ... 

 

Last week I bemoaned Axe Shower Gel's packaging and noted that we were working on some changes to our Penetration Testing service offerings to better meet our client's assurance objectives.

Over the last 9 years we have found you can generally divide our Penetration Testing clients up into a few broad "stereotypes", clients who:

1. View a penetration test as a necessary evil (e.g., small banks and smaller SAAS providers who conduct them to satisfy a regulatory or customer requirement).

2. Are pretty confident that they have things "screwed down tight" but just want a quick test to make sure.

3. Have a business driver (e.g., regulations, client attestation) and consider penetration testing to be integral to their security program (e.g., larger banks and SAAS providers).

4. Operate in a high threat/high impact environment where penetration testing is viewed as critical (e.g., critical infrastructure, law enforcement, eGovernment).

Recognizing that "one size" doesn't fit all, we have tried to align our Penetration Testing offerings to provide assurance consistent with our clients' varying objectives:

• An Investigative Pen Test - emulates an attacker that doesn't have a lot of time, and doesn't have a lot of tools, and may not even be targeting you specifically. He may stumble upon an interesting portion of your infrastructure during a broader sweep and will leave  relatively quickly if he doesn't find an obvious security problem. Attackers that get in through a blank or default password on an administrative account are Investigative Attackers.

• An Intentioned Pen Test - emulates an attacker that has more time, and a few more tools than the Investigative attacker. More importantly, she has intent. She is targeting you and wants to find a weakness in your network. Attackers that get in by exploiting an unpatched vulnerability in an operating system or network service are Intentioned Attackers.

• A Tenacious Pen Test - emulates an attacker that has time, tools, intent, and determination. She is willing to go the extra mile to make it past your defenses. She may even attempt social engineering to find a way beyond your perimeter defenses. She will do it quietly, though, and take care to go unnoticed. Attackers who convince your help desk to reset an account password for them are Tenacious Attackers.

• A Zealous Pen Test - The primary difference between a Tenacious Attacker and a Zealous Attacker is that a Zealous Attacker won't try to stay under the radar. He will do things that get noticed. He may even intentionally disable access to services to see what happens. More than intent and determination, he has a belief that he needs to breech or damage your systems, one way or another. If he has any worries about covering his tracks, they are secondary to the success of the attack itself. Attackers who crash your mail server and deface your website are Zealous Attackers.

Just as packaging matters when it comes to shower gel, we believe it also matters when it comes to security testing. So choose wisely, and dispense exactly what you need. Remember, "one size does not fit all!"
 

With apologies to Steven Levett (Super Freakonomics) ...

Just spent the finer part of my afternoon in a meeting further refining our Penetration Testing services to better align with the differing objectives of our diverse client base. Many of our banking clients want a penetration test for one reason (and one reason only), to hand to the FDIC/OTS/SEC auditor to satisfy their requirements. On the other end of the spectrum many of our government clients want a high level of assurance that a knowledgeable and intentioned malicious individual with an extended time window can't gain access to certain facilities/systems/data. The challenge is that as an industry we have failed to "package" Penetration Testing in a manner that is relevant and easily consumable. Only intelligent and innovative consumers can wade through the idiosyncrasies of the varying ill-defined activities to arrive at a solution that is optimized to their requirement.

I have overcome a similar challenge to the one I just raised, in my shower. I have recently grown fond of the Axe shower gel my son left in our shower. The problem is that the packaging is almost comical. It's in an oversized bottle that becomes as slippery as a watermelon seed the minute that it gets wet. It's virtually impossible to squeeze out anything but 3X the amount of gel that you need to cleanse your entire body. Worse, you lose most of down the drain before you can leverage it across more than a single leg and midriff. So, back to the bottle, which is impossible to pick up now that your hands are soapy, and repeat the folly.

Problem solved. I bought a $1.49 foaming hand soap product, dumped the hand soap, and watered down the shower gel 5 to 1.  Life is good: oodles of perfectly fluffy shower gel foam with nary a drop wasted.

So why is it that the multi-billion dollar consumer product companies haven't properly packaged shower gel and we have to live with a one-size-fits-all ill-packaged offering?  My guess is they don't know enough to know, they don't actually use shower gel themselves, they make more money with a poorly designed product, and they don't think we are smart enough to differentiate product offerings.

I wish there was a $1.49 (Walmart) solution to the penetration testing packaging as well. Unfortunately, it's not quite that easy,  but we are confident that we have solved the problem and that our clients are smart enough to differentiate the product offerings and ultimately benefit from our approach. We'll be rolling out our new services over the next few weeks ... so please stop back.

---

For further information, check out our white paper – available for download – “Stop Wasting Money on Penetration Testing" - on our Penetration Testing resource page.

... but I don't feel fine (see REM if you don't get the reference).

Over the last few years significant changes have taken place in the vulnerability discovery space. In the "old days" a vulnerability researcher would discover a vulnerability, report it to the vendor, wait an "acceptable" period of time (for the vendor to (hopefully) issue a patch) and then publically publish their work (and "exploit code").

Fast forward to today, the picture is radically different. Vulnerabilities are bought and sold by both hackers and security companies, often on the shady side of the internet. To anyone in the InfoSec field this is old news; So how does this change Penetration Testing?

Network Penetration Testing is a form of substantiative testing, with three predominant objectives:

• Determine the probability that a system vulnerability can be exploited
• If so, determine the impact that the exploit would have on the entity?
• If so, "shock" management into action by demonstrating the impact in a non-ambiguous manner.

With the number of exploits being "publicly" disclosed dwindling (because the vulnerabilities are being purchased by a security company or other hackers) the amount of "safe" exploit code available to an ethical hacker is dwindling as well. Without exploit code the ability to achieve those three objectives  is radically reduced. Ethical Hackers looking to still "exploit" critical vulnerabilities have two choices:

• Leverage potentially dangerous exploit code acquired from sites like millw0rm. (However, the exploit may actually contain additional malicious code leaving your client's machine compromised)
• License a commercial automated penetration testing application (e.g., Core Impact, Canvas) that is buying much of the exploit code on the market).
  

Frankly, I don't think either option is all that good.

The first encumbers both the ethical hacker and the client with significant risk. Questionable exploit code could contain malicious content resulting in an activity intended to improve the security posture of the network, significantly reducing it.

The latter adds considerable cost, reduces the likelihood that multiple "lower risk" vulnerabilities that can yield access (i.e., leapfrogging or privilege escalation) will be identified (as experience is replaced by push button automation), and reduces/distorts the ability to truly measure probability and/or impact.

We all (clients and ethical hackers alike) have some tough decisions to make. It's the end of network penetration testing as we know it.

---

To help put it all in perspective, be sure to review – “Stop Wasting Money on Penetration Testing" - on our Pen Testing resource page. You'll find valuable tips to help you properly determine your needs.

On first blush providing credentials to a tiger team conducting penetration tests sounds like giving the fox a key to the chicken coop.  However, there are many cases where it can provide significant value.  For example; you want to assess whether an authenticated user (network or application) can escalate privilege.  Another great place to use credentials is during the Vulnerability Assessment phase of a network Penetration Test.

A network vulnerability scan is essentially a "best effort".  The three predominant challenges to Vulnerability Assessments are;

• The scanner assumes that the host it is interrogating is "trustworthy" (the level of trust is usually adjustable) and bases its assumptions as to the services, versions, and vulnerabilities on the answers it receives. The false positives we are all familiar with are assumptions gone awry.
• The scanner cannot directly assess many important system settings, for example the password policies' complexity setting or the system audit policies event logging settings.
• Packet filtering "devices" in the network path between the scanner and the device (e.g., firewalls, load balancers, routers, network IPs, Host-based IPs) may respond on behalf of the device, providing incorrect data and a false sense of security.

The key benefit to running the vulnerability scan with administrative level credentials is that it allows the scanner to directly assess the system's configuration rather than guess it based on the answers it received.  This not only provides a greater quantity of, and more accurate, information, but it opens up the possibility of using the vulnerability assessment as a compliance check against relevant standards (e.g., PCI, Center for Internet Security, or organization specific).  The last benefit is that a vulnerability  scan with credentials avoids most of the problems encountered with packet filtering devices in the path as the scan is essentially local and authorized.

In a future blog we will look at one of the other unique benefits of running a credentialed scan - running a content scan on the hosts at the same time to determine whether sensitive data (e.g., credit card, medical, identity theft, intellectual property) exists on the systems in violation of policy.

---

Don’t miss our video, from the Master Assurance Series on Network Vulnerability Assessment: Key Decision Points, to help guide you through this valuable tool in the information security arsenal! Find it and more on our Penetration Testing resource page.

It wouldn't be a stretch to say that Windows 2000 and all the Windows Iterations before it were insecure by default and that Windows 2003 and forward are secure by default.  So why is it that Windows 2003 may make your environment more secure while at the same time it makes your environment less trustworthy?

I believe there are two predominant reasons:

1. As we become more confident in Windows security we become less vigilant in our due diligence to validate that Windows is actually operating as intended.
2. Even when we are vigilant, a network penetration test, the defacto "gold standard" for substantiative testing that an environment as a whole is operating is intended is no longer sufficient.

Windows 2003 is significantly more secure by default than Windows 2000, however, it is not immune to less than optimal configuration and vulnerability patch management issues.  Further, there are often other "less important" (and likely less secure) older platforms, network devices, or older applications that may denigrate the security (and trustworthiness) of the environment as a whole.  So Trust the environment, but remain diligent and verify by appropriate activities.

That segues directly to point two, the "appropriate" activities have changed. As an environment becomes more technically secure there is an implicit and incorrect assumption that the probability that it will be "hacked" definitely goes down.  In fact, I would argue that if the attacker is intentioned, that it does not go down at all.  An intentioned attacker will just alter his attack vector to seek the "new" weakest link (e.g., Social Engineering).   There is an old adage that professionals don't hack systems they hack people (think Kevin Mitnick).

So as your environment grows more secure the tests that you use to measure its security/trustworthiness need to change as well. Many of our more risk averse clients are adding social engineering and/or physical penetration testing into their "verify" activities.  Should you?

---

Our “slidepaper” on Network Vulnerability Assessment: Key Decision Points, will help broaden your knowledge on VAPT. Find this and more on our Penetration Testing resource page.

… the whole world looks like a nail. I think this is especially true in the world of Security Assessment firms. Where we have the greatest expertise, we recognize the greatest risk and either explicitly or implicitly we communicate those risks to our clients. Over the last few years we have been working hard to broaden our skill-set (e.g., networking, application development, system administration, databases) to encompass as many areas of expertise as possible so that we are not guilty of spending all of our time looking for “nails” (the vulnerabilities we know how to recognize/exploit (as opposed to those that put them at the most risk)). A recent engagement was really eye opening and re-enforced to me how much of information security risk is non-technical in nature.

During a broad security assessment for a client in the governmental sector, we were asked to assess the physical security of three buildings, two of which would be considered high security. Physical Security and Social Engineering are two areas that our skill set is rapidly growing in – but I would not yet consider it one of our core practice areas. This is due to, in large part, the fact that the vast majority of our customers either do not consider it enough of a risk to check OR acknowledge that it is a problem. Subsequently, they believe testing its efficacy does not make sense to them. I have rarely argued this point with a client, however, that is about to change.

Based on our initial reconnaissance of the client’s site I thought we had a pretty good chance of getting into the building that was not highly secured. Entry through the front door was not likely, as you needed to be “buzzed” through two doors by a security guard, with the second requiring you to be badged and escorted. However, there was a smoker’s door on one side of the building, and a back door that was used by employees as a primary entrance if they were leaving or returning from another building on the campus. The last three doors mentioned were all monitored by video cameras, but based on previous experiences; we felt that we could create a fake employee badge of sufficient quality that a “tailgating” attack would be successful. As expected, a tailgating attack against the backdoor was successful (after a few clumsy attempts). Once inside, one of our social engineers gained access to sensitive data via empty cubicles, unlocked computers, unsecured BlackBerry’s, and network printers/copiers.

Where the testing got interesting was the attempts on the second building. This building also had a security guard at the front door and it had a side door that was used as an emergency exit. The first floor of the building was considered a “secure” area while the second floor, which housed the organization’s “command center”, was considered “highly secure.” During our preliminary reconnaissance we had noted a construction crew renovating a bathroom area. Slipping into a nearby phone booth one of our social engineers emerged as an appropriately badged township building inspector – complete with inspection paperwork, digital camera, and flashlight. The other social engineer, having already penetrated the first building with his fake badge, acted as the inspectors “escort” which we had learned was their policy.

As a construction worker emerged from the side door pushing a wheelbarrow full of debris - - the tandem entered through the door and into the construction area. After briefly “inspecting” the construction area – they slipped through a second door and into the “secured” first floor area. The “inspector” continued his inspection while his escort “cubicle surfed” his way to a wealth of sensitive data.

However, our goal was to gain access to the second floor. It became apparent that the only way to the second floor was to enter through the lobby that we had bypassed to get into the building. The lobby housed a staircase and elevator to the second floor, however both were key card secured and physically observed by the security guard. Needless to say they were concerned that the guard would realize that neither this “inspector” nor this “employee” entered through the front door of the facility. Not to worry – the guard was more interested in Google News than our social engineers. However, as they did not have a key card to open the stairwell or elevator – they were out of options. As they stalled by feigning note-taking, someone exited through the elevator. Neither the guard nor the employee noticed as they slid into the elevator and upstairs into the control center. The control center was “abuzz” with an issue – so our team was barely noted as they slipped into offices, conference rooms, and observed critical systems in the command and control area. After documenting the access with the camera in their iPhones – the team left via the elevator with a wave to the security guard. Could we go three for three?

The third building was a very secure underground data center. We had only been able to perform very limited reconnaissance of the building due to the high level of security/observation associated with it. On two previous walk bys we had only noted a single entrance which was key carded, man-trapped, and was directly observed by at least three security guards (hidden behind darkened glass). The entrance was also monitored by at least one video camera and we had been warned that the entrance was protected by a tailgating sensor. Lacking any “viable” avenue of attack we were reduced to trying a basic tailgating attack with the expectation of being immediately caught by a guard, the tailgating detector, an alert employee, or some combination of the three.

We took a break on a bench where we could observe the parking lot and building entrance. When an employee emerged from his car, pizza in hand, and headed towards the data center - our social engineers timed their arrival at the data center door to coincide with his. Concentrating more on keeping his lunch upright, than in complying with the organizations anti-tailgating policies, our team was able to tailgate him through the first two doors. The anti-tailgating device triggered, but to our surprise, the security guards did not appear to notice. The initial sense of accomplishment quickly dimmed as they did not reach the next door in time to catch it before it closed. They now found themselves in a short hallway with only two cardkey secured doors that would yield further access. Worse, they were in full view of the security guards separated by a few inches of bullet-proof glass. Pausing to make a phone call was the only stalling option. It paid dividends as a maintenance worker emerged from the key carded stairwell and held the door as they slithered down the stairs.

Two full flights of stairs later we were on the data center level which was further divided into four secured areas, all requiring card key and biometric (handprint) authentication. The level was also fully covered by security cameras so our “inspection” continued as we tried to determine our next line of attack. As a worker emerged from one of the biometric enabled doors he cast a wary eye at us and pushed the door closed behind him. After 10 full minutes, and several more wary employees’ stares, we assumed we were running out of time before security came calling. Emboldened, we rang the video enabled buzzer at one of the doors and were challenged by the person, not visible to us, on the other end of the line.

Our “employee” held his badge to the camera and explained his need for access, which related to the build out of another secure data center in a remote area in the Rocky Mountains. He was at the main facility to see how they “did things” here so they could do the same there. The conversation was peppered with enough names we had gathered via other reconnaissance activities (including an org chart we had photocopied in the “command center”) and supposed “confidential” knowledge of the Rocky Mountain data center (we had overheard that tidbit at the local Starbucks) that we sounded legitimate. The last hurdle was the inspector. Our “employee” explained that although the inspector was a local inspector, he was also certified in the building codes for the Rocky Mountains as he previously lived there. After a slight pause, the door buzzed and we were in the data center. As per the rules of our engagement … we placed a business card on handle of one of the server racks and immediately exited the facility.

Suddenly I have a different hammer in my tool belt – and the entire world is a new type of nail. If we can gain access to two highly secured facilities in a single day, what is the likelihood that the security that we find in corporate America or local government facilities is sufficient to deter a determined individual? Once inside the perimeter of a building – the millions of dollars spent on information security is quickly bypassed by simple “non-technical” measures. Stealing a laptop or using a physical keystroke logger doesn’t require much mental horsepower.

So the next time we sit down to plan a security assessment together – you will have to forgive me for arguing a bit when you tell me that physical security and social engineering are out of scope.

---

Don’t miss our white paper – available for download – “Stop Wasting Money on Penetration Testing" - on our Penetration Testing resource page.