Penetration Testing considered Harmful Today

(This talk was given at 44Con in London (2010))
Brief details on it can be found here.

The point of the next four slides is merely to establish some sort of credibility. Essentially it’s to try and establish that when I talk about pen testing, I do actually have some background in it.

This is the central thesis of the talk, and I’ll try to explain why I believe this is true..

 In 2010 we wrote a blog post titled “Our upcoming security apocalypse“.

The post contends that we are heading for a crisis in information security and that the crisis is a non technical one: a crisis of confidence.

We have entered a new phase, but are using the same old excuses. (management buy-in, etc)
Looking around the room, we have well paying jobs and midweek conferences at swanky hotels. It’s not like we are part timers anymore. Management have already bought in.

 At some point, we declared “mission accomplished”.

It seemed like we had for the most part won (until lulzsec and anon proved very clearly otherwise).

Our past (where infosec wasn’t given any attention or management buy in) created a strange kind of communication gap, where today boards are paying for infosec teams and 3rd party products and analysts but just have no idea how little control we (infosec guys) have over things.

Most board members know that they occasionally hear whining from security folks (they probably hear whining from finance and payroll too) but most don’t know that we are just hoping the attacks dont happen on our watch.

Most boards believe they are doing something, because they pay us (and others like us) and so they probably think they have things under control. I willing to bet that if you went to a board member and told them what their true exposure is (in a way that only we truly understand it) most board members would have a fit. If they knew that despite all that they pay us, their financials are one zero day away from being published on the Internet, they would fire us.

It has many shades of the global financial crisis. Lots of money changing hands, lots of people getting rich, some surface level checks and balances but ultimately nobody is really saying the truth that needs to be said. Nobody admits how truly brittle it all is.

These 2 slides are stolen from a Dan Geer talk (which explains why they are clearly cleverer sounding than the others) .

Dr Geer points out that fallacy at play is exactly what Taleb warned about in the Black Swan..

That the fact that bad stuff hasn’t happened in the past is not a guarantee that it won’t happen in the future, it’s a guarantee that it will be more surprising when it does happen.

A simple quick test here follows:
If we imagine the most important person at your organization. (possibly) The CFO with access to all of the important financials. Would you be willing to bet, that you have that guy protected?
Would you be willing to bet that he won’t get owned tomorrow?

Pen testing today is big business

I wanted to quickly cover things I won’t spend too much time on:
– I’m not talking about tests where scopes are completely neutered: “only test these 2 IPs”. In those cases its pretty clear that the test doesn’t illustrate risk reasonably
– I’m also not talking about the times when you hired completely useless pen testers. That’s a different problem
– for this talk I’m also ignoring the actual risk introduced by pen testers (but this is probably worth touching on)

Zf0 published home directories and mail spools of a bunch of famous security folks but for the most part we look and that and know that “there it for the grace of god, go I”.
Ie that we too could have been taken, and our home directories too could quite possibly have had client reports and customer data.

But I’m not even talking about those.. I’m saying even with good scope, and good testers, pen testing probably shouldn’t be your first thought.

One caveat here: if you have done a lot to secure your network, and are looking for a binary answer: “is my installation take-able ?” then sure go ahead and have the pen-test done, but if that’s not the question you are asking, then it probably isn’t for you. If you are asking how should I make myself more secure? Then pen testing isn’t the answer

Quick poll #2

How many ppl have been involved in a pen test in the past 2 years ?
(answer – everyone)

How many good 0days wold I need to go from external, to your crown jewels?

I pitched the question to a bunch of smart friends in the industry. I went in saying “I guess that with 3 good bugs, I’ll get to the jewels”

Almost everyone came back with “1”.

Ie that in most cases, despite being pen tested for years, a single 0day would translate to virtually guaranteed access to the crown jewels.

This one point kinda drives my whole talk.

Ie, if we are spending millions on security, but are still one zero day away from our jewels being lost, then something is very Very wrong

This isn’t just about 0day, but 0day clearly needs discussion.

0day still divides testers.

Those with it claim that a pen test is incomplete without it, and those without it claim they don’t need it because they break in anyway.

Let’s examine the 2nd group first.

They say: “I never need 0day, and I still manage to break in every time, so why does it matter?”

Well.. It matters because the point was not just to see how you would break in, the point was to see how _anyone_ would break in.

Do real attackers use 0day?

Many moons ago we used to argue if testing security at all mattered. We were doing complex attacks against networks and apps it it didnt seem as if real attacks were happening at all. Were we the only credible threat?

So then you see the many public attacks that happened and it answers “do attacks happen” and the questions turn to “does 0day happen?”

Operation Aurora was interesting for this. Google came out and said: “we are changing the way we deal with China”.

Something like 38 companies were owned at around the same time (even though Google were the only guys to ‘fess up).

All of these massive companies were essentially taken by 0day.

Now theres a red herring here.

When people talk about aurora. They mention that it targeted IE6, but this ignores that the bug existed in modern browsers too.
This makes the attack even more targeted since the attackers clearly did target profiling first.

The other incident that was interesting in 2011 was HBGary vs Anonymous.

If we ignore the drama, the HBGary mail spools were a treasure trove for a peak behind the curtain of the Military Industrial complex.

0days were being actively pitched and sold and we would be foolish to think that they were being bought for academic purposes.

0day at those prices seems steep, so we can look elsewhere to try to get a price for 0day.

Charlie miller wrote a really interesting paper on is adventures selling a samba bug to the government.

He eventually sold it for 50k despite  initially being promised 80k

He also talks of another bug that was bargained down to 8k (and was actually killed before he sold it). This takes 0day to around the 10k mark.

(Then we have pwn2own where 0day gets published for 10k and a laptop.)

 His talk on “attacker maths” figures the going price between 5k & 10k

Pwn2own gives us other valuable lessons.

People feel nervous during the period between “we have seen working exploits for our browser” and “browser vendor has released a patch”.
This is natural, but so clearly misguided. I.e those exploits were not only possible because they were demonstrated, they were always possible.
Charlie miller mentions how he once sat on a bug for a year to win the following years pwn2own.

It’s clear that browsers have become the weakest link on our machines. Browsers give such fine grained control of memory space that no matter what is level mitigations we currently have, browsers enable an attacker to work around them.

Early pwn2own regarded browser hacks as 2nd class citizens. They grabbed a smaller prize. Today the contest is all about browser hacks.

Now this is interesting because despite the fact that the browser is the place to play, you seldom see it in pen test reports.

Reports will still be looking at the results of server vulnerability scans.

No mentions of browsers at all even though they are the most vulnerable pieces of software we are running.

Test #3

What’s the current version of flash? Or the current version of the java runtime that should be on our machines?

We are still thinking about ms0- bugs because that’s still how we do our pen tests.

We see sublime exploits come out through the jailbreak community from people who were previously unknown in infosec.

We never heard of them, and because they wanted to jailbreak their phones they have built exquisite exploits that manage to go from a web browser click to modified kernels on our handsets. It’s a good indication that the talent pool around exploitation is much deeper than most realize.

All of it just makes it clear that if we are ignoring 0day on our tests, we are ignoring reality.

Now, back to the guys who say that the pen test is all about the 0day.

Dave Aitel says a 0day takes about 450 hours to create.

This is too long to build into engagements, and I’m saying that even if you do, you don’t address the problem completely.

It becomes an arms race.

A tester tests you with the 0day he has, but what about the 0day he doesn’t?

If we just changed the question, we can come up with better solutions.

“How can we allow the CEO to surf securely without risking everything? Let him surf from a device, let him use a terminal solution”

We can come up with many solutions if we ask the right questions.

We were supposed to be emulating attackers but these days we just emulate other pen testers.

We used to always say “how you could tell the real pen testers by their tool chains” ie. its not just nmap and nessus.

I used to happily quote the following as proof:

In 2002 we showed how to use DNS exfiltration with SQLInjection.

In 2007 we released “squeeza” which had DNS as a possible extrusion channel.

squeeza managed to work without elevated priveleges and was able to transfer full binaries over SQLInjection over DNS (or through timing or error messages).

Our 2008 tool ReDuh essentially allowed clean TCP tunneling over HTTP/HTTPs.

We then showed how we could use SQL Injection to build binaries in SQL server.

Combining this meant that we could use SQL injection to build a binary version of squueza that would essentially allow us to tunnel arbitrary TCP traffic through SQL Injection.

This all made us feel very clever..

But this is a classic “draining the swamp” situation.

The story usually goes that we are sent to drain a swamp, but cant because the swamp is full of alligators. So we start fighting alligators, and we get better and better at fighting alligators. Soon we have alligator-o-matic-2000 and we have certified alligator fighters but we forget that the original plan was to drain the swamp.

We keep bumping into this problem of missing entire avenues of attack.

So you could have pen-testers who come in an 0day your network (and you will get your report)
or you could get in testers who use sql injection and password re-use (and you will get your report)

or you could get in testers who would use a spear phishing attack (and you will get your results)

All 3 are valid, but all miss the other avenues

So its possible to have a pen-test, be happy with the results, but still be perfectly ownable.

This brings us to the next problem, which is that Pen test companies today almost perfectly demonstrate “the market for lemons”

In the original paper, we saw a market where people sold used cars that could be either a lemon (an inferior car) or a cherry.

The asymmetry of information means that buyers dont know which cars are lemons and which cars are cherries.

So buyers offer a price that averages their expectation.

This means that over time, cherry owners will stop putting cherries on the market.

The market will be dominated by lemons which leads to market failure.

The main cause of this is the asymmetry of information and matches almost perfectly the problem we have in infosec.

So. We have incomplete coverage;
We avoid the 0day question;
We focus on cool attacks that might not matter and we increasingly we are building a market for lemons.

So why is pen-testing so popular?

These days its easy to sell. Customers know about it and have budgeted for it.
Pen testing also feels like we are doing something. Its like busy-work.

Pen testing always gets a result. Pen testers look down on the IDS guys who never caught a 0day in the wild, because everytime we step out to do a pen-test we have a real result, but its not a result that matters if the customer is still just one 0day away from losing.

I think what we have here is like a hill climbing problem.
With optimization problems, Hill climbing refers to when you pick a solution, and then you keep iterating to make the solution better and better.
Pen testing started as a candidate solution, and we kept making it better and better. We get to the point now where we are really good at pen-testing, but the classic hill climbing problem is that you get stuck on a local optimum.
I.e. if we started on another path, we could have gotten to a better answer, but maybe we have optimized as much as we can on this one.

PTES is a noble goal, but is already massive and adoption is slow. I’m not particularly hopeful.

Adam Shostacks EOP game is a good idea, which is aimed at teaching developers how to threat model but doesn’t quite extend.

We could consider pen-testing by visio.. ie. looking at the diagrams and saying “This can be taken and then we could take that.”

Of course whats on visio isnt what the network always is.

We cant remove operators since there is operator skill that cant be discounted.

I suspect we need to move to a less adversarial model of testing.

We made this jump in app testing.

Nessus used to try to do it all remotely, till Renaud and the Nessus team figured hold on.. We are the good guys.
Lets add credentials, let the scanner log in and collect info cleanly.

We cant however do this completely friendly and paper based, because then you miss unexpected things that we discover on the test.

Taking a low value box that just happens to have passwords to everything on the network.

I think we can try gamification..

Not the type thats currently making headlines

I say we sart by figuring who our opponents are, and what they look like.

We consider what it is that we are actually protecting.

If we can categorize what our opponents budget is, we can agree on a number of 0days he is able to buy when going after us.

So the tester gets to say: “I’m using this card, give me system on that server”

This addresses the issue of limiting you to my 0day arsenal.

It means we still get to stumble onto password caches and other lucky finds.

If we are printing cards…

It also makes sense to have unlimited pivot and tunnel cards. ie. what we did with ReDuh, what metasploit does with meterpreter, are all established facts. We all know that if i can take a box, i can pivot on that box. Its silly for the test to be limited to my skill to pivot on the box, when the real attacker might not have that limit. Ie. we need to get past the point where i need to prove to you that im cool by pivoting. We should be grown up enough to just move our shell and do our testing from there..

It probably will make pen tests less fun, but we cant keep getting paid to have fun.

Sometimes customers will say it cant be done, and we can then spend the time proving it to them.

So we have to finally ask if we really need to change?

I obviously think we desperately have to.

When lockheed and RSA and SONY got hacked pen-testers across twitter were subtly shilling their wares going “when last did they have a pen-test?”

The truth is that any one of our customers could have gotten taken like Sony and Lockheed. We could have tested them last month.

We are simply not helping matters, and are rapidly approaching being in the same boat as Anti Virus.

Anti Virus didnt start off being lame.

They started off trying to solve the problem. They just carried on optimizing their niche solution.  Pen-testing is now on this same route.

We’ve got guys willing to pay us to do it so we keep doing it and they keep paying us.

So if you sell pen-tests, lets make sure customers really need it.

We need to bring integrity back so we can sell people what they need instead of what they think they want.

If you buy pentests, you should question anyone willing to sell you a one.

If you are smart, lets start thinking about how we can do a reset here.

The ray of sunshine with resets is that a reset is exactly how you address the local maxima problem.
ie. you periodically reset, start on another hill and see if that gives you better results.
Right now, we desperately need a reset.