Penetration Testing considered harmful today


Early last year we presented at 44con with a talk titled: "Penetration Testing considered harmful today".

44con have just released the video so we figured it was worth a quick recap (for anyone not willing to tolerate the whiny voice!)

The original slides (in PDF) are available (here)

The central thesis of the talk is that penetration testing has established itself as a necessary activity for securing a network and is now pushed forward by a multi million dollar industry despite the clear signs that it is not helping all that much. (Read the annotated slides here)

Watch the video here:


14 comments :

  1. The is a very great talk; a necessary talk; a talk that everyone in this field should hear.

    I find the conclusions in the talk to be unsatisfying (from my perspective at HP, which is where I work). I'm not going to say that more penetration-testing is necessary or even that you need to buy anyone's products per se, though. I'm not here to shill, but instead to attempt to say profound things.

    I think there is a lot more to be said about appsec risk management and penetration-testing-focused/oriented risk management, which is almost the point that you get to in your talk. In your view, it's a balance between a pen-test "Visio" and a playing-card game. Or worse, PTES. I'm not ready to jump to these conclusions without evidence, although I agree that it is likely more sound that what we already do in the pen-testing industry.

    My view of the "new" pen-testing industry is one of simplicity: "Trust, but Verify". The best way to drive costs down in the verification process is to:

    1) Standardize on 'secure-by-default' application, data, and infrastructure components
    2) Penalize cutting/bleeding-edge and legacy (i.e. non-'secure-by-default') components by compartmentalizing/segregating them to a nearly non-functional state
    3) Reduce the number of components to the smallest possible amount

    Verification costs a lot of money, but it must be done. Another avenue of verification cost reduction is proper maintenance of coordinated disclosure through 'outside-bug-reporter-friendly' incident response teams (at all layers of arch: app and data, too!) and bug bounty programs.

    "Knowing is half the battle" -- GI Joe wisdom that should be part of any information security and risk management program. Add the necessary intelligence. Add total intelligence, and especially add TECHINT (or technical intelligence, aka the study of one's adversaries' weapons and resources).

    Proper risk communication from subject-matter-experts to decision-makers should evolve modern information security and risk management programs to evidence-based conclusions. When it doesn't, then there is probably an issue with the core business decision-making process, which is often a greater risk to information assets than either one 0day or even a thousand.

    ReplyDelete
  2. Hi Andre,

    > The is a very great talk; a necessary talk; a talk that everyone in this field should hear.

    Thanks a lot, thats awesomely flattering..

    I agree re: conclusions.. In truth the main thrust of the talk is to point out that we need a reset, and to try to justify why i think we do.
    I offer some (possible) solutions, but im fundamentally saying: "we need more ppl thinking about this"

    "Trust, but verify" is gold and i think "TECHINT" will start making its way more and more into mature corps. All in all.. i think there are important conversations to have, and more deep thinking to be done..

    ReplyDelete
  3. Great talk. Very thought provoking points. Not sure I agree with ALL of them, but many are absolutely valid. Thanks for raising the bar in our profession by promoting this dialogue.

    ReplyDelete
    Replies
    1. Hi Jeff.

      Thanks. I'm pleased that it didn't take too much of a beating, because I genuinely think the dialogue is necessary.

      Thanks for the kind words.. :>

      Delete
  4. Hi

    Very thought provoking presentation. I have been going back and forth with whether I agree or disagree with the article! Your presentation brings to the forefront a lot of problems (which needed to be done) but doesn't necessarily provide any great solution.

    The problem is of zero day's and your explanation and description really hits home the problem of just doing a standard pen test. However, information security is all about risk appetite and other great jazzy terms like risk tolerance and risk avoidance and risk acceptance. would it be too crazy to say that a lot of firms just accept the risk of 0-days and realise that if a hacker wanted to get to the crown jewels by either buying a 0-day or finding one themselves then they have accepted that risk.

    I guess one point I agree with you is that what use is the pentest nowdays? It probably is more a feel good factor for IT security managers, and at best a defense at script kiddies. Is that a great situation for IT security to be in, probably not.

    Another interesting point you bring up is browser vulnerabilities - these are definitely an issue, but I guess their place is not necessarily the pen test report. I guess it comes down to threat modelling again, and determining what, where are your major threats, and if one is a weakness in a browser then that needs to be classed as a threat and treated like any other threat.

    All in all, interesting article with a lot to think about.

    Good work.

    DR

    ReplyDelete
    Replies
    1. Thanks DR. I think if it get us all thinking about it again, then it has served its purpose.

      Delete
  5. One thing that is left out of the talk is the business 0-day issues that pentest simply does not have time to address. Any sufficiently curious internal user who has put time into learning internal environment can and will be superior in results to majority of pentest teams.

    ReplyDelete
  6. yes. Not sure who was saying "mission accomplished". Not me. There were plenty who were saying that there is no need for tech analysis of risk any more because it suited their agenda. This is the real reason for the mess we're in today. Most issues can be traced back to this.

    re: boards. Growing numbers of them suspect problems because they see the lack of confidence in their CISO. They don't know what to do about it though. None of them have really been well advised to be honest.

    I agree pen testing, even under ideal conditions, should feature quite low in the whole strategy, but for different reasons. Not that i disagree with the reasons given, just I see it in a slightly different way.

    Companies can use zero days in testing but do it in a balanced way...e.g. first zero day compromises the listening service which doesn't give us root. then the tester may have a zero day for a setuid root binary..so, easy, game over. However, they could go about privilege escalation by other means so returning more value for the client paying the bill.

    I cover some of these issues in Chapter 7 of my book, Security De-engineering.

    good stuff re: extrusion.

    Overall, good presentation - refreshing aspects on pen testing.

    ReplyDelete
  7. Just read this, which i think is your point exactly!
    http://www.forbes.com/sites/andygreenberg/2012/03/21/meet-the-hackers-who-sell-spies-the-tools-to-crack-your-pc-and-get-paid-six-figure-fees/

    ReplyDelete
    Replies
    1. The timing was perfect! (thanks for the link)

      Delete
  8. Thanks Hannah and Matthew. After seeing your post on here i contacted nightwatch366@gmail.com and then i actually thought he's among them scammers but he did my job within 24hrs he's work is cheap and secured without trace.
    he is sure the best when it comes to hacking services like:
    1. School Grades Change
    2. Drivers License
    3. Hack email
    4. Database hack
    5. Facebook hack, Whatsapp hack,
    6. Hack Call Logs,
    7. Retrieve messages, deleted data and recovery of messages on cell phone and many more.
    8. Do you need to check on your partner’s sincerity,employee’s honesty.

    You can also contact nightwatch366@gmail.com and thank me later.

    ReplyDelete
  9. Are you in need of best hacking services in the world? contact hacking solution temple They guarantee clients with affordable and reliable hacking services includes....

    *School grade hack,
    *hack into email accounts,
    *all social media accounts,
    *school database to clear or change grades,
    *Retrieval of lost documents
    *DUIs
    *company records and systems
    *bank accounts
    *clearing bad driving and criminal records,
    + credit score hack,
    + Monitor your partners phone,
    CONTACT: hacksolutiontemple@cyberservices.com
    For reliable and guaranteed result

    ReplyDelete
  10. For the very first time in my life, I was able to meet a real hacking team. They completed a secret hack for my company in 48 hours and we were able to protect a lot of information and their charges are reasonable.
    You just need to try them out and you won't go anywhere else as far as hacking is concerned.

    Mobile Phone Hack, Whatsapp Hack, Instagram Hack, Snapchat, Spyware, RAT, Gmail Hack, Yahoomail Hack, Hotmail Hack, Company Email Hack, Website Database Hack, Educational Portal, Credit Records, Job protection, Record Cleaning, iPhone 5, iPhone5s, iPhone 6, iPhone 6s, iPhone 7 Hack, Samsung Hack, Linux Hacking, Mac Hacking, ios Hacking, WIndows Hacking, All kind of phone hacks, and any kind of hack. You can also find stolen gadgets, or retrieve hacked phones, you can also run a check on your mobile to confirm if you are been spyed on.

    24/7 Support

    Visit ETHICALHACKERS009@GMAIL.COM OR TEXT/CALL 1 669 225 2032

    ReplyDelete
  11. HEY GUYS,CONTACT GHOSTHACKER2351@GMAIL.COM FOR ALL TYPES OF HACKING JOBS(CELLPHONE HACK,CHANGE OF GRADE,BOOST YOUR CREIT SCORE).IF YOU ARE INTRESTED TELL HIM I REFERRED YOU.

    ReplyDelete