There’s been a radical shift in the assessment industry over the last couple of years. We’ve all probably heard that Artificial Intelligence, Machine Learning, User and Entity Behavioral Analysis, Analytics, Detection and Response tools, etc., are advancing and improving defensive postures.  According to Gartner, annual spending on defensive security technology will exceed $82 Billion dollars in 2018, growing to $110 Billion by 2022.  With that kind of spending on technologies like these as well as an increase in maturity for many organizations, penetration testing has gotten more difficult...That is if your organization is looking to reduce risk with the knowledge they were meant to provide. Cruddy pen tests are dead.   Really? ...Nah, not so much.  I wanted to write the “(Bad) pen tests are dead.  Long live (great) pen tests!” line, but the fact is, bad penetration tests are still alive and well. This clearly makes it that much more difficult for clients to sift through vendor pitches and proposals.   And ‘cruddy’ now means not just the “running a vulnerability scan with validation” pen test that many security companies do. In today’s world, scanning with cut-and-paste findings is almost a complete joke and certainly a waste of money.  No, what we’re seeing first hand is that even those well-intentioned folks using “manual techniques” are likely not keeping up to give you proper value. Bad intel kills!   Advanced research is critical. The ecosystem surrounding vulnerability research has been affected by the need to deal with lengthy, targeted attacks.  Many of those attacks now come due to zero-day exploits.  Advanced research allows penetration testers to anticipate the exploit landscape and craft solutions in advance appropriately.   Thus, in the current threat environment, if researchers can’t intelligently pinpoint and address system flaws, they will likely be exploited by cybercriminals.  These are not simply software flaws, but also Tactics, Techniques and Procedural (TTP) flaws that can hit various systems or users that are beyond a vendor’s ability to patch.  Researching and understanding the new and shifting environment is critical, even for smaller or less well-known organizations now. The real world works less and less like a traditional pen test.  In addition, one of the most challenging aspects of security today is understanding the real-world effectiveness of your existing security controls against an active, human, skilled attacker, with unlimited time using cutting-edge research and techniques. In real-life attacks, the bad actors don’t have a time limit. They can spend months or even years on their target. In a pen test, the tester has a week or two to do his assessment and write up the findings. We can’t expect a team of testers to be able to cover all possible vectors in a few weeks. To address this, we are seeing more customers move to ‘advanced’ penetration tests such as red teaming (or adversarial attack simulation), as well as joint efforts from offensive (red) and defensive (blue) teams called purple teaming. Purple team (or Adversarial Detection & Countermeasures) engagements are designed to evaluate the effectiveness of the information security program, with a focus on Detection, Deflection, and Deterrence. Greatness requires sacrifice  To get to where we need to be on defense, the adversary simulation (or offense) needs to be simply great.  As we know, most assessment firms simply don’t want to invest the time and resources necessary to research and innovate new and cutting-edge attacks.  Critically, this involves threat research, adversary emulation, R&D of new and custom red team tooling, and analysis of indicators of compromise and technologies to help customers understand and improve the security posture of their environment.  It’s what the adversaries are doing, and it must be done on your behalf to get the value out of penetration testing.  Aligning to Risk - You once were found, but now you’re lost. Since everything in security ultimately rolls up to managing risk, risk assessments also need to evolve to include attack and threat intelligence.  As technology advances, without incorporating the valuable contributions of researchers, penetration testers, and advanced security community contributions, traditional risk assessments are becoming less valuable as technology advances. Risk assessments must include Adversary/ Threat Agent Simulation that compromises the confidentiality, availability, and integrity of some asset. The more you can emulate the TTP’s of a modern advanced adversary, the better the accuracy and value of your risk assessment.  Without that, what are you truly assessing?  It can’t be garbage in and garbage out anymore when modeling risk for sound decision-making.  By garbage, I mean either derived information, made up data, or simply basic vulnerability info.   This “model risk” means that there are hidden risks with most current assessments themselves.  This challenge will escalate in visibility as many high-profile breaches already had sophisticated risk management systems in place.  The business execs and boards are watching with much more scrutiny than in the past.

“It’s not just what we don’t know that hurts us...It’s what we think we know that ain’t so!”

With the state of security today, you need to know what you don’t know…That is, we must minimize the ‘unknown unknowns” in environments and use the latest advances to improve processes.    What we ‘know’ about penetration testing and risk assessments must be seen as a “work-in-progress” that needs to be continually re-examined and improved, rather than as a one-time learning.  With the average tenure of a CISO currently only 18 months, incomplete information from mediocre assessments can provide a false sense of security.  This truly increases the risk to your organization and, frankly, to you personally.

Listen to "The Evolution of Pen Testing" webinar, from June 20, 2018.