Do not kill your pentester for little or no value-add

Disclaimer: This would be a long post (culmination of many old posts) with lot of different opinions, thoughts. If weaving is not right, please provide feedback on how it could be corrected.

I had the good fortune of reading couple of threads by gentlemen whom I respect for their grounded advices. Their posts triggered some thoughts in me that I had jotted in few old posts. So i tried to weave them into one post.

The thoughts are my own and hence, may read slightly off the mark. This is NOT a scientific post, but my observations. You have been warned.

I have divided my thoughts into 3 buckets. They are

1. Technology
2. People
3. Process

Technology

Penetration testing (or pentesting) is evolving along with technology. Earlier days of nmap-metasploit-jackpot are over (although I have a nagging feeling that this may not be case for every organization). Defense is becoming smarter (EDR, better firewalls). Although more buzzwords have entered the market (red-team, breach and attack simulation, intelligence based penetration test, etc.), the implementation has not yet reached the mass.

People

Assuming there is certain body of knowledge related to pentesting, there are two ways to achieve it — in a top-down manner or bottoms-up. The BoK doesn’t change, only the method of attaining it changes. The method depends on the way how your brain is wired to learn things.

Top-down guys always look at things from high level and then take each step down. For example, a top down guy who is tasked with learning pentesting will almost always start from outer perimeter of the BoK (the process of pentesting, different methodologies, types of pentest, high level execution methodology, etc. This type of thinking suits a person at a middle level to higher levels of management, because this thought process supports an abstraction based thinking. These are also the people who will, usually, collect their thoughts in a way so as to publish a book.

Now, you are asking for disaster if you ask a top-down person to:-

• Perform a technical penetration test (I am not talking your nmap-metasploit type);
• Explain why a particular nmap switch has to be used;
• Learn things by reading a book that doesn’t have any hands-on element in it;
• Argue the differences between different firewall evasion strategies, etc.

A bottom-up guy will always start with getting their hands dirty (running the tool rather than going through the man-page). Then, slowly, this person will start picking different tools, running them, then combining them in a script. Understanding each tool’s output, executing a full-blown engagement will come later. These are the people who amass lot of knowledge & wisdom over a period of time, but organizing it into a book — sorry! These people don’t have patience to read a book from cover to cover, let alone design and publish one.

Now, hard-core penetration testers usually are of the bottoms-up type. Which means you are asking for disaster if you ask them to:-

• Prepare an executive summary that can be consumed by an executive / decision maker
• Align their findings / vulnerabilities with any particular risk management standard (e.g., FAIR, ISO 27001, etc.)
• Perform risk assessment.
• Explain differences between CVSS v2 & v3

Another question that needs thought — where to use bug-bounty hunters and pentesters?

TLDR — use bug bounty hunters after you reach “unknown unknowns”. See the sub-section “timing has to be right” below for more details.

Process

This is where I think we need to work our things out. To that extent, I agree with most of the points that were made here. Here’s my commentary on Greg’s observations: -

Scoping it right, but it gets left all the time

Pentesting is both an offensive and individual engagement. Setting the rules of engagement is one of the toughest things to get right. Both pentester (am I doing more for less?) and the customer (am I getting less for more?) grapple with it. While customer don’t want to get hacked, they don’t want to expose their systems to harsh testing because they have a fear that it will break. They just want pentester to inform them of any vulnerability without exploiting it so that they can close it. However, pentester often fail to communicate it properly that without trying to break, they won’t know if a vulnerability exists! It is one of those catch-22 situations that lives long after the engagement is over.

This issue leads to ambiguous language in contracts that becomes cause of lot of heart-break and half-assed outcomes from a pentest. However, it is mostly because the engagement was not thought through.

If you want more vulnerabilities to be identified, provide information to your penetration tester and allow them to break it to test it.

Check inside, not just outside

I agree with Greg here. Testing outside perimeter alone won’t help in understanding a company’s security posture. Internal pentest need to happen as well. It will help understand the impact of what happens when an internal credentials / system is compromised. There is a lot more information that gets accessible to an adversary once a working AD credential is compromised.

Timing has to be right

I think every company, from a security posture point of view, goes through these phases (based on a loose interpretation of the Johari Window (https://en.wikipedia.org/wiki/Johari_window): -

Known Knowns (they know they have issues, and others do as well)

The companies in this segment know that they have to improve on their security posture a lot and understand the major areas to focus (knowing fully well that vulnerabilities must be present in a plenty). Further, they have issues in their external perimeter (leaked info, keys, access, etc.). Pentester’s wet dream. These companies should start with regular scanning of their infrastructure (outside in, i.e., outside perimeter first, then inside) with a focus towards internalizing patch management (infra, apps, especially third-party apps like Java, Adobe etc.). Don’t dream of getting a red-team / purple-team engagement. You won’t get as much benefit as you should as you won’t stretch those red-teamers.

Known Unknowns (they know they have issues, but others don’t know that yet)

Most of companies fall in this bucket. They have an understanding of critical assets that they should protect and have implemented controls in / around those assets. However, they are not sure if that could be all. Also, these issues mostly exist in the internal environment. These companies should have 1 red-team engagement covering outside perimeter (without phishing in scope), coupled with round the year internal pentesting.

Unknown Unknowns (neither they know that they have issues, nor others yet)

It could be most dangerous (if the company doesn’t have a single decision maker who thinks security should be at-least an agenda point in decision making meetings) or most mature state in a company’s life, depending on various factors.

• If it is former, ISO 27001 (led by a good consultant) should be a good choice, as it will start the institutionalization of security processes from top-down. Good consultant is imperative as it will THE deciding factor between an excellent ISMS that delivers value & a botched up one. ISMS implementation is a favorite topic of mine, more on that later.
• If it is latter, they should hire red-teamers & bug-bounty hunters with focus on one area per quarter (e.g., phishing, external firewall bypass, physical security, chaining multiple vulnerabilities to access key company data, etc.).

Needs to align with high level controls to give a sense of state of security

I have been saying this in my previous articles as well. There is value in both top down approach (governance, assurance, audit) & bottoms-up approach (regular housekeeping activities related to security). One cannot exist without another. Top down and bottom-up people need to meet halfway before deciding on the security assessment part. Imagine this (sort of ideal scenario):-

• The CISO consults involves internal auditors to set the scope for the RFP towards getting a vendor for their yearly penetration testing;
• The Internal auditors design a set of technical controls, mapped to the company’s security policy / procedures.
• Internal auditors plan/schedule their internal process audits along with these penetration testing exercises;
• The technical controls are provided to pentesters to test, along with the normal pentesting workflow;
• The output of the pentest exercise (aka findings) are then mapped to company’s security procedures & the internal audit findings, to provide a holistic view of security status of the company.

— End of post —