Note: This article has been updated for 2021.
Selecting a penetration testing company can be a daunting task. It’s an industry plagued with misleading sales tactics, weak certifications, and simply unqualified professionals. To make things worse, penetration testing is an art, which is has inherent challenges to structure as a professional service.
Remember that penetration testing is specifically designed to simulate malicious hackers who do not follow rules and use creative ways to abuse your technology. This has an unavoidable conflict with the trend many pentest companies have to offer a predefined, reproducible, and structured service.
I’ve been a professional penetration tester for almost 20 years, and I greatly sympathize with those seeking their first assessment. Below are my five most sincere tips for sorting through the hype, bogus claims, and general lack of capabilities of most pentest companies.
1. A penetration testing company should be creative
A long time ago I worked at a large penetration testing company that assured customers they had a “100% reproducible methodology”. This was boasted confidently at every sales meeting, met with approving nods from other executives. The idea was that their methodology was so great, and had a super-awesome checklist that could find every bug in an application. I had never seen this methodology.
Pentesting is far more of an art than science; and any competent pentesting company has an obligation to set those expectations. While there are very important checklists and methodologies to follow during a pentest, this is largely considered a trivial and straight forward part of the process.
The value of an experienced pentester always lies in the manual analysis; this is where they attempt to undermine security controls, bypass business logic, and use behaviors and technologies in unexpected ways.
This is a creative process and unlikely to be reproduced identically even by the most skillful pentesters. For example, we recently identified a Twitter XSS which involved a complex analysis of application behaviors. These things combined led to the discovery of one of the most critical vulnerabilities affecting the platform.
2. Expertise in your application stack and technologies
When you speak to a pentest company for the first time the conversation should be a two way street. If you tell a vendor your application is hosted on AWS, they should have a slew of questions about your stack; “What services specifically do you use?”, “Do you leverage s3 for storage?”, “Is your authentication backed by Cognito?” are all example questions you would want to hear.
“What is your codebase?” and “How many lines of code?” are two very typical scoping questions, but if the pentest company doesn’t dig much deeper beyond that to truly understand your attack surface, it may be a good idea to shop around.
3. Evaluating the scoping process
The scoping process is typically the time a penetration testing company will need to understand your application and determine a level of effort for testing. This, however, is an even better opportunity for you to evaluate them as a vendor.
Ask yourself if the company is truly trying to understand your app and technologies, or simply running through a checklist of questions. If the application scoping process feels like it’s just a checklist of questions, the actual pentest is likely to be just a checklist as well.
Do their questions include follow up questions that demonstrate they actually care and want to do the best assessment possible? Or are you being funneled strictly through their process? If everything feels routine, your pentest is unlikely to get the special consideration it needs.
4. Certifications (or lack thereof?)
Pentest certifications are a double edged sword. On one hand, they assure a moderate level of competence. On another, they still fall very short of whats expected of a skillful pentester. Remember that certification bodies inherently must target a large enough group of people to stay profitable.
Pentest companies which boast of “certified hackers” should be evaluated cautiously. In the pentesting community, there is very little weight is given to most of these certifications. For those seeking penetest services, it’s important to understand that as well.
Instead of making judgement based on certifications, check their Github, their blogs, and their research. Any skillful pentesting company will usually contribute a great deal to the security community.
5. Cost
Cost is suddenly a high priority for most organizations. As I write this quarantined in the midst of a pandemic, more companies than ever are looking to get more value for the dollar.
The importance of right-sizing your penetration testing company has never been more important.
At the end of the day, the vast majority of penetration tests will be performed by one person. When using a vendor there’s an unavoidable part of cost that goes to company’s operational overhead. There are two rules I recommend most to follow:
- Pick a company you feel is competent to assess your application.
- Pay as little as possible for overhead
Consider our breakdown of cost fictional big-box penetration testing company:
There are a lot of approximations and variables here. However, when you compare the cost of a pentest to the salary of the pentester, you can get a pretty clear idea of how much value is returned. This is one reason many large institutions are beginning to prefer smaller pentest companies as vendors.
6. The “Uber model”, or not?
The rise of on-demand testing performed by a number of pentesters has gained some popularity over the last few years. Instead of one individual tester, several testers review the application in parallel. The idea is that multiple testers should find bugs that the others members have missed, potentially giving better coverage.
But there’s a fundamental flaw, which is the payment must be divided between the 4 or 5 testers. The business model unfortunately must use a good number of entry-level or low skill pentesters. To add to the problem with this, the testers still perform a large amount of redundant work.
We would argue that one highly skilled tester is far more effective than more low skill testers.
Conclusion
If you made it to the end of this I hope this was helpful in your search!
Virtue Security is a trusted penetration testing company located in New York City. We are an engineering focused team offering continuous penetration testing through our platform PurpleLeaf as well as traditional point in time penetration tests of SaaS platforms, mobile applications, and internal/external networks.