Testing Guidelines

Summary

For a checklist of specific requirements expected from a penetration test report, please see our Guidelines for a High-Quality Penetration Testing Report page, which provides a clear set of questions that need to be answered by the report and supporting documentation.

The following are explicitly not a replacement for regular third-party penetration testing and appropriate remediation:

  • Adherence to a security standard (e.g. ISO, NIST, PCI-DSS) [?]
  • Automated security scanning/testing [?]
  • BugBounty or Vulnerability reward programs [?]

Although these practices form part of a mature and successful security and privacy program, they do not provide the level of technical verification provided by regular third-party penetration testing with a clear scope and coverage across critical systems.


This document provides guidance on planning and scoping of third-party penetration tests that are required when working on projects touching sensitive data or systems. We also discuss several types of security testing that may be requested additionally based on the specific project that you are working on.

Contents


Different types of security tests1

The below table describes the most common types of security testing that may be needed for a partner working with Google.

Vulnerability Scan Static Source Code Analysis Penetration Test Source Code Audit
Method Automated Automated Manual Manual
Purpose Identify known vulnerabilities based on pre-configured signatures. Identify a specific set of vulnerabilities in source code. In-depth, manual security assessment of a defined scope. Very in-depth review of a single application/product.
Benefits Fast way to identify misconfigurations and missing security updates. Since these scans are automated, they can be run often and complement a vulnerability management process very well. Fast way to identify vulnerabilities newly introduced into code. Can identify previously unknown vulnerabilities. Very low false positive rate. Can find subtle logic errors that lead to security issues. Can follow vulnerability chains (exploiting vulnerabilities to identify further issues). Humans are still best at assessing risks. Can identify pretty much all types of vulnerabilities and security weaknesses in an application's code. Also able to identify design issues and recommend improvements.
Limitations Vulnerabilities in custom or uncommon software cannot be identified. Only finds vulnerabilities a signature has been created for. False positive rate is often very high. Still in its infancy. Usually reports a very high number of false positives making it hard to identify real findings. Limited support for discovery of logic errors (which includes entire classes of vulnerabilities, such as authentication bypass, some types of privilege escalation, etc.) Time intensive and expensive. Quality highly dependent on provider. Scoping often dictates the quality of the test. Very time intensive and expensive.
VSA/IPA Requirements2 Attestation of Quarterly scanning N/A Annual third-party penetration test report N/A
Example Providers3 Nessus, Nexpose, Qualys, Trustwave App Scanner, … Fortify, Veracode, CheckMarx, Whitehat, … NCC Group, Bishop Fox, Leviathan Security, …

1 This list is not intended to be exhaustive. It lists the most common types, as well as the ones that Google requires from its vendors.

2 Please check your Information Protection Addendum (IPA) for further guidance on these requirements for your specific project. Where additional types of testing are required, these will be clearly requested as part of the IPA and VSA process.

3 This list of includes security companies that are known to provide these services, and does not constitute a recommendation on the quality or coverage provided.

Google's requirements

Specific requirements may differ between vendors due to special agreements, but generally, when Google requests a penetration test, the requirements for those tests include:

  1. It must be performed by an appropriately qualified third-party provider; it is usually not sufficient that an internal team conducts the penetration test unless it is clear that the internal testing team meets strict standards. The quality between different providers can vary significantly, so we recommend to carefully compare providers. Upon request, Google can make recommendations for companies that we believe provide good quality services.
  2. The scope of the penetration test must include all infrastructure used to provide services to Google. If there is a security dependency (i.e. the infrastructure used to provide services to Google depends on other systems for its security), those systems need to be included in the scope of testing.
  3. The total testing effort (person days) must be appropriate for the number and scale of the systems in scope. Typically penetration tests take at least a few days for very small scopes, and at least a few weeks for larger scopes and applications.
  4. The largest portion of the penetration test must use manual methods for identifying security vulnerabilities. Purely automated methods are not sufficient, even if the results of the scan are manually verified.
  5. The penetration test must be authenticated - i.e. the penetration testers must be provided with credentials for applications that are not accessible without authentication (note: access to source code or to systems on the operating system level is not necessary).
  6. Bug bounty programs are not a sufficient replacement for penetration tests.

What do we look for in a pentest report?

Different pentest providers write their reports differently. This difference sometimes results in key information missing from reports. In order to ensure that the correct information is available in the report to meet requirements, we have outlined the requirements and questions that we need to clearly answer in our pentest report guidelines page.


The remainder of this section provides more details and the rational for these requirements.

Picking a qualified pentest provider

There are a lot of companies that claim to offer penetration testing services, but unfortunately only few provide high-quality services. This often makes it hard for companies who are not security experts themselves to distinguish between qualified and unqualified providers.

Here are a few tips and recommendations on how to find good pentesters:

  • Companies that are specialized on security-services often provide better penetration tests than companies that "also" do pentests in addition to entirely unrelated services.
  • Sometimes even security companies use penetration tests to pitch their "main" product. Picking a company where the main products are security consulting (including pentesting) is often a safer bet.
  • Most highly qualified penetration testing providers run a blog where they provide technical details of security vulnerabilities and research they have recently conducted. It also helps to look for "CVEs", that is, vulnerabilities in standard software that the company identified in the course of their work.
  • Presentations at industry conferences such as BlackHat, DEF CON, Hack-in-the-Box, CanSec West, etc. are usually a good indicator that the company knows what they are doing.
  • Even within qualified companies, expertise between individual people can vary significantly. To get the most out of a penetration test, try insisting to get senior security consultants, ideally folks credited with having found security vulnerabilities (CVEs), or who presented at industry conferences.

The importance of third-party testing

Although many companies have internal teams capable of performing detailed penetration testing, it is important to include third-party testing as part of your testing program. Third parties often come at testing with a different mindset than an internal team, and as a result are more likely to examine processes and functions without being prejudiced by previous uses of the application. It is often easy as an internal tester to discount the presence of certain bug classes due to familiarity with the product, deployment style, or existing security mechanisms that should be in place to protect against such attacks. A third-party penetration tester can provide an unbiased and honest look at an application that is often lacking with internal teams.

This is equally important when performing annual penetration testing, and we would suggest rotating testing teams or third-party providers every 2-3 years to ensure a fresh view on the application.

Compliance and penetration testing

Compliance with common security standards (e.g. ISO, NIST, PCI-DSS) may require you to perform specific types of security scanning and testing. Although these types of testing are all part of a well managed and mature security and privacy program, they may not in themselves meet Google's requirements alone.

A common example is PCI-DSS which requires penetration testing to be performed as part of the compliance requirements. Simply having PCI Compliance, does not however mean that the penetration testing that was performed to obtain that compliance meets Google's standards. Specifically the scoping of systems under test will likely differ between the two requirements. Google also specifically calls out validity periods (12 months) on penetration testing, and calls for testing to be performed in an authenticated and manual fashion.

Penetration test scoping

The penetration test must include application-level testing of any Internet-exposed application related to the processing or storage of sensitive data.

Limiting the scope of a penetration test to unauthenticated users may result in only a small subset of issues being discovered compared to a more comprehensive test that gives test credentials to access the application itself. A limited test can find issues that an casual external attacker would discover, but it often overlooks more detailed attacks and functionality that can be abused by existing users of the system.

Specifically, an authenticated user might be able to access information that should not be user-accessible, or the ability to perform actions that should be restricted to administrative users only. With the increase in multi-tenant systems, attacks from legitimate users of the system, either directly, or due to hijacked accounts (via weak passwords, phishing, malware, etc...) are seen as a threat that is often overlooked when planning the scope and type of a penetration test.

Unauthenticated vs. Authenticated Testing

The below table compares the pros and cons of authenticated vs. unauthenticated penetration testing.

Unauthenticated Testing Authenticated Testing4
Definition Considers that the testers have no knowledge of the environment under test. No credentials are provided to the testers. Testers have full access to information about the platform being tested. This often includes accounts (including administrative users), and access to discuss functionality with developers during the testing process.
Level of detail Low Medium/High
Benefits Simulates casual attackers and automated attacks on the first line of defense within an application. Does not require a high level of expertise to perform, no knowledge of the business logic is required. Detail orientated testing, including business logic type issues. More likely to find issues that would otherwise not be discovered during code audit, unauthenticated testing, or automated scanning alone.

Can be useful in testing monitoring and alerting capabilities when used in coordination with an internal IR/Monitoring team exercise.

Limitations Often misses issues that require knowledge of the application or infrastructure, or require a minimum level of access as a user. Can take additional resources from the development team, as well as testers. Often requires more experienced testers due to the increased complexity and type of issues.
IPA Requirements Annually

4 Google's testing requirements mandates third-party penetration testing that is authenticated. This ensures that all areas of the application are tested equally to discover issues that may otherwise not be easily identified without full access to the application.

What part do vulnerability scans play in security testing?

Regular vulnerability scans of your network, servers, and workstations are a valuable part of a mature security program, and help your security team to identify areas where patching against known5 vulnerabilities and misconfigurations has not occurred.

Example of automated vulnerability scanning tools: Nessus, Qualys, …

5 Google's testing requirements mandate manual discovery of security relevant issues as part of an authenticated test. Automated vulnerability scans do not meet this requirement as they are automated, commonly un-authenticated, and focus on known security misconfiguration and vulnerabilities in common software, instead of being tailored to the needs of your organization.

What are validated scans?

Various providers offer validated application or dynamic source code scans as a replacement for penetration testing. As the manual component of this style of testing is only performed during the final "validation" stage, these cannot be classed as a suitable replacement for manual penetration testing 6.

Example of validated scans: WhiteHat Sentinel dynamic scanning, …

6 Google's testing requirements mandate manual discovery of security relevant issues as part of an authenticated test.

What part can Bug Bounties play in security testing?

Bug Bounty programs can provide useful input into a mature security program as long as they are properly scoped and managed. Many companies chose to run security programs that offer rewards for reported bugs or security issues, including the Google Vulnerability Reward Program.

Although program such as these are widely used and growing in popularity, we currently see limitations in the crowd-sourced "penetration testing" system, and do not consider them to be a drop-in replacement for high quality third-party penetration test7.

7 The quality of Bug Bounty programs vary based on a number of factors: - including scope, exclusions, repeatability, reward, interest, program visibility. As such it is very hard to measure the overall quality and coverage, which we feel are all required for a high quality penetration test. Based on these factors, Google is not able to accept reports from Bug Bounty programs or providers as a replacement for a third-party penetration test.