Manual vs Automated Testing

Google testing requirements mandate manual discovery of security relevant issues as part of an authenticated penetration test. This ensures that complex logic errors and unknown vulnerabilities are discovered by experienced penetration testers.

	Manual Testing	Automated Testing
Definition	Penetration testers perform manual checks of the security controls in place and attempt to bypass or circumvent them in the process of identifying vulnerabilities.	Security scanner software is launched against the asset in scope. The scanner works through a predefined set of known vulnerabilities or malicious payloads to test if the asset is vulnerable.
Level of detail	Medium/High	Low
Benefits	Very low false positive rate. Allows for identification of previously unknown vulnerabilities and logic errors that lead to security issues. Testers can exploit vulnerabilities in a chain to identify further issues.	Fast way to identify misconfigurations and missing security updates, as well as known vulnerabilities. Can be used to complement a vulnerability management program.
Limitations	The quality of the testing depends on the service provider and the skills of the individual penetration testers. Manual testing is also time-intensive and expensive. Defining the right scope is crucial.	Vulnerabilities in custom software cannot be identified at all or only in generic cases. The false positive rate is often very high.
VSA/IPA Requirements	Annual third-party penetration test report	Attestation of Quarterly scanning

Regular automated vulnerability scans of your network, servers, and workstations are a valuable part of a mature security program. The results help your security team identify areas where patching against known vulnerabilities and misconfigurations has not occurred.

However, automated vulnerability scans alone do not meet Google’s requirements as they are automated, commonly unauthenticated, and focus on known security misconfiguration and vulnerabilities in common software, instead of being tailored to the needs of your organization.

Validated Automated Security Scans

It is often argued that this type of scan is a combination of the manual and automated methodologies. However, the operation model often consists of automated discovery and manual validation. This improves the false-positive rate, but apart from that this approach still shares the same limitations as automated security scanning and doesn’t bring any of the benefits of manual testing. As such, validated automated scans are not considered a sufficient replacement for manual testing