← All field notes

Black Lantern Labs Team  ·  Research

What a Penetration Test Should Give You

A lot of what gets labeled penetration testing is a vulnerability scan with a cover page. The scanner runs, findings land in a template, and you receive a long document that nobody reads and nothing changes. You paid for testing. You got a binder.

The difference between a scan and a test is a human adversary. A scanner checks systems against known issues. A tester chains things together: a low-severity leak here, a misconfiguration there, and suddenly there is a path to something that matters. The findings that hurt in real life are often chains. Scanners do not report chains well.

Four things a real test delivers

Evidence. Every finding should come with proof: the request that triggered it, the result, the steps to reproduce it. If engineers cannot reproduce a finding from the report alone, the finding is a rumor. A clean rule: if it cannot be reproduced, it does not go in the report.

Severity you can defend. Scanner scores are context-free. A "critical" on a system with no sensitive data and no path to anything valuable may matter less than a "medium" on a payment flow. Real severity accounts for what the system does in the business.

A fix path in two languages. Each finding needs remediation guidance engineers can execute and a plain-language impact statement leadership can act on. A report only specialists understand is half a report.

Verification after remediation. Fixing is the point. When a team remediates, someone should confirm the path actually closed. A test with no verification tells you where you were, not where you are.

Questions worth asking any testing provider

  • Who is actually doing the testing, and what is their background?
  • Can you see a sanitized sample report before you sign?
  • How are findings verified, and is retesting part of the engagement model?
  • What happens if something looks actively exploited?

Providers worth hiring have crisp answers and respect the questions. Treat vagueness as data.

Bottom line

Demand evidence, business-aware severity, fix guidance, and verification. Everything else is formatting.

← All field notes