List: websecurity@lists.webappsec.org
From: Adam Muntner
Re: [WEB SECURITY] program to crawl website looking for string patterns
Fri, Sep 16, 2011 5:23 PM
Many websites have entire sections that automated web crawlers can
never reach. The crawler may also not handle authenticated session
state well, without some effort. Solution: If depth is critical, make
sure to do a manual crawl, as well. You can load the PII regex
patterns into something like Burp, in order to monitor the traffic
passively for them.