Subtle changes, bugs and feedback

And if you're wondering why the HEAD requests...

I'm scanning the forum for malware and links to bad places.

The scan works by crawling from the root page, and indexing all links on a page, and ignoring /search /huddles and /comments. It then queues those links as HEAD requests, and if the HEAD returns a content-type of text/html it knows it can then GET those pages and index more links. On every page it crawls, it then finds every link to anything at all and checks them against various blacklists.

{"fqdn":"http://www.lfgss.com","started":"2015-06-20T19:28:08.645145731+01:00","inProgress":true,"scannedURLs":6388,"headReqs":263,"getReqs":120}

So 263 links within the site I've made HEAD requests which identified 120 web pages, and on those 120 pages I've found 6,388 URLs that I've scanned.

I've not yet turned up a single piece of malware.