robots-txt-scanner
JavaScript
★ 2
updated 8mo ago
Scans robots.txt files and logs the Disallow - used for some research
No plain-English explanation yet — one is being written right now. Check back in a minute.