RobotsDisallowed
Shell
★ 1.5k
updated 3y ago
A curated list of the most common and most interesting robots.txt disallowed directories.
No plain-English explanation yet — one is being written right now. Check back in a minute.