protego
DIGITAL Command Language
★ 90
updated 2d ago
A pure-Python robots.txt parser with support for modern conventions.
No plain-English explanation yet — one is being written right now. Check back in a minute.