robots-parser

JavaScript ★ 0 updated 8y ago ⑂ fork

NodeJS robots.txt parser with support for wildcard (*) matching.

No plain-English explanation yet — one is being written right now. Check back in a minute.