fetchbot
Go
★ 791
updated 5y ago
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
No plain-English explanation yet — one is being written right now. Check back in a minute.