pubcrawl
Python
★ 0
updated 15y ago
⑂ fork
A short and sweet Python web crawler using Redis as the process queue, seen set and Memcache style rate limiter for robots.txt
No plain-English explanation yet — one is being written right now. Check back in a minute.