cobweb
JavaScript
★ 3
updated 14y ago
⑂ fork
Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
No plain-English explanation yet — one is being written right now. Check back in a minute.