simple-web-crawler
HTML
★ 0
updated 6y ago
A simple crawler that takes a URL as input and tries to crawl all the web pages connected to this page and crawl the crawled pages and so on. It is designed in a multithreaded way which allows to run several crawlers concurrently.
No plain-English explanation yet — one is being written right now. Check back in a minute.