feapder

Feapder is a Python web scraping framework designed to simplify building spiders that collect data from websites. The README is written in Chinese, but the project description and code examples are accessible regardless. The framework includes four built-in spider types intended for different use cases: AirSpider for lightweight tasks, Spider and TaskSpider for distributed collection, and BatchSpider for large batch jobs.

The framework supports resumable crawling, meaning if a spider stops mid-run it can pick up where it left off rather than starting over. It also includes monitoring and alerting, browser rendering for pages that require JavaScript to load, and deduplication tools for large datasets to avoid processing the same item twice.

Getting started is straightforward. You install it via pip, run a command to generate a new spider file, and the generated code shows a basic structure: a start_requests method that produces URLs to fetch and a parse method that handles the responses. The quick example in the README creates a spider that fetches a single page and prints the response.

There are three install variants: a minimal version, a version with browser rendering support, and a complete version that includes all features including MongoDB support. The complete version may require some extra setup steps, which the README links to a troubleshooting guide for.

A companion web application called feaplat provides a management interface for deploying and scheduling spiders. The project requires Python 3.6 or newer and runs on Linux, Windows, and macOS. Documentation is available at feapder.com.