|logo| .. |logo| image:: https://raw.githubusercontent.com/scrapy/scrapy/master/docs/_static/logo.svg :target: https://scrapy.org :alt: Scrapy :width: 480px |version| |python_version| |ubuntu| |macos| |windows| |coverage| |conda| |deepwiki| .. |version| image:: https://img.shields.io/pypi/v/Scrapy.svg :target: https://pypi.org/pypi/Scrapy :alt: PyPI Version .. |python_version|…
|logo|
.. |logo| image:: https://raw.githubusercontent.com/scrapy/scrapy/master/docs/_static/logo.svg
:target: https://scrapy.org
:alt: Scrapy
:width: 480px
|version| |python_version| |ubuntu| |macos| |windows| |coverage| |conda| |deepwiki|
.. |version| image:: https://img.shields.io/pypi/v/Scrapy.svg
:target: https://pypi.org/pypi/Scrapy
:alt: PyPI Version
.. |python_version| image:: https://img.shields.io/pypi/pyversions/Scrapy.svg
:target: https://pypi.org/pypi/Scrapy
:alt: Supported Python Versions
.. |ubuntu| image:: https://github.com/scrapy/scrapy/workflows/Ubuntu/badge.svg
:target: https://github.com/scrapy/scrapy/actions?query=workflow%3AUbuntu
:alt: Ubuntu
.. |macos| image:: https://github.com/scrapy/scrapy/workflows/macOS/badge.svg
:target: https://github.com/scrapy/scrapy/actions?query=workflow%3AmacOS
:alt: macOS
.. |windows| image:: https://github.com/scrapy/scrapy/workflows/Windows/badge.svg
:target: https://github.com/scrapy/scrapy/actions?query=workflow%3AWindows
:alt: Windows
.. |coverage| image:: https://img.shields.io/codecov/c/github/scrapy/scrapy/master.svg
:target: https://codecov.io/github/scrapy/scrapy?branch=master
:alt: Coverage report
.. |conda| image:: https://anaconda.org/conda-forge/scrapy/badges/version.svg
:target: https://anaconda.org/conda-forge/scrapy
:alt: Conda Version
.. |deepwiki| image:: https://deepwiki.com/badge.svg
:target: https://deepwiki.com/scrapy/scrapy
:alt: Ask DeepWiki
Scrapy_ is a web scraping framework to extract structured data from websites.
It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_
(formerly Scrapinghub) and many other contributors_.
.. _many other contributors: https://github.com/scrapy/scrapy/graphs/contributors
.. _Scrapy: https://scrapy.org/
.. _Zyte: https://www.zyte.com/
Install with:
.. code:: bash
pip install scrapy
And follow the documentation_ to learn how to use it.
.. _documentation: https://docs.scrapy.org/en/latest/
If you wish to contribute, see Contributing_.
.. _Contributing: https://docs.scrapy.org/en/master/contributing.html
Members
-
scrapy ★ PINNED
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python ★ 63k 1d agoExplain → -
scrapyd ★ PINNED
A service daemon to run Scrapy spiders
Python ★ 3.1k 5d agoExplain → -
parsel ★ PINNED
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
Python ★ 1.3k 1d agoExplain → -
w3lib ★ PINNED
Python library of web-related functions
Python ★ 419 2d agoExplain → -
protego ★ PINNED
A pure-Python robots.txt parser with support for modern conventions.
DIGITAL Command Language ★ 90 5d agoExplain → -
itemadapter ★ PINNED
Common interface for data container classes
Python ★ 70 2d agoExplain → -
scrapely
A pure-python HTML screen-scraping library
HTML ★ 1.9k 4y agoExplain → -
dirbot
Scrapy project to scrape public web directories (educational) [DEPRECATED]
Python ★ 1.6k 8y agoExplain → -
quotesbot
This is a sample Scrapy project for educational purposes
Python ★ 1.4k 2y agoExplain → -
scrapyd-client
Command line client for Scrapyd server
Python ★ 772 4mo agoExplain → -
cssselect
CSS Selectors for Python
Python ★ 309 2d agoExplain → -
queuelib
Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python
Python ★ 299 2d agoExplain → -
loginform
Fill HTML login forms automatically
Python ★ 279 2y agoExplain → -
slybot
No description.
★ 224 11y agoExplain → -
scrapy.org ▣
The scrapy.org website (old code)
HTML ★ 66 1y agoExplain → -
itemloaders
Library to populate items using XPath and CSS with a convenient API
Python ★ 49 2d agoExplain → -
booksbot ⑂
A crawler for http://books.toscrape.com
Python ★ 42 3y agoExplain → -
scrapy-bench
A CLI for benchmarking Scrapy.
Python ★ 32 1y agoExplain → -
scrapy-lint
A linter for Scrapy projects.
Python ★ 22 19d agoExplain → -
scurl
Performance-focused replacement for Python urllib
Python ★ 21 3mo agoExplain → -
pypydispatcher
A fork of http://pydispatcher.sourceforge.net/ with PyPy support
Python ★ 16 9y agoExplain → -
xtractmime
https://mimesniff.spec.whatwg.org/ implementation for Python
Python ★ 13 17d agoExplain → -
base-chromium
base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
C++ ★ 8 8y agoExplain → -
scrapy-itemloader ▣
[Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
Python ★ 7 6y agoExplain → -
form2request
Python library to build HTTP requests out of HTML forms
Python ★ 5 25d agoExplain → -
url-chromium
url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url
C++ ★ 4 8y agoExplain → -
gsoc2014-integration-tests
GSoC2014 - Scrapy Integration tests project
Shell ★ 3 12y agoExplain → -
scrapy-bench-speedcenter ⑂
Codespeed for scrapy-bench
Python ★ 2 9y agoExplain → -
sphinx-scrapy
Sphinx extension for documentation in the Scrapy ecosystem
Python ★ 1 1mo agoExplain →
No repos match these filters.