Scrapy 2.11.2

A high-level Web Crawling and Web Scraping framework

Stars: 52733, Watchers: 52733, Forks: 10507, Open Issues: 670

The scrapy/scrapy repo was created 14 years ago and the last code push was 2 days ago.
The project is extremely popular with a mindblowing 52733 github stars!

How to Install scrapy

You can install scrapy using pip

pip install scrapy

or add it to a project with poetry

poetry add scrapy

Package Details

Author

Scrapy developers

License

BSD

Homepage

https://scrapy.org

PyPi:

https://pypi.org/project/Scrapy/

Documentation:

https://docs.scrapy.org/

GitHub Repo:

https://github.com/scrapy/scrapy

Classifiers

• Internet/WWW/HTTP
• Software Development/Libraries/Application Frameworks
• Software Development/Libraries/Python Modules

Related Packages

Errors

A list of common scrapy errors.

Code Examples

Here are some scrapy code examples and snippets.

GitHub Issues

The scrapy package has 670 open issues on GitHub

• Proxy + invalid domain makes Downloader stuck
• Fix command parse unhandled error :AttributeError: 'NoneType' object has no attribute 'start_requests'(#3264)
• Issue #5366 Move from optparse to argparse
• remove pywin32 docs section
• deprecate scrapy.pipelines.images.NoimagesDrop
• Remove scrapy.pipelines.images.NoimagesDrop
• Removed the pywin32 docs section
• Remove the pywin32 docs section
• Deprecate scrapy.pipelines.images.NoimagesDrop
• Move from optparse to argparse
• Create SECURITY.md
• Add SECURITY.md
• Use coverage data from Windows runs in addition to Ubuntu ones
• Document coroutines for signals.
• Updated overview for video intro

See more issues on GitHub