Replacement robots.txt Parser in pure Python
Project description
Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:
Memoization of fetched robots.txt
Expiration taken from the Expires header
Batch queries
Configurable user agent for fetching robots.txt
Automatic refetching basing on expiration
This is a patched fork of the last pure Python version that works on Python 2 and 3.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
reppy2-0.3.6.tar.gz
(72.7 kB
view hashes)
Built Distribution
reppy2-0.3.6-py3-none-any.whl
(12.2 kB
view hashes)