Replacement robots.txt Parser
Project description
Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:
Memoization of fetched robots.txt
Expiration taken from the Expires header
Batch queries
Configurable user agent for fetching robots.txt
Automatic refetching based on expiration
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
reppy-0.4.10rc1.tar.gz
(89.5 kB
view hashes)
Built Distribution
Close
Hashes for reppy-0.4.10rc1-cp37-cp37m-macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df4fbd32625fd44c8f5fa4877c881cb4b13336e750e0be179b9b1a1837ac3f35 |
|
MD5 | b8247c2e9039707502e507e2fb43c781 |
|
BLAKE2b-256 | 70e40f7e3d618542a86f90f27308febb51b6c63f749322b828acba03dc70daf3 |