Pure-Python robots.txt parser with support for modern conventions
Project description
Protego
Overview
Protego is a pure-Python robots.txt parser with support for modern conventions.
Requirements
- Python 2.7 or Python 3.4+
- Works on Linux, Windows, Mac OSX, BSD
Install
To install Protego, simply use pip:
pip install protego
Usage
>> from protego import Protego
>> import requests
>> r = requests.get('https://google.com/robots.txt')
>> rp = Protego.parse(r.text)
>> # That's it! We can now perform queries.
>> rp.can_fetch('https://google.com/search', 'mybot')
False
>> rp.can_fetch('https://google.com/search/about', 'mybot')
True
>> list(rp.sitemaps)
['https://www.google.com/sitemap.xml']
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Protego-0.1.dev0.tar.gz
(4.9 kB
view details)
Built Distributions
Protego-0.1.dev0-py3.7.egg
(9.3 kB
view details)
Protego-0.1.dev0-py2.7.egg
(9.3 kB
view details)
File details
Details for the file Protego-0.1.dev0.tar.gz.
File metadata
- Download URL: Protego-0.1.dev0.tar.gz
- Upload date:
- Size: 4.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | adda509c6926c1c18b06445cc1b6b71e95e43ef17afc4ebd0d02ad313f954947 |
|
| MD5 | bf3c4c823aba0244dd8abcf637ab9909 |
|
| BLAKE2b-256 | a05901c3a825711cfa01583144fe1f7f05bd59661b700a9638daa846466a59f1 |
Provenance
File details
Details for the file Protego-0.1.dev0-py3.7.egg.
File metadata
- Download URL: Protego-0.1.dev0-py3.7.egg
- Upload date:
- Size: 9.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a688ba1f8fae4a8968730f32b3ca3a274ca464a058267a7c4a987b20d2b6b86f |
|
| MD5 | 438b39f45b2c719a28e308b2c971b8b9 |
|
| BLAKE2b-256 | 17f7b70f56760c04b3c7f76d4f61e9879ef18e69728a4a1b86954240dde87155 |
Provenance
File details
Details for the file Protego-0.1.dev0-py2.7.egg.
File metadata
- Download URL: Protego-0.1.dev0-py2.7.egg
- Upload date:
- Size: 9.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 860f5734f43f2bdb96b90457772fa9d198c9d7a31c3ddf24195d1798efbae2fb |
|
| MD5 | 42bc4c9c481da417b320c7578a791e12 |
|
| BLAKE2b-256 | 959592cdfab438613721c84e550fbfccd9e3a7393de307ca2ee3a62774f5aef0 |
