Web Scraping Framework
Project description
## IOWeb Framework
Python framework to build web crawlers.
Good things:
- system designed to run large number of network threads (like 100 or 500) on
single CPU core
- feature to combine things in chunks and then doing something with
chunks (like mongodb bulk write)
asynchronous network operations are powered by gevent
network requests are handled with urllib3
HTML is parsed with lxml
ability to do CSS/XPATh queries to DOM tree of downloaded HTML document
ability to extract cert details
ability to resolve particular domain to custom IP
stat module to count events
logging statistics to influxdb
retrying on network errors
Bad things:
not fully covered with tests
no documentation
## Feedback
[t.me/grablab](https://t.me/grablab) - English chat about web scraping
[t.me/grablab_ru](https://t.me/grablab_ru) - Russian chat about web scraping
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file ioweb-0.0.18.tar.gz.
File metadata
- Download URL: ioweb-0.0.18.tar.gz
- Upload date:
- Size: 24.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/2.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 792426824ef83f5023a822bb434a8f69c29ea37bf168f29cf7ec4fdd755239a1 |
|
| MD5 | babcb18a530d0798cfa0269b454e33f6 |
|
| BLAKE2b-256 | 66216791b5a0595fe1e31640a4f06a52e000996e3fcb638c2f027e4038ffc76e |
