Web Scraping Framework
Project description
## IOWeb Framework
Python framework to build web crawlers.
What we have at the moment:
- system designed to run large number network threads (like 100 or 500) on
on CPU core
- built-in feature to combine things in chunks and then doing something with
chunks (like mongodb bulk write)
asynchronous things are powered by gevent
network requests are handled with urllib3
urllib3 monkey-patched to extract cert details
urllib3 monkey-patched to not do domain resolving if domain IP has been provided
built-in stat module to count events, built-in logging into influxdb
retrying on errors
no tests
no documentation
I am using ioweb to do bulk web scraping like crawling 500M pages in few days.
## Places to talk
[t.me/grablab](https://t.me/grablab) - English chat about web scraping
[t.me/grablab_ru](https://t.me/grablab_ru) - Russian chat about web scraping
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file ioweb-0.0.11.tar.gz.
File metadata
- Download URL: ioweb-0.0.11.tar.gz
- Upload date:
- Size: 19.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/2.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b74b76be187975e4ebde1ebc32662869a0209ba1144ea58e83744c33ad6682dc |
|
| MD5 | a8719172d0264cc3df4be1fe3363cb83 |
|
| BLAKE2b-256 | 6d658a1dac6804add8f4b03ce8cfbb3bc67fb389a63f06c40807f18c8c63e0fa |
