A minimalistic, recursive web crawling library for Python.
Project description
The solitary and lucid spectator of a multiform, instantaneous and almost intolerably precise world.
—Funes the Memorious, Jorge Luis Borges
memorious is a light-weight web scraping toolkit. It supports scrapers that collect structured or un-structured data. This includes the following use cases:
Make crawlers modular and simple tasks re-usable
Provide utility functions to do common tasks such as data storage, HTTP session management
Integrate crawlers with the Aleph and FollowTheMoney ecosystem
Get out of your way as much as possible
Design
When writing a scraper, you often need to paginate through through an index page, then download an HTML page for each result and finally parse that page and insert or update a record in a database.
memorious handles this by managing a set of crawlers, each of which can be composed of multiple stages. Each stage is implemented using a Python function, which can be re-used across different crawlers.
The basic steps of writing a Memorious crawler:
Make YAML crawler configuration file
Add different stages
Write code for stage operations (optional)
Test, rinse, repeat
Documentation
The documentation for Memorious is available at alephdata.github.io/memorious. Feel free to edit the source files in the docs folder and send pull requests for improvements.
To build the documentation, inside the docs folder run make html
You’ll find the resulting HTML files in /docs/_build/html.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file memorious-2.6.4.tar.gz.
File metadata
- Download URL: memorious-2.6.4.tar.gz
- Upload date:
- Size: 40.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 6dddd8f23b098fde481e3d193e25c25ef743944c79474aac430c1e03ebcc7fa3 |
|
| MD5 | 8559e4179db9985aa8d3e90748148418 |
|
| BLAKE2b-256 | 869d4571945124ee88cb8454e8ff34972568aaa5a7d1c7b203509e40ec56acf2 |
Provenance
File details
Details for the file memorious-2.6.4-py2.py3-none-any.whl.
File metadata
- Download URL: memorious-2.6.4-py2.py3-none-any.whl
- Upload date:
- Size: 52.4 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4ff704bfb6558995fae68f50afff39304e0a1eb10196d674d0a0e0653507c1be |
|
| MD5 | 3081cc379a5c36c7a75cc04de511a2a1 |
|
| BLAKE2b-256 | dfb10508ae8e96d7559be3c077b989ffe56180a9f1787d277bdcaeb03d2170c8 |
