A webmining CLI tool & library for python.
Project description
minet is a webmining CLI tool & library for python that can be used to collect and extract data from a large variety of web sources such as raw webpages, Facebook, CrowdTangle, YouTube, Twitter, Media Cloud etc.
It adopts a lo-fi approach to various webmining problems by letting you perform a variety of actions from the comfort of your command line. No database needed: raw data files such as CSV should be sufficient to do the work.
In addition, minet also exposes its high-level programmatic interface as a python library so you can tweak its behavior at will.
Use cases
- Downloading large amount of urls very fast. (example)
- Writing scrapers to extract structured data from HTML pages.
- Writing crawlers to automatically browse the web.
- Extract raw text content from HTML pages. (example)
- Normalize batches of urls contained in a CSV file to perform relevant aggregations (dropping irrelevant query items, extracting domain name etc.) (example)
- Join two CSV files based on columns containing urls needing to be organized hierarchically.
- Collecting data from CrowdTangle API (to collect and search posts mainly from Facebook and Instagram).
- Collecting data from Facebook (comments, likes etc.)
- Parsing Facebook urls in a CSV file.
- Collecting data from Twitter (users, followers, followees etc.)
- Collecting data from YouTube (captions, comments, video metadata etc.)
- Parsing YouTube urls in a CSV file.
- Dumping a Hyphe corpus.
- Collecting data from Media Cloud (search stories, dump topics etc.).
Features (from a technical standpoint)
- Multithreaded, memory-efficient fetching from the web.
- Multithreaded, scalable crawling using a comfy DSL.
- Multiprocessed raw text content extraction from HTML pages.
- Multiprocessed scraping from HTML pages using a comfy DSL.
- URL-related heuristics utilities such as extraction, normalization and matching.
- Data collection from various APIs such as CrowdTangle.
Installation
minet can be installed using pip:
pip install minet
If you need more help to install and use minet from scratch, you can check those installation documents.
Cookbook
To learn how to use minet and understand how it may fit your use cases, you should definitely check out our Cookbook.
Usage
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file minet-0.35.4.tar.gz.
File metadata
- Download URL: minet-0.35.4.tar.gz
- Upload date:
- Size: 61.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 5b70bfb1a7f947a2d0778e3422e9ca6196df8f2eef70b2cc18490b06c74a5676 |
|
| MD5 | e89ec0b9dea6e2227072b572d8b52b11 |
|
| BLAKE2b-256 | 0f3a4d30b6d4b5acde18979a861857d4e1c34ed20d38d74918b17564246ce2eb |
Provenance
File details
Details for the file minet-0.35.4-py3-none-any.whl.
File metadata
- Download URL: minet-0.35.4-py3-none-any.whl
- Upload date:
- Size: 91.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 77d82ccd3689d36258ea0dd796ba60c491f0f7196b14d728d3a416eddbc69e0e |
|
| MD5 | 3027de754725c1782a5c63dfce60dea2 |
|
| BLAKE2b-256 | 5bc6c7ff42051ead30eab8f6caa470fcb5d8f27063073be6c4a1ec4275b99443 |
