A webmining CLI tool & library for python.
Project description
minet is a webmining CLI tool & library for python that can be used to collect and extract data from a large variety of web sources such as raw webpages, Facebook, CrowdTangle, YouTube, Twitter, Media Cloud etc.
It adopts a lo-fi approach to various webmining problems by letting you perform a variety of actions from the comfort of your command line. No database needed: raw data files such as CSV should be sufficient to do the work.
In addition, minet also exposes its high-level programmatic interface as a python library so you can tweak its behavior at will.
Use cases
- Downloading large amount of urls very fast. (example)
- Writing scrapers to extract structured data from HTML pages.
- Writing crawlers to automatically browse the web.
- Extract raw text content from HTML pages. (example)
- Normalize batches of urls contained in a CSV file to perform relevant aggregations (dropping irrelevant query items, extracting domain name etc.) (example)
- Join two CSV files based on columns containing urls needing to be organized hierarchically.
- Collecting data from CrowdTangle API (to collect and search posts mainly from Facebook and Instagram).
- Collecting data from Facebook (comments, likes etc.)
- Parsing Facebook urls in a CSV file.
- Collecting data from Twitter (users, followers, followees etc.)
- Scraping data (tweets etc.) from Twitter's website public facing search API.
- Collecting data from YouTube (captions, comments, video metadata etc.)
- Parsing YouTube urls in a CSV file.
- Dumping a Hyphe corpus.
- Collecting data from Media Cloud (search stories, dump topics etc.).
Features (from a technical standpoint)
- Multithreaded, memory-efficient fetching from the web.
- Multithreaded, scalable crawling using a comfy DSL.
- Multiprocessed raw text content extraction from HTML pages.
- Multiprocessed scraping from HTML pages using a comfy DSL.
- URL-related heuristics utilities such as extraction, normalization and matching.
- Data collection from various APIs such as CrowdTangle.
Installation
minet can be installed as a standalone CLI tool (currently only on mac, ubuntu & similar) by running the following command in your terminal:
curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash
Don't trust us enough to pipe the result of a HTTP request into bash? We wouldn't either, so feel free to read the installation script here and run it on your end if you prefer.
On ubuntu & similar you might need to install curl before running the installation script if you don't already have it:
sudo apt-get install curl
Else, minet can be installed directly as a python CLI tool and library using pip:
pip install minet
If you need more help to install and use minet from scratch, you can check those installation documents.
Finally if you want to install the standalone binaries by yourself (even for windows) you can find them in each release here.
Upgrading
To upgrade the standalone version, simply run the install script once again:
curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash
To upgrade the python version you can use pip thusly:
pip install -U minet
Uninstallation
To uninstall the standalone version:
curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/uninstall.sh | bash
To uninstall the python version:
pip uninstall minet
Cookbook
To learn how to use minet and understand how it may fit your use cases, you should definitely check out our Cookbook.
Usage
Contributing
To contribute to minet you can check out this documentation.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file minet-0.41.2.tar.gz.
File metadata
- Download URL: minet-0.41.2.tar.gz
- Upload date:
- Size: 69.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.6.1 requests/2.25.1 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 6522c70d6a7c0765eff14cb6a527fb134a41386357c2f4d038340b186d439d1b |
|
| MD5 | 2bfe0ed3a88f13962c0c27a9deb01f2a |
|
| BLAKE2b-256 | 9c7fa805d2a0324ab91ecf906258bf20eae101cd066ffe87f67c2ca7319715dc |
Provenance
File details
Details for the file minet-0.41.2-py3-none-any.whl.
File metadata
- Download URL: minet-0.41.2-py3-none-any.whl
- Upload date:
- Size: 114.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.6.1 requests/2.25.1 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | bf1959ea21733f59af83541c01f6e8088f7f8b5f54bda8edcd876cb9fae578d0 |
|
| MD5 | a637b8f7902a8e31c34d2137e29396ff |
|
| BLAKE2b-256 | e2473fd592e80d89d82c638d784a887d23cb6a7c6ea422fa9cc4430c937e6f0d |
