Scrapy-based Web Crawler with an UI
Project description
Arachnado
Arachnado is a tool to crawl a specific website. It provides a Tornado-based HTTP API and a web UI for a Scrapy-based crawler.
License is MIT.
Install
Arachnado requires Python 2.7. To install Arachnado use pip:
pip install arachnado
To install Arachnado with MongoDB support use this command:
pip install arachnado[mongo]
Run
To start Arachnado execute arachnado command:
arachnado
and then visit http://0.0.0.0:8888 (or whatever URL is configured).
To see available command-line options use
arachnado –help
Arachnado can be configured using a config file. Put it to one of the common locations (‘/etc/arachnado.conf’, ‘~/.config/arachnado.conf’ or ‘~/.arachnado.conf’) or pass the file name as an argument when starting the server:
arachnado --config ./my-config.conf
For available options check https://github.com/TeamHG-Memex/arachnado/blob/master/arachnado/settings/defaults.conf.
Development
Source code: https://github.com/TeamHG-Memex/arachnado
Issue tracker: https://github.com/TeamHG-Memex/arachnado/issues
To build Arachnado static assets node.js + npm are required. Install all JavaScript requirements using npm - run the following command from the repo root:
npm install
then rebuild static files (we use Webpack):
npm run build
or auto-build static files on each change during development:
npm run watch
Changes
0.2 (2015-08-07)
Initial release.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file arachnado-0.2.tar.gz
.
File metadata
- Download URL: arachnado-0.2.tar.gz
- Upload date:
- Size: 140.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9125eba9e911caba1434462606eed212c4e4998720dc04a67669bffc4961a9fd |
|
MD5 | b50fddf0e9c7b26b905216b9f3ba8b2c |
|
BLAKE2b-256 | 47644b5f813d98099aff09c763450b56cf1b6e8fcbf8b7178ada7f6bea6e4fda |
File details
Details for the file arachnado-0.2-py2-none-any.whl
.
File metadata
- Download URL: arachnado-0.2-py2-none-any.whl
- Upload date:
- Size: 156.3 kB
- Tags: Python 2
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62271c8c3b240e60d90d57228faada0dcd1254cc1eab749060a097409d19272a |
|
MD5 | 23a354bc270bbf9159fce343aff2b3a6 |
|
BLAKE2b-256 | 5e5c0301f345ef205f1680dd595c2eb700af3b7a61b2626ee679d524a343b4c7 |