Skip to main content

Scrapy-based Web Crawler with an UI

Project description

Arachnado

Arachnado is a tool to crawl a specific website. It provides a Tornado-based HTTP API and a web UI for a Scrapy-based crawler.

License is MIT.

Install

Arachnado requires Python 2.7. To install Arachnado use pip:

pip install arachnado

To install Arachnado with MongoDB support use this command:

pip install arachnado[mongo]

Run

To start Arachnado execute arachnado command:

arachnado

and then visit http://0.0.0.0:8888 (or whatever URL is configured).

To see available command-line options use

arachnado –help

Arachnado can be configured using a config file. Put it to one of the common locations (‘/etc/arachnado.conf’, ‘~/.config/arachnado.conf’ or ‘~/.arachnado.conf’) or pass the file name as an argument when starting the server:

arachnado --config ./my-config.conf

For available options check https://github.com/TeamHG-Memex/arachnado/blob/master/arachnado/settings/defaults.conf.

Development

To build Arachnado static assets node.js + npm are required. Install all JavaScript requirements using npm - run the following command from the repo root:

npm install

then rebuild static files (we use Webpack):

npm run build

or auto-build static files on each change during development:

npm run watch

Changes

0.2 (2015-08-07)

Initial release.

Project details


Release history Release notifications | RSS feed

This version

0.2

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

arachnado-0.2.tar.gz (140.0 kB view details)

Uploaded Source

Built Distribution

arachnado-0.2-py2-none-any.whl (156.3 kB view details)

Uploaded Python 2

File details

Details for the file arachnado-0.2.tar.gz.

File metadata

  • Download URL: arachnado-0.2.tar.gz
  • Upload date:
  • Size: 140.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for arachnado-0.2.tar.gz
Algorithm Hash digest
SHA256 9125eba9e911caba1434462606eed212c4e4998720dc04a67669bffc4961a9fd
MD5 b50fddf0e9c7b26b905216b9f3ba8b2c
BLAKE2b-256 47644b5f813d98099aff09c763450b56cf1b6e8fcbf8b7178ada7f6bea6e4fda

See more details on using hashes here.

File details

Details for the file arachnado-0.2-py2-none-any.whl.

File metadata

File hashes

Hashes for arachnado-0.2-py2-none-any.whl
Algorithm Hash digest
SHA256 62271c8c3b240e60d90d57228faada0dcd1254cc1eab749060a097409d19272a
MD5 23a354bc270bbf9159fce343aff2b3a6
BLAKE2b-256 5e5c0301f345ef205f1680dd595c2eb700af3b7a61b2626ee679d524a343b4c7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page