Scrapy extenstion to control spiders using JSON-RPC
Project description
scrapy-jsonrpc is an extension to control a running Scrapy web crawler via JSON-RPC. The service provides access to the main Crawlera object via the JSON-RPC 2.0 protocol.
It’s enabled by the JSONRPC_ENABLED setting. The web server will listen in the port specified in JSONRPC_PORT, and will log to the file specified in JSONRPC_LOGFILE.
The endpoint for accessing the crawler object is:
http://localhost:6080/crawler
Example client
There is a command line tool provided for illustration purposes on how to build a client. You can find it in example-client.py. It supports a few basic commands such as listing the running spiders, etc.
Settings
These are the settings that control the web service behaviour:
JSONRPC_ENABLED
Default: True
A boolean which specifies if the web service will be enabled (provided its extension is also enabled).
JSONRPC_LOGFILE
Default: None
A file to use for logging HTTP requests made to the web service. If unset web the log is sent to standard scrapy log.
JSONRPC_PORT
Default: [6080, 7030]
The port range to use for the web service. If set to None or 0, a dynamically assigned port is used.
JSONRPC_HOST
Default: '127.0.0.1'
The interface the web service should listen on.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file scrapy-jsonrpc-0.2.0.tar.gz.
File metadata
- Download URL: scrapy-jsonrpc-0.2.0.tar.gz
- Upload date:
- Size: 5.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7247a35a77d99e20830f6a6341cf06058396481b5a004fcbc9ddb56f29b8ff44 |
|
| MD5 | 28570720fe74ee3b3795197fd0105d64 |
|
| BLAKE2b-256 | 51235e860a823012f72e21dcc76d211d56732c87f9fa26984e2b3ccebe19182f |
Provenance
File details
Details for the file scrapy_jsonrpc-0.2.0-py2-none-any.whl.
File metadata
- Download URL: scrapy_jsonrpc-0.2.0-py2-none-any.whl
- Upload date:
- Size: 6.9 kB
- Tags: Python 2
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3610403c4e7954a410c35afb9742aec3e2ffc5730e91fbea7621bcda5b25c355 |
|
| MD5 | fcf3fde0b3c17b94bbba5487171b8b34 |
|
| BLAKE2b-256 | 7f1d21b2e86dc423510441bc490e7bbb7092fe21e73de18e9f14aea86056a89d |
