Scrapy http proxy middleware that gets proxy parameters from settings
Project description
Scrapy middlewares that reads proxy config from settings.
Install
pip install proxy-middleware
ProxyFromSettingsMiddleware
A middleware that sets proxy from settings file.
Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:
DOWNLOADER_MIDDLEWARES = {
'proxy_middleware.ProxyFromSettingsMiddleware': 10,
...
Pass proxy config via HTTP_PROXY and HTTPS_PROXY settings variables. HTTPPROXY_AUTH_ENCODING is also respected:
scrapy crawl my-spider -s HTTP_PROXY=http://localhost:8118
ProxyOnlyTorMiddleware
A middleware that proxies only requests to TOR sites (domain ends with “.onion”).
Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:
DOWNLOADER_MIDDLEWARES = {
'proxy_middleware.ProxyOnlyTorMiddleware': 10,
...
Settings: HTTP_PROXY for an http proxy and HTTPS_PROXY for an https proxy.
License
License is MIT.
History
0.2.0 (2017-05-29)
ProxyOnlyTorMiddleware added.
0.1.1 (2017-04-07)
Initial release
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file proxy-middleware-0.2.0.tar.gz.
File metadata
- Download URL: proxy-middleware-0.2.0.tar.gz
- Upload date:
- Size: 2.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2cf6a0531af0cc728208fc247a2e0e33e9ac9ff771d4d4407e5622d8ebe676da |
|
| MD5 | f63e8b339908127e12cdc496af6fef4c |
|
| BLAKE2b-256 | ad910004924204956f08365e4c55dc07ed3f8ced2d23baeede0cceece2177294 |
