Scrapy http proxy middleware that gets proxy parameters from settings
Project description
Scrapy middlewares that reads proxy config from settings.
Install
pip install proxy-middleware
ProxyFromSettingsMiddleware
A middleware that sets proxy from settings file.
Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:
DOWNLOADER_MIDDLEWARES = { 'proxy_middleware.ProxyFromSettingsMiddleware': 10, ...
Pass proxy config via HTTP_PROXY and HTTPS_PROXY settings variables. HTTPPROXY_AUTH_ENCODING is also respected:
scrapy crawl my-spider -s HTTP_PROXY=http://localhost:8118
ProxyOnlyTorMiddleware
A middleware that proxies only requests to TOR sites (domain ends with “.onion”).
Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:
DOWNLOADER_MIDDLEWARES = { 'proxy_middleware.ProxyOnlyTorMiddleware': 10, ...
Settings: HTTP_PROXY for an http proxy and HTTPS_PROXY for an https proxy.
License
License is MIT.
History
0.2.0 (2017-05-29)
ProxyOnlyTorMiddleware added.
0.1.1 (2017-04-07)
Initial release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
proxy-middleware-0.2.0.tar.gz
(2.4 kB
view hashes)