A zope.testbrowser extension with useragent faking and proxy abilities
Project description
Introduction
This Yet-Another-Mechanize implementation aims to give the developper those new features:
It can be proxified
It fakes user agent by default
It does not handle robots by default
There is a ‘real” modification which uses an underlying moz repl server to control a distance firefox instance
It uses sys.prefix/etc/config.ini with a part [collective.anonymousbrowser] for its settings:
[collective.anonymousbrowser] proxies= ; for a mozrepl server host = localhost port = 4242
This file is generated at the first run without proxies. It s your own to feed it with some open proxies.
Of course, it can take another configuration file, please see the __init__ method.
TODO
lxml integration, maybe steal z3c.etestbrowser
Tests and Handbook
First, we need to instantiate the sources where we come from:
>>> import BaseHTTPServer >>> from SimpleHTTPServer import SimpleHTTPRequestHandler >>> from collective.anonymousbrowser.browser import Browser, FF2_USERAGENT >>> from threading import Thread
Run a basic request printers to check user agent and further requests:
>>> class ReqHandler(SimpleHTTPRequestHandler): ... def do_GET(self): ... self.end_headers() ... self.send_response(200, '\n\n<html>%s</html>' % self.headers) >>> httpd0 = BaseHTTPServer.HTTPServer(('', 45675,) , ReqHandler) >>> httpd1 = BaseHTTPServer.HTTPServer(('', 45676,) , ReqHandler) >>> httpd2 = BaseHTTPServer.HTTPServer(('', 45677,) , ReqHandler) >>> httpd3 = BaseHTTPServer.HTTPServer(('', 45678,) , ReqHandler) >>> httpd4 = BaseHTTPServer.HTTPServer(('', 45679,) , ReqHandler) >>> for item in (httpd0, httpd1, httpd2, httpd3, httpd4): ... t = Thread(target=item.serve_forever) ... t.setDaemon(True) ... t.start()
User Agent
Oh, my god, we have a brand new user agent by default:
>>> br = Browser() >>> br.open('http://localhost:45678') >>> FF2_USERAGENT in br.contents True >>> br2 = Browser('http://localhost:45678') >>> FF2_USERAGENT in br2.contents True
Proxy mode
But, we want to be anonymous, and we ll set a proxy To define those proxies, just just a config.ini file like:
[collective.anonymousbrowser] proxies = host1:port host2:port
When the browser has many proxies defined, it will circly through those ones. But, it will not use the same host indefinitivly, just set the proxy_max_use argument:
>>> from StringIO import StringIO >>> from tempfile import mkstemp >>> __, config = mkstemp() >>> open(config, 'w').write("""[collective.anonymousbrowser] ... proxies = ... 127.0.0.1:45675 ... 127.0.0.1:45676 ... 127.0.0.1:45677 ... 127.0.0.1:45678 ... 127.0.0.1:45679 ... """) >>> b = Browser(config=config) >>> b._config._sections {'collective.anonymousbrowser': {'__name__': 'collective.anonymousbrowser', 'proxies': '\n127.0.0.1:45675\n127.0.0.1:45676\n127.0.0.1:45677\n127.0.0.1:45678\n127.0.0.1:45679'}} >>> b.proxies ['127.0.0.1:45675', '127.0.0.1:45676', '127.0.0.1:45677', '127.0.0.1:45678', '127.0.0.1:45679'] >>> b.proxified True >>> b.open('http://localhost:45678') >>> 'Host: localhost:45678' in b.contents True >>> b._lastproxy['count'] == 1 and b._lastproxy['proxy'] in [0,1,2,3,4] True
We can have a normal unproxified browser too
>>> b1 = Browser(proxify=False) >>> b1.proxified False
Next thing to verify is that we have our pseudo-random loop running First thing is we will choose 2 times the 2nd proxy, then the third And of course, we will set the mocker to change the proxy at each row.:
>>> import mocker >>> import random >>> mocked = mocker.Mocker() >>> custom_random_int = mocked.replace('random.randint') >>> custom_random_int(0, 4) <mocker.Mock ... >>> mocked.result(2) >>> custom_random_int(0,1) <mocker.Mock ... >>> mocked.result(0) >>> custom_random_int(0, 4) <mocker.Mock ... >>> mocked.result(2) >>> custom_random_int(0,1) <mocker.Mock ... >>> mocked.result(0) >>> custom_random_int(0, 4) <mocker.Mock ... >>> mocked.result(2) >>> custom_random_int(0,1) <mocker.Mock ... >>> mocked.result(0) >>> custom_random_int(0, 4) <mocker.Mock ... >>> mocked.result(3) >>> custom_random_int(0,1) <mocker.Mock ... >>> mocked.result(0) >>> custom_random_int(0, 4) <mocker.Mock ... >>> mocked.result(4) >>> custom_random_int(0,1) <mocker.Mock ... >>> mocked.result(0) >>> custom_random_int(0, 4) <mocker.Mock ... >>> mocked.result(2) >>> custom_random_int(0,1) <mocker.Mock ... >>> mocked.result(0) >>> custom_random_int(0, 4) <mocker.Mock ... >>> mocked.result(1) >>> custom_random_int(0,1) <mocker.Mock ... >>> mocked.result(0) >>> mocked.replay() >>> b = Browser('http://localhost:45678', config=config) >>> b.open('http://localhost:45678') >>> b._lastproxy {'count': 1, 'proxy': 2} >>> b.open('http://localhost:45678') >>> b._lastproxy {'count': 2, 'proxy': 2} >>> b.open('http://localhost:45678') >>> b._lastproxy {'count': 3, 'proxy': 2} >>> b.open('http://localhost:45678') >>> b._lastproxy {'count': 1, 'proxy': 0} >>> b.open('http://localhost:45678') >>> b._lastproxy {'count': 1, 'proxy': 3} >>> b.open('http://localhost:45678') >>> b._lastproxy {'count': 1, 'proxy': 4} >>> b.open('http://localhost:45678') >>> b._lastproxy {'count': 1, 'proxy': 2} >>> b.open('http://localhost:45678') >>> b._lastproxy {'count': 1, 'proxy': 1} >>> mocked.restore()
If the proxies are dead, we remove them from the list:
>>> __, config = mkstemp() >>> open(config, 'w').write("""[collective.anonymousbrowser] ... proxies = ... 127.0.0.1:35675 ... 127.0.0.1:35676 ... 127.0.0.1:35677 ... 127.0.0.1:45678 ... """) >>> mybrowser = Browser(config=config) >>> mybrowser.proxies ['127.0.0.1:35675', '127.0.0.1:35676', '127.0.0.1:35677', '127.0.0.1:45678'] >>> mybrowser.open('http://localhost:45678') >>> mybrowser.proxies ['127.0.0.1:45678'] >>> mybrowser.proxies = ['127.0.0.1:34785'] >>> mybrowser.open('http://localhost:45678') Traceback (most recent call last): ... Exception: There are no valid proxies left
The loop is recursion protected. If we return always the same host, so the chooser cannot choose anything else. It will loop until it crashes or it handle the recursion:
>>> def randomint(a,b): ... return 2 >>> import random; random.randint = randomint >>> b2 = Browser(config=config) >>> b2.proxy_max_use 3 >>> b2._lastproxy['count'] 0 >>> b2.chooseProxy() '... >>> b2._lastproxy['count'] 1 >>> b2.chooseProxy() '... >>> b2._lastproxy['count'] 2 >>> b2.chooseProxy() '... >>> b2._lastproxy['count'] 3 >>> b2.chooseProxy() '... >>> b2.chooseProxy() Ho, seems we got the max wills to choose, something has gone wrong '127.0.0.1:35675'
HISTORY
0.4
doc + bugfixes
0.3
adding error message
0.2
Adding proxy fallback facility
0.1
Initial release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for collective.anonymousbrowser-0.5dev-r79038.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68f2ca3e9fbaf12b3106bda24b94ac0faa1676c0247917f5fe3a55fc0b6ebbf1 |
|
MD5 | 3df8f38041eb3445137e521ee61d535b |
|
BLAKE2b-256 | 4a53fcf51a40de4eae3e3248ff5174717764c187c85df30b87908d0cc79156f2 |