A utility for finding login links, forms and autologging into websites with a set of valid credentials.
Project description
Autologin is a library that makes it easier for web spiders to crawl websites that require login. Provide it with credentials and a URL or the html source of a page (normally the homepage), and it will attempt to login for you. Cookies are returned to be used by your spider.
The goal of Autologin is to make it easier for web spiders to crawl websites that require authentication without having to re-write login code for each website.
Autologin can be used as a library, on the command line, or as a service. You can make use of Autologin without generating http requests, so you can drop it right into your spider without worrying about impacting rate limits.
If you are using Scrapy for crawling, check out autologin-middleware, which is a scrapy middleware that uses autologin http-api to maintain a logged-in state for a scrapy spider.
Autologin works on Python 2.7 and 3.3+.
Features
Automatically find login forms and fields
Obtain authenticated cookies
Obtain form requests to submit from your own spider
Extract links to login pages
Use as a library with or without making http requests
Command line client
Web service
UI for managing login credentials
Captcha support
Quickstart
Don’t like reading documentation?
from autologin import AutoLogin url = 'https://reddit.com' username = 'foo' password = 'bar' al = AutoLogin() cookies = al.auth_cookies_from_url(url, username, password)
You now have a cookiejar that you can use in your spider. Don’t want a cookiejar?
cookies.__dict__
You now have a dictionary.
Installation
Install the latest release from PyPI:
$ pip install -U autologin
or the version with the latest changes from Github:
$ pip install git+https://github.com/TeamHG-Memex/autologin.git
Autologin depends on Formasaurus for field and form classification, which has quite a lot of dependencies. These packages may require extra steps to install, so the command above may fail. In this case install dependencies manually, one by one (follow their install instructions).
A recent pip is recommended (update it with pip install pip -U). On Ubuntu, the following packages are required:
$ apt-get install build-essential libssl-dev libffi-dev \ libxml2-dev libxslt1-dev \ python-dev # or python3-dev for python 3
Login request
This method extracts the login form (if there is one), fills the fields and returns a dictionary with the form url and args for your spider to submit. No http requests are made:
>>> al.login_request(html_source, username, password, base_url=None) {'body': 'login=admin&password=secret', 'headers': {b'Content-Type': b'application/x-www-form-urlencoded'}, 'method': 'POST', 'url': '/login'}
Relative form action will be resolved against the base_url.
Command Line
$ autologin usage: autologin [-h] [--splash-url SPLASH_URL] [--http-proxy HTTP_PROXY] [--https-proxy HTTPS_PROXY] [--extra-js EXTRA_JS] [--show-in-browser] username password url
HTTP API
You can start the autologin HTTP API with:
$ autologin-http-api
and use /login-cookies endpoint. Make a POST request with JSON body. The following arguments are supported:
url (required): url of the site where we would like to login
username (optional): if not provided, it will be fetched from the login keychain
password (optional): if not provided, it will be fetched from the login keychain
extra_js (optional, experimental) is a string with an extra JS script that should be executed on the login page before making a POST request. For example, it can be used to accept cookie use. It is supported only when SPLASH_URL is also given in settings.
settings (optional) - a dictionary with Scrapy settings to override, useful values are described above.
If username and password are not provided, autologin tries to find them in the login keychain. If no matching credentials are found (they are matched by domain, not by precise url), then human is expected to eventually provide them in the keychain UI, or mark domain as “skipped”.
Response is JSON with a status field with the following possible values:
error status means an error occured, error field has more info
skipped means that domain is maked as “skipped” in keychain UI
pending means there is an item in keychain UI (or it was just created), and no credentials have been entered yet
solved means that cookies were obtained, they are returned in the cookies field, in Cookie.__dict__ format.
Captcha support
There is experimental captcha support: if the login form contains a captcha, we will try to solve it using an external service (DeathByCaptcha), and will submit it as part of login request. This does not affect API in any way, you only have to provide environment variables with your DeathByCaptcha account details: DEATHBYCAPTCHA_USERNAME and DEATHBYCAPTCHA_PASSWORD. This applies to all APIs: autologin-http-api, autologin, and the Python API.
You also need to install the decaptcha library:
pip install git+https://github.com/TeamHG-Memex/decaptcha.git
Support is still experimental, new Google ReCaptcha/NoCaptcha are not supported. Also, it currently works only with splash (when SPLASH_URL is passed in settings).
Keychain UI
Start keychain UI with:
$ autologin-server
Note that both autologin-server and autologin-http-api are not protected by any authentication.
Contributors
Source code and bug tracker are on github: https://github.com/TeamHG-Memex/autologin.
Run tests with tox:
$ tox
Splash support is not tested directly here, but there are indirect tests for it in the undercrawler test suite.
License
License is MIT.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for autologin-0.1.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c880eb1d71fd9f7dcc90037618502e79389001478189377ffc1271603ff8a0d |
|
MD5 | 64401564dc9d29787ea53be2baddcdd2 |
|
BLAKE2b-256 | 6dc4f48123db6a6c3aac7dd754c431fc20b6ef7a1ad71eb9a4d6685656fa7632 |