Manage outbound HTTP connections using Curl & CurlMulti
Project description
Manage outbound HTTP connections using Curl & CurlMulti
Description
mcurl is a Python wrapper for libcurl with a high-level API that makes it easy to interact with the libcurl easy and multi interfaces. It was originally created for the Px proxy server which uses libcurl to handle upstream proxy authentication.
Usage
mcurl can be installed using pip:
pip install pymcurl
Binary packages are provided the following platforms:
- aarch64-linux-gnu
- aarch64-linux-musl
- i686-linux-gnu
- x86_64-linux-gnu
- x86_64-linux-musl
- x86_64-macos
- x86_64-windows
mcurl leverages cffi to interface with libcurl and all binary dependencies are sourced from binarybuilder.org. auditwheel on Linux, delocate on MacOS and delvewheel on Windows are used to bundle the shared libraries into the wheels.
Thanks to cffi and Py_LIMITED_API, these mcurl binaries should work on any Python from v3.2 onwards.
Easy interface
from mcurl import Curl
c = Curl('http://httpbin.org/get')
c.buffer()
ret = c.perform()
if ret == 0:
ret, resp = c.get_response()
headers = c.get_headers()
data = c.get_data()
print(f"Response: {resp}\n\n{headers}{data}")
Multi interface
from mcurl import Curl, MCurl
m = MCurl()
c1 = Curl('http://httpbin.org/get')
c1.buffer()
m.add(c1)
data = "test8192".encode("utf-8")
c2 = Curl('https://httpbin.org/post', 'POST')
c2.buffer(data=data)
c2.set_headers({"Content-Length": len(data)})
m.add(c2)
ret1 = m.do(c1)
ret2 = m.do(c2)
if ret1:
c1.get_response()
c1.get_headers()
c1.get_data()
print(f"Response: {c1.get_response()}\n\n{c1.get_headers()}{c1.get_data()}")
else:
print(f"Failed with error: {c1.errstr}")
if ret2:
c2.get_response()
c2.get_headers()
c2.get_data()
print(f"Response: {c2.get_response()}\n\n{c2.get_headers()}{c2.get_data()}")
else:
print(f"Failed with error: {c2.errstr}")
m.close()
libcurl API
The libcurl API can be directly accessed as is done in mcurl if preferred.
from _libcurl_cffi import lib as libcurl
from _libcurl_cffi import ffi
url = "http://httpbin.org/get"
curl = ffi.new("char []", url.encode("utf-8"))
easy = libcurl.curl_easy_init()
libcurl.curl_easy_setopt(easy, libcurl.CURLOPT_URL, curl)
cerr = libcurl.curl_easy_perform(easy)
API reference
NAME
mcurl - Manage outbound HTTP connections using Curl & CurlMulti
CLASSES
builtins.object
Curl
MCurl
class Curl(builtins.object)
| Curl(url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
|
| Helper class to manage a curl easy instance
|
| Methods defined here:
|
| __del__(self)
| Destructor - clean up resources
|
| __init__(self, url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
| Initialize curl instance
|
| method = GET, POST, PUT, CONNECT, etc.
| request_version = HTTP/1.0, HTTP/1.1, etc.
|
| bridge(self, client_rfile=None, client_wfile=None, client_hfile=None)
| Bridge curl reads/writes to sockets specified
|
| Reads POST/PATCH data from client_rfile
| Writes data back to client_wfile
| Writes headers back to client_hfile
|
| buffer(self, data=None)
| Setup buffers to bridge curl perform
|
| get_activesocket(self)
| Return active socket for this easy instance
|
| get_data(self, encoding='utf-8')
| Return data written by curl perform to buffer()
|
| encoding = "utf-8" by default, change or set to None if bytes preferred
|
| get_headers(self, encoding='utf-8')
| Return headers written by curl perform to buffer()
|
| encoding = "utf-8" by default, change or set to None if bytes preferred
|
| get_primary_ip(self)
| Return primary IP address of this easy instance
|
| get_response(self)
| Return response code of completed request
|
| perform(self)
| Perform the easy handle
|
| reset(self, url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
| Reuse existing curl instance for another request
|
| set_auth(self, user, password=None, auth='ANY')
| Set proxy authentication info - call after set_proxy() to enable auth caching
|
| set_debug(self, enable=True)
| Enable debug output
| Call after set_proxy() and set_auth() to enable discovery and caching of proxy
| auth mechanism - libcurl does not provide an API to get this today - need to
| find it in sent header debug output
|
| set_follow(self, enable=True)
| Set curl to follow 3xx responses
|
| set_headers(self, xheaders)
| Set headers to send
|
| set_insecure(self, enable=True)
| Set curl to ignore SSL errors
|
| set_proxy(self, proxy, port=0, noproxy=None)
| Set proxy options - returns False if this proxy server has auth failures
|
| set_transfer_decoding(self, enable=False)
| Set curl to turn off transfer decoding - let client do it
|
| set_tunnel(self, tunnel=True)
| Set to tunnel through proxy if no proxy or proxy + auth
|
| set_useragent(self, useragent)
| Set user agent to send
|
| set_verbose(self, enable=True)
| Set verbose mode
|
class MCurl(builtins.object)
| MCurl(debug_print=None)
|
| Helper class to manage a curl multi instance
|
| Methods defined here:
|
| __init__(self, debug_print=None)
| Initialize multi interface
|
| add(self, curl: mcurl.Curl)
| Add a Curl handle to perform
|
| close(self)
| Stop any running transfers and close this multi handle
|
| do(self, curl: mcurl.Curl)
| Add a Curl handle and peform until completion
|
| remove(self, curl: mcurl.Curl)
| Remove a Curl handle once done
|
| select(self, curl: mcurl.Curl, client_sock, idle=30)
| Run select loop between client and curl
|
| setopt(self, option, value)
| Configure multi options
|
| stop(self, curl: mcurl.Curl)
| Stop a running curl handle and remove
FUNCTIONS
curl_version()
cvp2pystr(cvoidp)
Convert void * to Python string
debug_callback(easy, infotype, data, size, userp)
Prints out curl debug info and headers sent/received
dprint lambda x
# Debug shortcut
getauth(auth)
Return auth value for specified authentication string
Supported values can be found here: https://curl.se/libcurl/c/CURLOPT_HTTPAUTH.html
Skip the CURLAUTH_ portion in input - e.g. getauth("ANY")
To control which methods are available during proxy detection:
Prefix NO to avoid method - e.g. NONTLM => ANY - NTLM
Prefix SAFENO to avoid method - e.g. SAFENONTLM => ANYSAFE - NTLM
Prefix ONLY to support only that method - e.g ONLYNTLM => ONLY + NTLM
gethash(easy)
Return hash value for easy to allow usage as a dict key
header_callback(buffer, size, nitems, userdata)
multi_timer_callback(multi, timeout_ms, userp)
print_curl_version()
Display curl version information
py2cbool(pbool)
Convert Python bool to long
py2clong(plong)
Convert Python int to long
py2cstr(pstr)
Convert Python string to char *
py2custr(pstr)
Convert Python string to char *
read_callback(buffer, size, nitems, userdata)
sanitized(msg)
Hide user sensitive data from debug output
save_auth(curl, msg)
Find and cache proxy auth mechanism from headers sent by libcurl
save_upstream(curl, msg)
Find which server libcurl connected to - upstream proxy or target server
socket_callback(easy, sock_fd, ev_bitmask, userp, socketp)
sockopt_callback(clientp, sock_fd, purpose)
wa_callback(easy, infotype, data, size, userp)
curl debug callback to get info not provided by libcurl today
- proxy auth mechanism from sent headers
- upstream server connected to from curl info
write_callback(buffer, size, nitems, userdata)
yield_msgs(data, size)
Generator for curl debug messages
Building mcurl
mcurl is built using gcc on Linux, clang on MacOS and mingw-x64 on Windows. The shared libraries are downloaded from binarybuilder.org using jbb for Linux and Windows. MacOS packages the libcurl binaries installed via brew.
build.sh can be used to build mcurl for all supported platforms including Windows.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
File details
Details for the file pymcurl-8.6.0.1-cp32-abi3-win_amd64.whl
.
File metadata
- Download URL: pymcurl-8.6.0.1-cp32-abi3-win_amd64.whl
- Upload date:
- Size: 1.1 MB
- Tags: CPython 3.2+, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36f0b63f62e13221def4d93b98e179c3ff3939ba38e468bcee43102d9187dde4 |
|
MD5 | 369a3bd45a6a725709508e44a6115b93 |
|
BLAKE2b-256 | 2a0b5d204d60864fea399c68c7c4fe401fa4260780067f28e41738f2f141ceec |
File details
Details for the file pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_x86_64.whl
.
File metadata
- Download URL: pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_x86_64.whl
- Upload date:
- Size: 3.5 MB
- Tags: CPython 3.2+, musllinux: musl 1.1+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4d346532de0bd76e3eeab817eb584dfa5852fb3db88f4ffde40bf2bd115beee |
|
MD5 | b1bbe2c601e93c6dbb8a65e9069f88ef |
|
BLAKE2b-256 | 96b0cacc60a6904587b63f3c033d0782d05da5f55ad1db85cd274bcb980b66cb |
File details
Details for the file pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_aarch64.whl
.
File metadata
- Download URL: pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_aarch64.whl
- Upload date:
- Size: 3.5 MB
- Tags: CPython 3.2+, musllinux: musl 1.1+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 955a9617bc10d06f58a7f8bab8c711f64ec9e7278389e9133008338681df0698 |
|
MD5 | b0217de9ace2cce32bab018cd4217c59 |
|
BLAKE2b-256 | dce82f3c3b7170d574fb4fd3d19e5925f366f9d5282f5d1dafe8d48a1e5acaca |
File details
Details for the file pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 3.6 MB
- Tags: CPython 3.2+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2b71283a02b55d2807b561d7db2ce9177a0016240e7da0f2c1cc12834d5f8aa |
|
MD5 | b80649b6e5dd1bac9a0e37cb784c50b3 |
|
BLAKE2b-256 | ca567f3b1db0627b00050d0a07df3596c47f8eab1a9b9ec4fbf6027fa5743fb0 |
File details
Details for the file pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
.
File metadata
- Download URL: pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
- Upload date:
- Size: 3.4 MB
- Tags: CPython 3.2+, manylinux: glibc 2.17+ i686
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50c2d3f8aba360a818e356a4e5fa4dbd100dae3a6c00c88cedf004be452227a8 |
|
MD5 | e475dbd82e461b76394861b5be11adc9 |
|
BLAKE2b-256 | 498ad4a479c21db308d37990ba207557384130c64c2d66bef4008a34b463cfb2 |
File details
Details for the file pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
.
File metadata
- Download URL: pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 3.6 MB
- Tags: CPython 3.2+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c18136c013567897189f2fb06f815c2d83ad3e8abf8357f8b45d55b5098044b |
|
MD5 | 26d3adb2360d0d35765091198890eee6 |
|
BLAKE2b-256 | 2c13f8de31acc13d6ddb9eed84e8b88f670a747b90b1b8ec212b798db1a128fb |
File details
Details for the file pymcurl-8.6.0.1-cp32-abi3-macosx_12_0_x86_64.whl
.
File metadata
- Download URL: pymcurl-8.6.0.1-cp32-abi3-macosx_12_0_x86_64.whl
- Upload date:
- Size: 4.4 MB
- Tags: CPython 3.2+, macOS 12.0+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d927690b145877e2bb648de16212bd3691a9df1ba402cc1870002b1e977c46d |
|
MD5 | cdd935302e104a12fd070757b9824d86 |
|
BLAKE2b-256 | f3d47e41bdfe06f6ba5707fc76ecba6a74aa6b7c8e3d139c918f74b5a45fc27e |