Skip to main content

Manage outbound HTTP connections using Curl & CurlMulti

Project description

Manage outbound HTTP connections using Curl & CurlMulti

Description

mcurl is a Python wrapper for libcurl with a high-level API that makes it easy to interact with the libcurl easy and multi interfaces. It was originally created for the Px proxy server which uses libcurl to handle upstream proxy authentication.

Usage

mcurl can be installed using pip:

pip install pymcurl

Binary packages are provided the following platforms:

  • aarch64-linux-gnu
  • aarch64-linux-musl
  • i686-linux-gnu
  • x86_64-linux-gnu
  • x86_64-linux-musl
  • x86_64-macos
  • x86_64-windows

mcurl leverages cffi to interface with libcurl and all binary dependencies are sourced from binarybuilder.org. auditwheel on Linux, delocate on MacOS and delvewheel on Windows are used to bundle the shared libraries into the wheels.

Thanks to cffi and Py_LIMITED_API, these mcurl binaries should work on any Python from v3.2 onwards.

Easy interface

from mcurl import Curl

c = Curl('http://httpbin.org/get')
c.buffer()
ret = c.perform()
if ret == 0:
    ret, resp = c.get_response()
    headers = c.get_headers()
    data = c.get_data()
    print(f"Response: {resp}\n\n{headers}{data}")

Multi interface

from mcurl import Curl, MCurl

m = MCurl()

c1 = Curl('http://httpbin.org/get')
c1.buffer()
m.add(c1)

data = "test8192".encode("utf-8")
c2 = Curl('https://httpbin.org/post', 'POST')
c2.buffer(data=data)
c2.set_headers({"Content-Length": len(data)})
m.add(c2)

ret1 = m.do(c1)
ret2 = m.do(c2)

if ret1:
    c1.get_response()
    c1.get_headers()
    c1.get_data()
    print(f"Response: {c1.get_response()}\n\n{c1.get_headers()}{c1.get_data()}")
else:
    print(f"Failed with error: {c1.errstr}")

if ret2:
    c2.get_response()
    c2.get_headers()
    c2.get_data()
    print(f"Response: {c2.get_response()}\n\n{c2.get_headers()}{c2.get_data()}")
else:
    print(f"Failed with error: {c2.errstr}")

m.close()

libcurl API

The libcurl API can be directly accessed as is done in mcurl if preferred.

from _libcurl_cffi import lib as libcurl
from _libcurl_cffi import ffi

url = "http://httpbin.org/get"
curl = ffi.new("char []", url.encode("utf-8"))

easy = libcurl.curl_easy_init()
libcurl.curl_easy_setopt(easy, libcurl.CURLOPT_URL, curl)
cerr = libcurl.curl_easy_perform(easy)

API reference

NAME
    mcurl - Manage outbound HTTP connections using Curl & CurlMulti

CLASSES
    builtins.object
        Curl
        MCurl

    class Curl(builtins.object)
     |  Curl(url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
     |
     |  Helper class to manage a curl easy instance
     |
     |  Methods defined here:
     |
     |  __del__(self)
     |      Destructor - clean up resources
     |
     |  __init__(self, url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
     |      Initialize curl instance
     |
     |      method = GET, POST, PUT, CONNECT, etc.
     |      request_version = HTTP/1.0, HTTP/1.1, etc.
     |
     |  bridge(self, client_rfile=None, client_wfile=None, client_hfile=None)
     |      Bridge curl reads/writes to sockets specified
     |
     |      Reads POST/PATCH data from client_rfile
     |      Writes data back to client_wfile
     |      Writes headers back to client_hfile
     |
     |  buffer(self, data=None)
     |      Setup buffers to bridge curl perform
     |
     |  get_activesocket(self)
     |      Return active socket for this easy instance
     |
     |  get_data(self, encoding='utf-8')
     |      Return data written by curl perform to buffer()
     |
     |      encoding = "utf-8" by default, change or set to None if bytes preferred
     |
     |  get_headers(self, encoding='utf-8')
     |      Return headers written by curl perform to buffer()
     |
     |      encoding = "utf-8" by default, change or set to None if bytes preferred
     |
     |  get_primary_ip(self)
     |      Return primary IP address of this easy instance
     |
     |  get_response(self)
     |      Return response code of completed request
     |
     |  perform(self)
     |      Perform the easy handle
     |
     |  reset(self, url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
     |      Reuse existing curl instance for another request
     |
     |  set_auth(self, user, password=None, auth='ANY')
     |      Set proxy authentication info - call after set_proxy() to enable auth caching
     |
     |  set_debug(self, enable=True)
     |      Enable debug output
     |        Call after set_proxy() and set_auth() to enable discovery and caching of proxy
     |        auth mechanism - libcurl does not provide an API to get this today - need to
     |        find it in sent header debug output
     |
     |  set_follow(self, enable=True)
     |      Set curl to follow 3xx responses
     |
     |  set_headers(self, xheaders)
     |      Set headers to send
     |
     |  set_insecure(self, enable=True)
     |      Set curl to ignore SSL errors
     |
     |  set_proxy(self, proxy, port=0, noproxy=None)
     |      Set proxy options - returns False if this proxy server has auth failures
     |
     |  set_transfer_decoding(self, enable=False)
     |      Set curl to turn off transfer decoding - let client do it
     |
     |  set_tunnel(self, tunnel=True)
     |      Set to tunnel through proxy if no proxy or proxy + auth
     |
     |  set_useragent(self, useragent)
     |      Set user agent to send
     |
     |  set_verbose(self, enable=True)
     |      Set verbose mode
     |

    class MCurl(builtins.object)
     |  MCurl(debug_print=None)
     |
     |  Helper class to manage a curl multi instance
     |
     |  Methods defined here:
     |
     |  __init__(self, debug_print=None)
     |      Initialize multi interface
     |
     |  add(self, curl: mcurl.Curl)
     |      Add a Curl handle to perform
     |
     |  close(self)
     |      Stop any running transfers and close this multi handle
     |
     |  do(self, curl: mcurl.Curl)
     |      Add a Curl handle and peform until completion
     |
     |  remove(self, curl: mcurl.Curl)
     |      Remove a Curl handle once done
     |
     |  select(self, curl: mcurl.Curl, client_sock, idle=30)
     |      Run select loop between client and curl
     |
     |  setopt(self, option, value)
     |      Configure multi options
     |
     |  stop(self, curl: mcurl.Curl)
     |      Stop a running curl handle and remove

FUNCTIONS
    curl_version()

    cvp2pystr(cvoidp)
        Convert void * to Python string

    debug_callback(easy, infotype, data, size, userp)
        Prints out curl debug info and headers sent/received

    dprint lambda x
        # Debug shortcut

    getauth(auth)
        Return auth value for specified authentication string

        Supported values can be found here: https://curl.se/libcurl/c/CURLOPT_HTTPAUTH.html

        Skip the CURLAUTH_ portion in input - e.g. getauth("ANY")

        To control which methods are available during proxy detection:
          Prefix NO to avoid method - e.g. NONTLM => ANY - NTLM
          Prefix SAFENO to avoid method - e.g. SAFENONTLM => ANYSAFE - NTLM
          Prefix ONLY to support only that method - e.g ONLYNTLM => ONLY + NTLM

    gethash(easy)
        Return hash value for easy to allow usage as a dict key

    header_callback(buffer, size, nitems, userdata)

    multi_timer_callback(multi, timeout_ms, userp)

    print_curl_version()
        Display curl version information

    py2cbool(pbool)
        Convert Python bool to long

    py2clong(plong)
        Convert Python int to long

    py2cstr(pstr)
        Convert Python string to char *

    py2custr(pstr)
        Convert Python string to char *

    read_callback(buffer, size, nitems, userdata)

    sanitized(msg)
        Hide user sensitive data from debug output

    save_auth(curl, msg)
        Find and cache proxy auth mechanism from headers sent by libcurl

    save_upstream(curl, msg)
        Find which server libcurl connected to - upstream proxy or target server

    socket_callback(easy, sock_fd, ev_bitmask, userp, socketp)

    sockopt_callback(clientp, sock_fd, purpose)

    wa_callback(easy, infotype, data, size, userp)
        curl debug callback to get info not provided by libcurl today
        - proxy auth mechanism from sent headers
        - upstream server connected to from curl info

    write_callback(buffer, size, nitems, userdata)

    yield_msgs(data, size)
        Generator for curl debug messages

Building mcurl

mcurl is built using gcc on Linux, clang on MacOS and mingw-x64 on Windows. The shared libraries are downloaded from binarybuilder.org using jbb for Linux and Windows. MacOS packages the libcurl binaries installed via brew.

build.sh can be used to build mcurl for all supported platforms including Windows.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

pymcurl-8.6.0.1-cp32-abi3-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.2+ Windows x86-64

pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_x86_64.whl (3.5 MB view details)

Uploaded CPython 3.2+ musllinux: musl 1.1+ x86-64

pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.2+ musllinux: musl 1.1+ ARM64

pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.2+ manylinux: glibc 2.17+ x86-64

pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (3.4 MB view details)

Uploaded CPython 3.2+ manylinux: glibc 2.17+ i686

pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.6 MB view details)

Uploaded CPython 3.2+ manylinux: glibc 2.17+ ARM64

pymcurl-8.6.0.1-cp32-abi3-macosx_12_0_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.2+ macOS 12.0+ x86-64

File details

Details for the file pymcurl-8.6.0.1-cp32-abi3-win_amd64.whl.

File metadata

  • Download URL: pymcurl-8.6.0.1-cp32-abi3-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.2+, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.12.2

File hashes

Hashes for pymcurl-8.6.0.1-cp32-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 36f0b63f62e13221def4d93b98e179c3ff3939ba38e468bcee43102d9187dde4
MD5 369a3bd45a6a725709508e44a6115b93
BLAKE2b-256 2a0b5d204d60864fea399c68c7c4fe401fa4260780067f28e41738f2f141ceec

See more details on using hashes here.

File details

Details for the file pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 e4d346532de0bd76e3eeab817eb584dfa5852fb3db88f4ffde40bf2bd115beee
MD5 b1bbe2c601e93c6dbb8a65e9069f88ef
BLAKE2b-256 96b0cacc60a6904587b63f3c033d0782d05da5f55ad1db85cd274bcb980b66cb

See more details on using hashes here.

File details

Details for the file pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for pymcurl-8.6.0.1-cp32-abi3-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 955a9617bc10d06f58a7f8bab8c711f64ec9e7278389e9133008338681df0698
MD5 b0217de9ace2cce32bab018cd4217c59
BLAKE2b-256 dce82f3c3b7170d574fb4fd3d19e5925f366f9d5282f5d1dafe8d48a1e5acaca

See more details on using hashes here.

File details

Details for the file pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d2b71283a02b55d2807b561d7db2ce9177a0016240e7da0f2c1cc12834d5f8aa
MD5 b80649b6e5dd1bac9a0e37cb784c50b3
BLAKE2b-256 ca567f3b1db0627b00050d0a07df3596c47f8eab1a9b9ec4fbf6027fa5743fb0

See more details on using hashes here.

File details

Details for the file pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 50c2d3f8aba360a818e356a4e5fa4dbd100dae3a6c00c88cedf004be452227a8
MD5 e475dbd82e461b76394861b5be11adc9
BLAKE2b-256 498ad4a479c21db308d37990ba207557384130c64c2d66bef4008a34b463cfb2

See more details on using hashes here.

File details

Details for the file pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pymcurl-8.6.0.1-cp32-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1c18136c013567897189f2fb06f815c2d83ad3e8abf8357f8b45d55b5098044b
MD5 26d3adb2360d0d35765091198890eee6
BLAKE2b-256 2c13f8de31acc13d6ddb9eed84e8b88f670a747b90b1b8ec212b798db1a128fb

See more details on using hashes here.

File details

Details for the file pymcurl-8.6.0.1-cp32-abi3-macosx_12_0_x86_64.whl.

File metadata

File hashes

Hashes for pymcurl-8.6.0.1-cp32-abi3-macosx_12_0_x86_64.whl
Algorithm Hash digest
SHA256 2d927690b145877e2bb648de16212bd3691a9df1ba402cc1870002b1e977c46d
MD5 cdd935302e104a12fd070757b9824d86
BLAKE2b-256 f3d47e41bdfe06f6ba5707fc76ecba6a74aa6b7c8e3d139c918f74b5a45fc27e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page