Transport Adapters

The toolbelt comes with several different transport adapters for you to use with requests. The transport adapters are all kept in requests_toolbelt.adapters and include

AppEngineAdapter

New in version 0.6.0.

As of version 2.10.0, Requests will be capable of supporting Google’s App Engine platform. In order to use Requests on GAE, however, you will need a custom adapter found here as AppEngineAdapter. There are two ways to take advantage of this support at the moment:

  1. Using the AppEngineAdapter like every other adapter, e.g.,

    import requests
from requests_toolbelt.adapters import appengine

s = requests.Session()
s.mount('http://', appengine.AppEngineAdapter())
s.mount('https://', appengine.AppEngineAdapter())

  2. By monkey-patching requests to always use the provided adapter:

    import requests
from requests_toolbelt.adapters import appengine

appengine.monkeypatch()

If you should need to disable certificate validation when monkeypatching (to force third-party libraries that use Requests to not validate certificates, if they do not provide API surface to do so, for example), you can disable it:

from requests_toolbelt.adapters import appengine
appengine.monkeypatch(validate_certificate=False)

Warning

If validate_certificate is False, the monkeypatched adapter will not validate certificates. This effectively sets the validate_certificate argument to urlfetch.Fetch() to False. You should avoid using this wherever possible. Details can be found in the documentation for urlfetch.Fetch().

class requests_toolbelt.adapters.appengine.AppEngineAdapter(validate_certificate=True, *args, **kwargs)

The transport adapter for Requests to use urllib3’s GAE support.

Implements Requests’s HTTPAdapter API.

When deploying to Google’s App Engine service, some of Requests’ functionality is broken. There is underlying support for GAE in urllib3. This functionality, however, is opt-in and needs to be enabled explicitly for Requests to be able to use it.

FingerprintAdapter

New in version 0.4.0.

By default, requests will validate a server’s certificate to ensure a connection is secure. In addition to this, the user can provide a fingerprint of the certificate they’re expecting to receive. Unfortunately, the requests API does not support this fairly rare use-case. When a user needs this extra validation, they should use the FingerprintAdapter class to perform the validation.

class requests_toolbelt.adapters.fingerprint.FingerprintAdapter(fingerprint, **kwargs)

A HTTPS Adapter for Python Requests that verifies certificate fingerprints, instead of certificate hostnames.

Example usage:

import requests
import ssl
from requests_toolbelt.adapters.fingerprint import FingerprintAdapter

twitter_fingerprint = '...'
s = requests.Session()
s.mount(
    'https://twitter.com',
    FingerprintAdapter(twitter_fingerprint)
)

The fingerprint should be provided as a hexadecimal string, optionally containing colons.

SSLAdapter

The SSLAdapter is the canonical implementation of the adapter proposed on Cory Benfield’s blog, here. This adapter allows the user to choose one of the SSL/TLS protocols made available in Python’s ssl module for outgoing HTTPS connections.

In principle, this shouldn’t be necessary: compliant SSL servers should be able to negotiate the required SSL version. In practice there have been bugs in some versions of OpenSSL that mean that this negotiation doesn’t go as planned. It can be useful to be able to simply plug in a Transport Adapter that can paste over the problem.

For example, suppose you’re having difficulty with the server that provides TLS for GitHub. You can work around it by using the following code:

from requests_toolbelt.adapters.ssl import SSLAdapter

import requests
import ssl

s = requests.Session()
s.mount('https://github.com/', SSLAdapter(ssl.PROTOCOL_TLSv1))

Any future requests to GitHub made through that adapter will automatically attempt to negotiate TLSv1, and hopefully will succeed.

class requests_toolbelt.adapters.ssl.SSLAdapter(ssl_version=None, **kwargs)

A HTTPS Adapter for Python Requests that allows the choice of the SSL/TLS version negotiated by Requests. This can be used either to enforce the choice of high-security TLS versions (where supported), or to work around misbehaving servers that fail to correctly negotiate the default TLS version being offered.

Example usage:

>>> import requests
>>> import ssl
>>> from requests_toolbelt import SSLAdapter
>>> s = requests.Session()
>>> s.mount('https://', SSLAdapter(ssl.PROTOCOL_TLSv1))

You can replace the chosen protocol with any that are available in the default Python SSL module. All subsequent requests that match the adapter prefix will use the chosen SSL version instead of the default.

This adapter will also attempt to change the SSL/TLS version negotiated by Requests when using a proxy. However, this may not always be possible: prior to Requests v2.4.0 the adapter did not have access to the proxy setup code. In earlier versions of Requests, this adapter will not function properly when used with proxies.

HostHeaderSSLAdapter

New in version 0.7.0.

Requests supports SSL Verification by default. However, it relies on the user making a request with the URL that has the hostname in it. If, however, the user needs to make a request with the IP address, they cannot actually verify a certificate against the hostname they want to request.

To accomodate this very rare need, we’ve added HostHeaderSSLAdapter. Example usage:

import requests
from requests_toolbelt.adapters import host_header_ssl
s = requests.Session()
s.mount('https://', host_header_ssl.HostHeaderSSLAdapter())
s.get("https://93.184.216.34", headers={"Host": "example.org"})
class requests_toolbelt.adapters.host_header_ssl.HostHeaderSSLAdapter(pool_connections=10, pool_maxsize=10, max_retries=0, pool_block=False)

A HTTPS Adapter for Python Requests that sets the hostname for certificate verification based on the Host header.

This allows requesting the IP address directly via HTTPS without getting a “hostname doesn’t match” exception.

Example usage:

>>> s.mount('https://', HostHeaderSSLAdapter())
>>> s.get("https://93.184.216.34", headers={"Host": "example.org"})

SourceAddressAdapter

New in version 0.3.0.

The SourceAddressAdapter allows a user to specify a source address for their connnection.

class requests_toolbelt.adapters.source.SourceAddressAdapter(source_address, **kwargs)

A Source Address Adapter for Python Requests that enables you to choose the local address to bind to. This allows you to send your HTTP requests from a specific interface and IP address.

Two address formats are accepted. The first is a string: this will set the local IP address to the address given in the string, and will also choose a semi-random high port for the local port number.

The second is a two-tuple of the form (ip address, port): for example, ('10.10.10.10', 8999). This will set the local IP address to the first element, and the local port to the second element. If 0 is used as the port number, a semi-random high port will be selected.

Warning

Setting an explicit local port can have negative interactions with connection-pooling in Requests: in particular, it risks the possibility of getting “Address in use” errors. The string-only argument is generally preferred to the tuple-form.

Example usage:

import requests
from requests_toolbelt.adapters.source import SourceAddressAdapter

s = requests.Session()
s.mount('http://', SourceAddressAdapter('10.10.10.10'))
s.mount('https://', SourceAddressAdapter(('10.10.10.10', 8999)))

SocketOptionsAdapter

New in version 0.4.0.

Note

This adapter will only work with requests 2.4.0 or newer. The ability to set arbitrary socket options does not exist prior to requests 2.4.0.

The SocketOptionsAdapter allows a user to pass specific options to be set on created sockets when constructing the Adapter without subclassing. The adapter takes advantage of urllib3’s support for setting arbitrary socket options for each urllib3.connection.HTTPConnection (and HTTPSConnection).

To pass socket options, you need to send a list of three-item tuples. For example, requests and urllib3 disable Nagle’s Algorithm by default. If you need to re-enable it, you would do the following:

import socket
import requests
from requests_toolbelt.adapters.socket_options import SocketOptionsAdapter

nagles = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 0)]
session = requests.Session()
for scheme in session.adapters.keys():
    session.mount(scheme, SocketOptionsAdapter(socket_options=nagles))

This would re-enable Nagle’s Algorithm for all http:// and https:// connections made with that session.

class requests_toolbelt.adapters.socket_options.SocketOptionsAdapter(**kwargs)

An adapter for requests that allows users to specify socket options.

Since version 2.4.0 of requests, it is possible to specify a custom list of socket options that need to be set before establishing the connection.

Example usage:

>>> import socket
>>> import requests
>>> from requests_toolbelt.adapters import socket_options
>>> s = requests.Session()
>>> opts = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 0)]
>>> adapter = socket_options.SocketOptionsAdapter(socket_options=opts)
>>> s.mount('http://', adapter)

You can also take advantage of the list of default options on this class to keep using the original options in addition to your custom options. In that case, opts might look like:

>>> opts = socket_options.SocketOptionsAdapter.default_options + opts

TCPKeepAliveAdapter

New in version 0.4.0.

Note

This adapter will only work with requests 2.4.0 or newer. The ability to set arbitrary socket options does not exist prior to requests 2.4.0.

The TCPKeepAliveAdapter allows a user to pass specific keep-alive related options as keyword parameters as well as arbitrary socket options.

Note

Different keep-alive related socket options may not be available for your platform. Check the socket module for the availability of the following constants:

  • socket.TCP_KEEPIDLE

  • socket.TCP_KEEPCNT

  • socket.TCP_KEEPINTVL

The adapter will silently ignore any option passed for a non-existent option.

An example usage of the adapter:

import requests
from requests_toolbelt.adapters.socket_options import TCPKeepAliveAdapter

session = requests.Session()
keep_alive = TCPKeepAliveAdapter(idle=120, count=20, interval=30)
session.mount('https://region-a.geo-1.compute.hpcloudsvc.com', keep_alive)
session.post('https://region-a.geo-1.compute.hpcloudsvc.com/v2/1234abcdef/servers',
             # ...
             )

In this case we know that creating a server on HP Public Cloud can cause requests to hang without using TCP Keep-Alive. So we mount the adapter specifically for that domain, instead of adding it to every https:// and http:// request.

class requests_toolbelt.adapters.socket_options.TCPKeepAliveAdapter(**kwargs)

An adapter for requests that turns on TCP Keep-Alive by default.

The adapter sets 4 socket options:

  • SOL_SOCKET SO_KEEPALIVE - This turns on TCP Keep-Alive

  • IPPROTO_TCP TCP_KEEPINTVL 20 - Sets the keep alive interval

  • IPPROTO_TCP TCP_KEEPCNT 5 - Sets the number of keep alive probes

  • IPPROTO_TCP TCP_KEEPIDLE 60 - Sets the keep alive time if the socket library has the TCP_KEEPIDLE constant

The latter three can be overridden by keyword arguments (respectively):

  • idle

  • interval

  • count

You can use this adapter like so:

>>> from requests_toolbelt.adapters import socket_options
>>> tcp = socket_options.TCPKeepAliveAdapter(idle=120, interval=10)
>>> s = requests.Session()
>>> s.mount('http://', tcp)

X509Adapter

Requests supports SSL Verification using a certificate in .pem format by default. In some cases it is necessary to pass a full cert chain as part of a request or it is deemed too great a risk to decrypt the certificate into a .pem file.

For such use cases we have created X509Adapter. Example usage:

import requests
from requests_toolbelt.adapters.x509 import X509Adapter
s = requests.Session()
a = X509Adapter(max_retries=3,
                cert_bytes=b'...', pk_bytes=b'...', encoding='...')
s.mount('https://', a)
class requests_toolbelt.adapters.x509.X509Adapter(*args, **kwargs)

Adapter for use with X.509 certificates.

Provides an interface for Requests sessions to contact HTTPS urls and authenticate with an X.509 cert by implementing the Transport Adapter interface. This class will need to be manually instantiated and mounted to the session

Parameters

  • pool_connections – The number of urllib3 connection pools to cache.

  • pool_maxsize – The maximum number of connections to save in the pool.

  • max_retries – The maximum number of retries each connection should attempt. Note, this applies only to failed DNS lookups, socket connections and connection timeouts, never to requests where data has made it to the server. By default, Requests does not retry failed connections. If you need granular control over the conditions under which we retry a request, import urllib3’s Retry class and pass that instead.

  • pool_block – Whether the connection pool should block for connections.

  • cert_bytes (bytes) – bytes object containing contents of a cryptography.x509Certificate object using the encoding specified by the encoding parameter.

  • pk_bytes (bytes) – bytes object containing contents of a object that implements cryptography.hazmat.primitives.serialization.PrivateFormat using the encoding specified by the encoding parameter.

  • password – string or utf8 encoded bytes containing the passphrase used for the private key. None if unencrypted. Defaults to None.

  • encoding – Enumeration detailing the encoding method used on the cert_bytes parameter. Can be either PEM or DER. Defaults to PEM.

Usage:

>>> import requests
>>> from requests_toolbelt.adapters.x509 import X509Adapter
>>> s = requests.Session()
>>> a = X509Adapter(max_retries=3,
          cert_bytes=b'...', pk_bytes=b'...', encoding='...'
>>> s.mount('https://', a)