Support for proxy that requires DigestAuth

[raffaem]

Requests seems not to have support for proxies which requires DigestAuth, only BasicAuth.

In this SO post there is an authorization module that implements support for proxies that require DigestAuth, and also allow to connect to HTTPS websites.

requests-toolbet (I don't know whether it is an official project of requests) has a HTTPProxyDigestAuth class, but doesn't allow to connect to HTTPS websites.

[nateprewitt]

Hi @raffaem,

Can you try using the HTTPProxyDigestAuth class with Requests 2.25.1? There is currently a bug with Requests 2.26.0 with proxy authorization headers. I'm not certain, but I believe this may be related to what you're seeing. We're intending to release Requests 2.27.0 tomorrow which will address this issue.

[raffaem]

Hi @raffaem,

Can you try using the HTTPProxyDigestAuth class with Requests 2.25.1? There is currently a bug with Requests 2.26.0 with proxy authorization headers. I'm not certain, but I believe this may be related to what you're seeing. We're intending to release Requests 2.27.0 tomorrow which will address this issue.

Nada, it fails even with requests 2.25.1

Here is the code:

#!/usr/bin/env python3

import requests
from requests_toolbelt.auth.http_proxy_digest import HTTPProxyDigestAuth

print("Requests version="+str(requests.__version__))

proxies = {
    "http": "http://proxy.polimi.it:8080",
    "https": "http://proxy.polimi.it:8080",
}

auth = HTTPProxyDigestAuth("USERNAME", "PASSWORD")

url = "http://verify.proxy.polimi.it"
s = requests.get(url, proxies=proxies, auth=auth)
print(s.status_code)

url = "https://dev.elsevier.com"
s = requests.get(url, proxies=proxies, auth=auth)
print(s.status_code)

Here is the output:

$ ./toolbelt_test.py 
Requests version=2.25.1
200
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 366, in connect
    self._tunnel()
  File "/usr/lib64/python3.10/http/client.py", line 924, in _tunnel
    raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 407 Proxy Authentication Required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/raffaele/.local/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dev.elsevier.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/run/media/raffaele/55ab61c4-83cf-4d9f-a5cd-7fcfdc14b4fb/data/progetti_miei/pybliometrics_test/./toolbelt_test.py", line 20, in <module>
    s = requests.get(url, proxies=proxies, auth=auth)
  File "/home/raffaele/.local/lib/python3.10/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/raffaele/.local/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/raffaele/.local/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/raffaele/.local/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/raffaele/.local/lib/python3.10/site-packages/requests/adapters.py", line 510, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='dev.elsevier.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))

Works fine with the implementation of Tey' from StackOverflow

[nateprewitt]

Thanks for checking @raffaem. Looking at the StackOverflow post, there's some pretty extensive monkey-patching for both Requests, urllib3 and httplib to make this possible. This isn't something we're likely to ever add directly to Requests and I'm doubtful it would even make it into the Requests toolbelt. Unfortunately, I don't see a clean way for us to do this with what's currently exposed from urllib3/httplib.

[raffaem]

Uhm can we work with urllib3/httplib for them to expose what we need?

HTTPProxyDigestAuth is less useful if you can't connect to HTTPS websites, as most websites today are HTTPS

[nateprewitt]

You can read all of the patching changes made here, with several of them overriding behavior from the standard library. That would require changes to CPython itself, meaning the earliest this behavior could be available is in 3.11 or 3.12. We'd need someone to drive those changes to expose APIs for urllib3 to do this, or potentially rework how urllib3 interacts with tunnel calls. I'm not sure if the latter is even possible at first glance though.

That's all going to be an extremely time intensive change though that likely won't be widely available for at least a couple years. The short term solution may be the proposed patch in the StackOverflow question. It's not something we can support or include directly in Requests though.

[redbaron4]

http.client in Python3.12 has made some modifications to the HTTPConnection._tunnel() as seen here. It still raises OSError but preserves header information which can be accessed via self.get_proxy_response_headers().

Do these changes make it a little bit easier to extract the challenge and re-use it to send the correct header using requests or requests-toolbelt?

I understand it would require urllib3 to be updated but the change would be much smaller now that headers are accessible to urllib3 when calling _tunnel() and failing.

[raffaem]

Related request to httpx: encode/httpx#2033