Skip to content

gh-57945: Document exceptions not wrapped by urlopen() - #160830

Merged
orsenthil merged 1 commit into
python:mainfrom
orsenthil:issues/57945
Oct 28, 2026
Merged

orsenthil merged 1 commit into
python:mainfrom
orsenthil:issues/57945

Conversation

@orsenthil

@orsenthil orsenthil commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

The urlopen() documentation says only that it raises URLError on protocol errors. In practice, exceptions raised while reading or parsing the HTTP response come from http.client without being wrapped. These include:

  • http.client.HTTPException subclasses such as BadStatusLine. These are not OSError subclasses, so even except OSError misses them.
  • OSError subclasses such as TimeoutError (the server doesn't respond within timeout) and RemoteDisconnected.

ValueError is also raised for a URL with no scheme.

This happens because AbstractHTTPHandler.do_open() wraps OSError in URLError only around h.request(), not around h.getresponse().

Earlier discussion on the issue (Antoine Pitrou, Martin Panter) and the related issues gh-53069, gh-66986 and gh-69819 settled on documenting this rather than adding more wrapping, which would break existing code. This PR changes only the documentation.

🤖 Generated with Claude Code


Helpful reproducer for this.

import socket, threading, time, urllib.request, urllib.error

def serve(reply, delay=0):
    s = socket.socket(); s.bind(("127.0.0.1", 0)); s.listen(1)
    def run():
        c, _ = s.accept(); c.recv(65536)
        time.sleep(delay)
        if reply is not None: c.sendall(reply)
        time.sleep(0.5); c.close(); s.close()
    threading.Thread(target=run, daemon=True).start()
    return f"http://127.0.0.1:{s.getsockname()[1]}/"

cases = {
  "read timeout during getresponse": (None, 2, 0.5),
  "bad status line":                 (b"GARBAGE\r\n\r\n", 0, 5),
  "invalid status code":             (b"HTTP/1.1 1000 OK\r\n\r\n", 0, 5),
  "empty reply (RemoteDisconnected)":(b"", 0, 5),
  "too many headers":                (b"HTTP/1.1 200 OK\r\n" + b"X: y\r\n"*200 + b"\r\n", 0, 5),
}
for name, (reply, delay, tmo) in cases.items():
    url = serve(reply, delay)
    try:
        urllib.request.urlopen(url, timeout=tmo).read()
        print(f"{name:36} -> no error")
    except Exception as e:
        print(f"{name:36} -> {type(e).__module__}.{type(e).__qualname__}"
              f"  isURLError={isinstance(e, urllib.error.URLError)}  isOSError={isinstance(e, OSError)}")
# connection refused (connect phase) and bad scheme/value
for name, url in [("connection refused", "http://127.0.0.1:1/"), ("unknown scheme", "foo://x"), ("no scheme", "example.com")]:
    try: urllib.request.urlopen(url, timeout=2)
    except Exception as e:
        print(f"{name:36} -> {type(e).__module__}.{type(e).__qualname__}  isURLError={isinstance(e, urllib.error.URLError)}")
 ./python -I repro.py 
read timeout during getresponse      -> builtins.TimeoutError  isURLError=False  isOSError=True
bad status line                      -> http.client.BadStatusLine  isURLError=False  isOSError=False
invalid status code                  -> http.client.BadStatusLine  isURLError=False  isOSError=False
empty reply (RemoteDisconnected)     -> http.client.RemoteDisconnected  isURLError=False  isOSError=True
too many headers                     -> http.client.HTTPException  isURLError=False  isOSError=False
connection refused                   -> urllib.error.URLError  isURLError=True
unknown scheme                       -> urllib.error.URLError  isURLError=True
no scheme                            -> builtins.ValueError  isURLError=False

Exceptions raised while reading or parsing the HTTP response, such as
http.client.HTTPException subclasses and OSError subclasses like
TimeoutError, propagate from urlopen() without being wrapped in
URLError.  Document this, and that ValueError is raised for an invalid
URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #35474467 | 📁 Comparing abb69d4 against main (d7ea5ba)

  🔍 Preview build  

1 file changed
± library/urllib.request.html

@orsenthil
orsenthil enabled auto-merge (squash) October 28, 2026 15:52
@orsenthil
orsenthil merged commit 87f8c0d into python:main Oct 28, 2026
36 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Docs PRs Oct 28, 2026
@orsenthil orsenthil added needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes labels Oct 28, 2026
@miss-islington-app

Copy link
Copy Markdown

Thanks @orsenthil for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

@miss-islington-app

Copy link
Copy Markdown

Thanks @orsenthil for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

@bedevere-app

bedevere-app Bot commented Oct 28, 2026

Copy link
Copy Markdown

GH-160831 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Oct 28, 2026
@bedevere-app

bedevere-app Bot commented Oct 28, 2026

Copy link
Copy Markdown

GH-160832 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Oct 28, 2026
orsenthil added a commit that referenced this pull request Oct 28, 2026
) (#160831)

gh-57945: Document exceptions not wrapped by urlopen() (GH-160830)

Exceptions raised while reading or parsing the HTTP response, such as
http.client.HTTPException subclasses and OSError subclasses like
TimeoutError, propagate from urlopen() without being wrapped in
URLError.  Document this, and that ValueError is raised for an invalid
URL.
(cherry picked from commit 87f8c0d)

Co-authored-by: Senthil Kumaran <senthil@python.org>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
orsenthil added a commit that referenced this pull request Oct 28, 2026
) (#160832)

gh-57945: Document exceptions not wrapped by urlopen() (GH-160830)

Exceptions raised while reading or parsing the HTTP response, such as
http.client.HTTPException subclasses and OSError subclasses like
TimeoutError, propagate from urlopen() without being wrapped in
URLError.  Document this, and that ValueError is raised for an invalid
URL.
(cherry picked from commit 87f8c0d)

Co-authored-by: Senthil Kumaran <senthil@python.org>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation in the Doc dir skip news

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant