Skip to content

Close proxy tunnel connection when TLS raises during CONNECT (#921) - #1087

Open
phanen wants to merge 1 commit into
encode:masterfrom
phanen:fix/proxy-tls-connection-leak
Open

phanen wants to merge 1 commit into
encode:masterfrom
phanen:fix/proxy-tls-connection-leak

Conversation

@phanen

@phanen phanen commented Jun 11, 2026

Copy link
Copy Markdown

Summary

Close proxy tunnel connection when TLS raises during CONNECT (#921)

This happened when I use python-telegram-bot library.

Details
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File ".venv/lib/python3.13/site-packages/telegram/request/_httpxrequest.py", line 279, in do_request
    res = await self._client.request(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File ".venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv/lib/python3.13/site-packages/telegram/ext/_utils/networkloop.py", line 161, in network_retry_loop
    await do_action()
  File ".venv/lib/python3.13/site-packages/telegram/ext/_utils/networkloop.py", line 154, in do_action
    action_cb_task.result()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File ".venv/lib/python3.13/site-packages/telegram/ext/_updater.py", line 340, in polling_action_cb
    updates = await self.bot.get_updates(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/telegram/ext/_extbot.py", line 672, in get_updates
    updates = await super().get_updates(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<9 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/telegram/_bot.py", line 4865, in get_updates
    await self._post(
    ^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    ),
    ^
  File ".venv/lib/python3.13/site-packages/telegram/_bot.py", line 704, in _post
    return await self._do_post(
           ^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/telegram/ext/_extbot.py", line 370, in _do_post
    return await super()._do_post(
           ^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/telegram/_bot.py", line 733, in _do_post
    result = await request.post(
             ^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/telegram/request/_baserequest.py", line 198, in post
    result = await self._request_wrapper(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/telegram/request/_baserequest.py", line 305, in _request_wrapper
    code, payload = await self.do_request(
                    ^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File ".venv/lib/python3.13/site-packages/telegram/request/_httpxrequest.py", line 303, in do_request
    raise NetworkError(f"httpx.{err.__class__.__name__}: {err}") from err
telegram.error.NetworkError: httpx.ConnectError:
Exception happened while polling for updates.
Traceback (most recent call last):
  File ".venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File ".venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File ".venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        pool_request.request
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File ".venv/lib/python3.13/site-packages/httpcore/_async/http_proxy.py", line 316, in handle_async_request
    stream = await stream.start_tls(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 376, in start_tls
    return await self._stream.start_tls(ssl_context, server_hostname, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/httpcore/_backends/anyio.py", line 67, in start_tls
    with map_exceptions(exc_map):
         ~~~~~~~~~~~~~~^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File ".venv/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • (N/A) I've updated the documentation accordingly.

@phanen
phanen force-pushed the fix/proxy-tls-connection-leak branch from 76a9abb to 6db6963 Compare June 18, 2026 08:48
@ylxmf2005

Copy link
Copy Markdown

Thanks for this. One gap worth covering: except Exception does not catch cancellation, so the tunnel still leaks when the task is cancelled during start_tls. asyncio.CancelledError is a BaseException, and this is how a caller's own timeout usually arrives.

Repro against this branch:

import asyncio, httpcore

class Stall(httpcore.AsyncMockStream):
    async def start_tls(self, ssl_context, server_hostname=None, timeout=None):
        await asyncio.sleep(3600)

class Backend(httpcore.AsyncMockBackend):
    async def connect_tcp(self, host, port, timeout=None, local_address=None, socket_options=None):
        return Stall(list(self._buffer))

async def main():
    async with httpcore.AsyncConnectionPool(
        proxy=httpcore.Proxy("http://localhost:8080/"),
        max_connections=1,
        network_backend=Backend([b"HTTP/1.1 200 OK\r\n\r\n"]),
    ) as pool:
        task = asyncio.create_task(pool.request("GET", "https://example.com/"))
        await asyncio.sleep(0.05)
        task.cancel()
        try:
            await task
        except asyncio.CancelledError:
            pass
        print(pool.connections)

asyncio.run(main())
Code pool.connections after cancelling
master [<AsyncTunnelHTTPConnection [..., HTTP/1.1, ACTIVE, Request Count: 1]>]
this PR [<AsyncTunnelHTTPConnection [..., HTTP/1.1, ACTIVE, Request Count: 1]>]
except BaseException with a shield []

The third row uses the pattern http11.py and connection_pool.py already follow:

except BaseException as exc:
    with AsyncShieldCancellation():
        await self._connection.aclose()
    raise exc

For context on the impact: we hit the timeout path in production with httpcore 1.0.9 behind a CONNECT proxy, on a client with max_connections=4. A short network blip failed four TLS handshakes at once. All four slots were lost, and every later request ended in PoolTimeout for hours, with the proxy and upstream healthy, until the process restarted. We now close the connection on BaseException in a local patch. With it, the client recovers as soon as the proxy does, for both timeouts and cancellation.

@CN-Zephyr

Copy link
Copy Markdown

I independently reproduced the proxy-tunnel leak on httpcore 1.0.9 with httpx 0.25.2. The cancellation gap described above also reproduces with a real loopback HTTP CONNECT proxy, using uvloop and a one-slot connection pool.

The proxy accepts CONNECT with a 200, then waits after receiving the TLS ClientHello. This lets the test trigger a handshake timeout, an explicit Task.cancel(), or cancellation from an AnyIO cancel scope while the TLS handshake is in progress. With #1087 backported as written, the handshake timeout releases the connection, but cancelling the task leaves the slot occupied and the next request raises PoolTimeout.

I tested the proposed async change that catches BaseException around start_tls, closes the tunnel inside AsyncShieldCancellation, and re-raises the original exception. All three cases left the pool empty. After the proxy resumed normal TLS, the same client and same pool completed an HTTPS GET with HTTP 200. A regression that asserts successful reuse of the pool after cancellation would cover the case.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants