Skip to content

Fix socket/fd leak in streaming endpoints (#2766)#3412

Open
ykstorm wants to merge 2 commits into
docker:mainfrom
ykstorm:fix-streaming-socket-leak-2766
Open

Fix socket/fd leak in streaming endpoints (#2766)#3412
ykstorm wants to merge 2 commits into
docker:mainfrom
ykstorm:fix-streaming-socket-leak-2766

Conversation

@ykstorm

@ykstorm ykstorm commented Jun 23, 2026

Copy link
Copy Markdown

Fixes #2766.

The bug

container.logs(stream=True), events(), attach(), stats() and the raw
exec streams all hand back a generator that reads from a long-lived socket. If
you stop reading before the stream ends — break out of the loop, hit an
exception, or just drop the generator — the underlying response was never
closed, so the socket and its fd leaked. _get_raw_response_socket() even pins
the response onto the socket (sock._response = response), so the garbage
collector can't always clean it up either. _read_from_socket(stream=True) was
explicit about it in a comment: "the caller is responsible for closing the
response."

In a long-running process that tails logs and bails out early, the fds pile up
until you hit the open-file limit — the ResourceWarning: unclosed <socket.socket> reports on that issue.

The fix

Wrap the yield loop of each streaming generator in try/finally: response.close(). When you stop iterating, Python raises GeneratorExit into
the suspended generator (on .close() and on GC), so the finally is the right
place to release the socket. Socket setup stays lazy — it still happens on the
first read, exactly as before — so nothing about the call semantics changes.

for line in container.logs(stream=True):
    if i_have_what_i_need(line):
        break          # socket/fd is closed for you now

Touched: _stream_raw_result, _multiplexed_response_stream_helper,
_read_from_socket in docker/api/client.py.

Showing it actually leaks (and stops leaking)

benchmarks/stream_leak.py runs without a daemon — it serves an endless chunked
response locally, opens N streams, reads one chunk from each and stops early,
then counts how many sockets are still open (client-side, cross-checked with
psutil). Same generator with and without the try/finally:

opening 200 streams, reading one chunk, then stopping each early

impl       streams  sockets leaked   ESTABLISHED conns
------------------------------------------------------
old            200             200                 200
fixed          200               0                   0

Tests / docs

  • New transport-agnostic unit tests in tests/unit/stream_leak_test.py (no
    requests internals): early break, exception, and explicit .close() all
    close the response.
  • ruff is clean and the unit suite passes locally. I don't have a daemon on
    this machine, so the integration suite is down to CI.
  • New Streaming endpoints docs page + a changelog entry.

No new dependencies. Default behaviour is unchanged apart from streams now being
closed when you stop reading them.

@ykstorm ykstorm changed the title Fix socket/fd leak in streaming endpoints (#2766) + opt-in stream collector Fix socket/fd leak in streaming endpoints (#2766) Jun 24, 2026
@ykstorm

ykstorm commented Jun 24, 2026

Copy link
Copy Markdown
Author

Pushed an update: added benchmarks/stream_leak.py, a daemon-free reproduction of the leak (opens 200 streams, reads one chunk, stops early — old code leaks all 200 sockets, the fix closes every one), and referenced it from the streaming docs. Also reworded the PR description. No code changes to the fix itself.

The streaming generators returned by APIClient (logs, events, attach,
stats and raw exec streams) read from a raw socket but never closed the
underlying response if the consumer broke out of the loop early, raised,
or dropped the generator. Because _get_raw_response_socket keeps a strong
reference to the response on the socket, the socket/fd was leaked,
producing 'ResourceWarning: unclosed <socket.socket>' and the fd
exhaustion reported in docker#2766.

Wrap the yield loops of _multiplexed_response_stream_helper and
_stream_raw_result, and the generator returned by _read_from_socket
(stream=True), in try/finally so the response is closed on early break,
exception, .close(), or garbage collection. Python raises GeneratorExit
into a suspended generator on close/GC, so finally is the correct hook.

Add transport-agnostic regression tests.

Closes docker#2766

Signed-off-by: ykstorm <balveer767@gmail.com>
Document that streaming iterators now close their socket/fd on early
break, exception, .close(), or GC. Add benchmarks/stream_leak.py, a
daemon-free reproduction of docker#2766: opening many streams and stopping each
after one chunk leaks every socket on the pre-fix generators and none on
the current code.

Signed-off-by: ykstorm <balveer767@gmail.com>
@ykstorm ykstorm force-pushed the fix-streaming-socket-leak-2766 branch from 7533234 to 3e5cfca Compare June 24, 2026 01:22
@ykstorm

ykstorm commented Jun 24, 2026

Copy link
Copy Markdown
Author

Scoped this PR down to just the leak fix. I removed the optional stream-collector that was bundled in earlier — it added a background thread and a new public API that turn a surgical fix into a design discussion, and the try/finally change already closes the leak for normal consumers. The PR is now: the fix in docker/api/client.py, unit tests, a daemon-free benchmark, and docs. Happy to discuss the collector separately if there's interest, but it doesn't belong in this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Leaking file descriptors

2 participants