Skip to content

Commit

Permalink
[3.9] bpo-42853: Fix http.client fails to download >2GiB data over TLS (
Browse files Browse the repository at this point in the history
GH-27405)

Revert "bpo-36050: optimize HTTPResponse.read() (GH-12698)"

This reverts commit d6bf6f2.
  • Loading branch information
methane authored Jul 28, 2021
1 parent a13edfb commit 153365d
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 10 deletions.
42 changes: 32 additions & 10 deletions Lib/http/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@
# Mapping status codes to official W3C names
responses = {v: v.phrase for v in http.HTTPStatus.__members__.values()}

# maximal amount of data to read at one time in _safe_read
MAXAMOUNT = 1048576

# maximal line length when calling readline().
_MAXLINE = 65536
_MAXHEADERS = 100
Expand Down Expand Up @@ -604,24 +607,43 @@ def _readinto_chunked(self, b):
raise IncompleteRead(bytes(b[0:total_bytes]))

def _safe_read(self, amt):
"""Read the number of bytes requested.
"""Read the number of bytes requested, compensating for partial reads.
Normally, we have a blocking socket, but a read() can be interrupted
by a signal (resulting in a partial read).
Note that we cannot distinguish between EOF and an interrupt when zero
bytes have been read. IncompleteRead() will be raised in this
situation.
This function should be used when <amt> bytes "should" be present for
reading. If the bytes are truly not available (due to EOF), then the
IncompleteRead exception can be used to detect the problem.
"""
data = self.fp.read(amt)
if len(data) < amt:
raise IncompleteRead(data, amt-len(data))
return data
s = []
while amt > 0:
chunk = self.fp.read(min(amt, MAXAMOUNT))
if not chunk:
raise IncompleteRead(b''.join(s), amt)
s.append(chunk)
amt -= len(chunk)
return b"".join(s)

def _safe_readinto(self, b):
"""Same as _safe_read, but for reading into a buffer."""
amt = len(b)
n = self.fp.readinto(b)
if n < amt:
raise IncompleteRead(bytes(b[:n]), amt-n)
return n
total_bytes = 0
mvb = memoryview(b)
while total_bytes < len(b):
if MAXAMOUNT < len(mvb):
temp_mvb = mvb[0:MAXAMOUNT]
n = self.fp.readinto(temp_mvb)
else:
n = self.fp.readinto(mvb)
if not n:
raise IncompleteRead(bytes(mvb[0:total_bytes]), len(b))
mvb = mvb[n:]
total_bytes += n
return total_bytes

def read1(self, n=-1):
"""Read with at most one underlying system call. If at least one
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix ``http.client.HTTPSConnection`` fails to download >2GiB data.

3 comments on commit 153365d

@Rob801130
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lib/http/client.py

@kypriakos
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@methane can you help us here a bit if possible? This reverts the chance that broke the downloads for > 2GB over http correct? Which sub-version of 3.9 is this on? And this means that 3.9 with OpenSSL versions < 1.1.1 would still work with this revert?

@methane
Copy link
Member Author

@methane methane commented on 153365d Nov 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reverts the chance that broke the downloads for > 2GB over http correct?

correct.

Which sub-version of 3.9 is this on?

You can see changelog. https://docs.python.org/3.9/whatsnew/changelog.html#changelog

Put "42853" into "Filter entries by content:" form. You can find which micro version shipped this fix.

And this means that 3.9 with OpenSSL versions < 1.1.1 would still work with this revert?

I am not OpenSSL expert but I think yes.

Please sign in to comment.