-
Notifications
You must be signed in to change notification settings - Fork 30
fix: extra byte read from chunk transfer #1294
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
return chunkReader.chunk.read(sink, limit) | ||
|
||
val totalBytesTransferred = chunkReader.chunk.read(sink, limit) | ||
contentBytesTransferred = totalBytesTransferred - chunkReader.chunkMetadataBytes | ||
|
||
return totalBytesTransferred |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question: I feel pretty strongly that AwsChunkedSource.read
should return the number of bytes which were read from the delegate source rather than the number of bytes written to the destination sink. Can we just return totalBytesTransferred - chunkReader.chunkMetadataBytes
instead of storing it in a property? What do we need the total bytes transferred for (at least, as the output from read
)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We still need to return totalBytes for some of our tests:
Line 445 in 444ff8a
val totalBytesExpected = encodedUnsignedChunkLength(CHUNK_SIZE_BYTES) + encodedUnsignedChunkLength(0) + "\r\n".length |
If we not return the
totalBytesTransferred
, those tests will fail.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm pretty sure we can find a way around that, maybe by exposing the total bytes as some kind of test utility?
Affected ArtifactsChanged in size
|
Issue #
Fixes #1285
Description of changes
Adds a public property
contentBytesTransferred
to track the content bytes transferred, excluding chunk metadata (need to opt in internal api to access).This allows monitor file transfer progress without counting the overhead bytes from chunk encoding.
example usage:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.