Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix 'I/O operation on closed file' and 'Form data has been processed already' upon redirect on multipart data #9201

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

GLGDLY
Copy link

@GLGDLY GLGDLY commented Sep 19, 2024

What do these changes do?

Fix 'I/O operation on closed file' and 'Form data has been processed already' upon redirect on multipart data, based on the discussion in #6853 .

This approach tries to pre-build Payload object for the data passing into the request, before the redirect while True loop starts, so we can reuse the same Payload object for the entire redirect chain. However, it is notable that I/O object is always an issue, so my thought here is to use the seek operation on the I/O object to allow chunk writing on redirect requests.

Yet, for non-seekable I/O objects, some possible solution are:

  • sending expect 100 first
  • local buffering
  • raise error

I think more discussion might be needed for non-seekable objects, so I just raise error in this PR first.

Another thing I think worth discussion is that I removed the close() operation on the I/O object in this PR due to the following reasons:

  • StringIOPayload do not close its I/O value in the original code,
  • Closing of the I/O value will make the payload un-reusable, even if it is seekable
  • We are not the one to create the I/O value, so maybe we should leave for user to close it.
  • standard I/O object will be closed upon destructure, so it won't affect the user interface

But I do think more discussions might be needed here.

Are there changes in behavior for the user?

No.

Is it a substantial burden for the maintainers to support this?

Yes.

Related issue number

Fixes #5577
Fixes #5530

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt
    • The format is <Name> <Surname>.
    • Please keep alphabetical order, the file is sorted by names.
  • Add a new news fragment into the CHANGES/ folder
    • name it <issue_or_pr_num>.<type>.rst (e.g. 588.bugfix.rst)

    • if you don't have an issue number, change it to the pull request
      number after creating the PR

      • .bugfix: A bug fix for something the maintainers deemed an
        improper undesired behavior that got corrected to match
        pre-agreed expectations.
      • .feature: A new behavior, public APIs. That sort of stuff.
      • .deprecation: A declaration of future API removals and breaking
        changes in behavior.
      • .breaking: When something public is removed in a breaking way.
        Could be deprecated in an earlier release.
      • .doc: Notable updates to the documentation structure or build
        process.
      • .packaging: Notes for downstreams about unobvious side effects
        and tooling. Changes in the test invocation considerations and
        runtime assumptions.
      • .contrib: Stuff that affects the contributor experience. e.g.
        Running tests, building the docs, setting up the development
        environment.
      • .misc: Changes that are hard to assign to any of the above
        categories.
    • Make sure to use full sentences with correct case and punctuation,
      for example:

      Fixed issue with non-ascii contents in doctest text files
      -- by :user:`contributor-gh-handle`.

      Use the past tense or the present tense a non-imperative mood,
      referring to what's changed compared to the last released version
      of this project.

Copy link

codecov bot commented Sep 19, 2024

Codecov Report

Attention: Patch coverage is 93.85246% with 15 lines in your changes missing coverage. Please review.

Project coverage is 98.28%. Comparing base (88f3834) to head (fb2d3b8).

Files with missing lines Patch % Lines
aiohttp/payload.py 72.72% 5 Missing and 7 partials ⚠️
aiohttp/formdata.py 33.33% 1 Missing and 1 partial ⚠️
tests/test_formdata.py 98.83% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9201      +/-   ##
==========================================
- Coverage   98.31%   98.28%   -0.04%     
==========================================
  Files         107      107              
  Lines       34508    34709     +201     
  Branches     4100     4131      +31     
==========================================
+ Hits        33928    34114     +186     
- Misses        410      417       +7     
- Partials      170      178       +8     
Flag Coverage Δ
CI-GHA 98.18% <93.85%> (-0.04%) ⬇️
OS-Linux 97.84% <93.85%> (-0.04%) ⬇️
OS-Windows 96.27% <93.82%> (-0.03%) ⬇️
OS-macOS 97.52% <93.85%> (-0.04%) ⬇️
Py-3.10.11 97.62% <93.85%> (-0.03%) ⬇️
Py-3.10.15 97.55% <93.85%> (-0.03%) ⬇️
Py-3.11.10 97.44% <93.85%> (-0.03%) ⬇️
Py-3.11.9 97.51% <93.85%> (-0.03%) ⬇️
Py-3.12.6 97.90% <93.85%> (-0.04%) ⬇️
Py-3.9.13 97.51% <93.85%> (-0.03%) ⬇️
Py-3.9.20 97.45% <93.85%> (-0.03%) ⬇️
Py-pypy7.3.16 97.07% <93.85%> (-0.03%) ⬇️
VM-macos 97.52% <93.85%> (-0.04%) ⬇️
VM-ubuntu 97.84% <93.85%> (-0.04%) ⬇️
VM-windows 96.27% <93.82%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

try:
# It is weird but some IO object dont have `seekable()` method as IOBase object,
# it seems better for us to direct try if the `seek()` and `tell()` is available
# e.g. tarfile.TarFile._Stream
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class doesn't appear to be an IOBase, so I'm wondering why it reaches this code. Does it match one of the payload subclasses?

Comment on lines +119 to +120
if not self._fields:
return self._writer
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't appear to save much, one call to .clear().

Suggested change
if not self._fields:
return self._writer

@Dreamsorcerer
Copy link
Member

At first glance, it looks like some reasonable changes for an awkward situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants