Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IQSS/6763-multi-part upload API calls #6995

Merged
merged 59 commits into from
Aug 31, 2020

Commits on Jun 26, 2020

  1. Configuration menu
    Copy the full SHA
    a02d41d View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2020

  1. add maxpartsize jvm option

    qqmyers committed Jul 7, 2020
    Configuration menu
    Copy the full SHA
    e6fd3c8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c9a03e5 View commit details
    Browse the repository at this point in the history
  3. remove @singleton

    qqmyers committed Jul 7, 2020
    Configuration menu
    Copy the full SHA
    92f1cc2 View commit details
    Browse the repository at this point in the history
  4. Don't update lastapiusetime

    qqmyers committed Jul 7, 2020
    Configuration menu
    Copy the full SHA
    26be2c4 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2020

  1. Configuration menu
    Copy the full SHA
    78540d0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2bc7e0d View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2020

  1. Configuration menu
    Copy the full SHA
    6810c4d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0f76d80 View commit details
    Browse the repository at this point in the history
  3. remove blocking tests

    qqmyers committed Jul 9, 2020
    Configuration menu
    Copy the full SHA
    9974a38 View commit details
    Browse the repository at this point in the history
  4. restore singleton

    qqmyers committed Jul 9, 2020
    Configuration menu
    Copy the full SHA
    4c1a21b View commit details
    Browse the repository at this point in the history
  5. change to min-part-size

    it's the max that a single part direct upload can be (as coded), and the
    minimum part size for multipart uploads.
    These could be separate settings, but it's not clear that that's useful
    to support. E.g. for AWS S3, the min-part-size is 5MB but single uploads
    could go to 5 GB so that is the max single upload.
    qqmyers committed Jul 9, 2020
    Configuration menu
    Copy the full SHA
    131e531 View commit details
    Browse the repository at this point in the history
  6. try to cleanup by calling abort on failure of complete upload

    at one point I was sending parts that were below the min-part-size on
    AWS and everything worked until a 400 response/Entity too small response
    when trying to complete. This commit would try to clean up after any
    similar problems (as the code now stops this aprticular one from
    happening).
    qqmyers committed Jul 9, 2020
    Configuration menu
    Copy the full SHA
    997f93f View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2020

  1. add release note

    djbrooke committed Jul 10, 2020
    Configuration menu
    Copy the full SHA
    bd05496 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b53ef80 View commit details
    Browse the repository at this point in the history
  3. initial convert to use class

    qqmyers committed Jul 10, 2020
    Configuration menu
    Copy the full SHA
    8487388 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4447ba1 View commit details
    Browse the repository at this point in the history
  5. pass fileSize correctly

    qqmyers committed Jul 10, 2020
    Configuration menu
    Copy the full SHA
    5758799 View commit details
    Browse the repository at this point in the history
  6. Merge pull request #1 from djbrooke/IQSS/6763

    add release note
    qqmyers committed Jul 10, 2020
    Configuration menu
    Copy the full SHA
    c3bb1b7 View commit details
    Browse the repository at this point in the history
  7. bug fixes

    qqmyers committed Jul 10, 2020
    Configuration menu
    Copy the full SHA
    c442b47 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    603b8d9 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2020

  1. Fix for IQSS#7060

    qqmyers committed Jul 14, 2020
    Configuration menu
    Copy the full SHA
    b7c0c02 View commit details
    Browse the repository at this point in the history
  2. start mp support

    qqmyers committed Jul 14, 2020
    Configuration menu
    Copy the full SHA
    1e4232b View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2020

  1. test session restart in login

    qqmyers committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    8f207ab View commit details
    Browse the repository at this point in the history
  2. typos

    qqmyers committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    5b7be84 View commit details
    Browse the repository at this point in the history
  3. debug logging

    qqmyers committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    4b39cca View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    caf5ec8 View commit details
    Browse the repository at this point in the history
  5. typos

    qqmyers committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    319b4b3 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2020

  1. Configuration menu
    Copy the full SHA
    5ee4f87 View commit details
    Browse the repository at this point in the history
  2. Merge branch 'IQSS/3254' into IQSS/6763

    Conflicts:
    	src/main/java/edu/harvard/iq/dataverse/LoginPage.java
    qqmyers committed Jul 22, 2020
    Configuration menu
    Copy the full SHA
    e0ebf62 View commit details
    Browse the repository at this point in the history
  3. support MP direct upload for datasets being created

    where the dataset.getId() is null
    qqmyers committed Jul 22, 2020
    Configuration menu
    Copy the full SHA
    13abba2 View commit details
    Browse the repository at this point in the history
  4. remove debug logging

    qqmyers committed Jul 22, 2020
    Configuration menu
    Copy the full SHA
    f5593e8 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    54739ed View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    ad92b0f View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2020

  1. enable direct upload/ingest of text/tsv

    when no mimetype is assigned by the browser
    qqmyers committed Jul 23, 2020
    Configuration menu
    Copy the full SHA
    cdd0db5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    14beb64 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2020

  1. mp upload fixes

    qqmyers committed Jul 24, 2020
    Configuration menu
    Copy the full SHA
    16a0f77 View commit details
    Browse the repository at this point in the history
  2. script fixes w.r.t. progress

    qqmyers committed Jul 24, 2020
    Configuration menu
    Copy the full SHA
    372d83c View commit details
    Browse the repository at this point in the history
  3. formatting

    qqmyers committed Jul 24, 2020
    Configuration menu
    Copy the full SHA
    8388ed3 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    832d196 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b18a583 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    9c696bd View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    fdbfb37 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    d2c8c8c View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2020

  1. Merge remote-tracking branch 'IQSS/develop' into IQSS/6763

    Conflicts:
    	src/main/webapp/editFilesFragment.xhtml
    qqmyers committed Jul 31, 2020
    Configuration menu
    Copy the full SHA
    6320049 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2020

  1. Configuration menu
    Copy the full SHA
    8ee0d07 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2020

  1. Merge remote-tracking branch 'IQSS/develop' into IQSS/6763

    Conflicts:
    	src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java
    qqmyers committed Aug 20, 2020
    Configuration menu
    Copy the full SHA
    63aa365 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2020

  1. Merge remote-tracking branch 'IQSS/develop' into IQSS/6763

    Conflicts:
    	src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java
    	src/main/java/edu/harvard/iq/dataverse/api/Datasets.java
    	src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java
    qqmyers committed Aug 24, 2020
    Configuration menu
    Copy the full SHA
    0b2f722 View commit details
    Browse the repository at this point in the history
  2. merge issues

    qqmyers committed Aug 24, 2020
    Configuration menu
    Copy the full SHA
    751f0b9 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2020

  1. Configuration menu
    Copy the full SHA
    04c6f44 View commit details
    Browse the repository at this point in the history
  2. add more documentation

    qqmyers committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    c37795f View commit details
    Browse the repository at this point in the history
  3. cleanup

    qqmyers committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    d81b05c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f490340 View commit details
    Browse the repository at this point in the history
  5. handle when last part upload fails

    still want to trigger the retry or report the error in this case. Prior
    code assumed the last call to finish succeeded.
    qqmyers committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    7d673a8 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2020

  1. Configuration menu
    Copy the full SHA
    a08c5d7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    35d6735 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2020

  1. Configuration menu
    Copy the full SHA
    ec7eb04 View commit details
    Browse the repository at this point in the history
  2. fix abort call, handle cancels, fix progress, limit parts to 10

    for cancel - catching the per datafile cancel button and handling the
    abort, after all currently uploading parts are done, in mp calls, to
    remove the file
    
    Note for single part uploads, cancelling the file does not yet remove
    the temp file, but in that case we do add the temp label so files can be
    cleaned up later. (Todo note includes more info)
    qqmyers committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    8b63d56 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2020

  1. Configuration menu
    Copy the full SHA
    c6d06e5 View commit details
    Browse the repository at this point in the history