Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn users when pinned dataset is larger than local GCThreshold #8413

Closed
3 tasks done
kallisti5 opened this issue Sep 7, 2021 · 1 comment
Closed
3 tasks done

Warn users when pinned dataset is larger than local GCThreshold #8413

kallisti5 opened this issue Sep 7, 2021 · 1 comment
Labels
kind/enhancement A net-new feature or improvement to an existing feature need/triage Needs initial labeling and prioritization

Comments

@kallisti5
Copy link

kallisti5 commented Sep 7, 2021

Checklist

  • My issue is specific & actionable.
  • I am not suggesting a protocol enhancement.
  • I have searched on the issue tracker for my issue.

Description

In #8103 we saw users attempting to pin "large" data sets (in this case, 90GiB) reach a hangup during the pinning process.

If the size of the dataset you're attempting to pin is larger then GCThreshold, the chunks will be immediately garbage collected when ctl+c'ing or terminating the pin. This defeats the purpose of IPFS as users should be able to "resume" downloading chunks from where they left off if pinning is interrupted.

Instead of letting users pin potentially massive amounts of data which will unknowingly be destroyed, maybe a warning when this condition exists in ipfs pin would be beneficial?

ipfs pin /ipns/blah
warning: pinned dataset is larger than your GCThreshold, data will need to be re-downloaded if interrupted!

It would be better to guide users to increase their GCThreshold instead of silently letting them believe that IPFS can't resume pinning.

#8412 is another alternative strategy to fix this (Chunk garbage collection minimum time)
#3121 seems to be another alternative strategy for this (best effort pins)

@kallisti5 kallisti5 added the kind/enhancement A net-new feature or improvement to an existing feature label Sep 7, 2021
@BigLep BigLep added the need/triage Needs initial labeling and prioritization label Sep 9, 2021
@lidel
Copy link
Member

lidel commented Dec 3, 2021

I don't believe this is feasible: it is not possible to tell the size of various non-UnixFS DAGs without fetching them. DAG-CBOR won't have size in the root node, so this warning would only work for UnixFS data, and the behavior of ipfs pin should be codec-agnostic as much as possible.

iiuc there is a way of doing best-effort pins with MFS – see #3121 (comment)
Closing this and suggest continuing in #3121 / #8412 where we discuss approaches that are codec-agnostic.

@lidel lidel closed this as completed Dec 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement A net-new feature or improvement to an existing feature need/triage Needs initial labeling and prioritization
Projects
None yet
Development

No branches or pull requests

3 participants