Large Directory Uploads

[MSevey]

Skynet Portals still don't support uploads of large directories.

Since tus doesn't work for directories, there have been stability concerns and a worry that any instability would reflect poorly on Skynet.

Previously, we ran tests and found directories stable up to 5GB. We did not test further.

Developer anesu reached out asking for the status of large uploads on skynetpro.

I propose we should uncapping their sizes for paid accounts, and consider requiring a flag to access it that acknowledges it's not normal Skynet usage (perhaps a reusable "experimental" flag).

We do actually have an endpoint that would in theory allow for uploading directories over TUS.

So basically tus allows for setting custom skyfile metadata now because that was a requirement for trustless uploads. So what you can do is concatenate all files on the client, construct the right metadata and then upload the dir as a seemingly single file to TUS with a custom metadata that lists all the subfiles.

There is no documentation on it but I might have something better.

You can go to the website and upload a small dir. e.g. I just uploaded https://10090cq75i79uog6mn0jaelndoosodofel22efp1c9dl7k6re0ksa08.skynetpro.net/

Then you use the base64 version of that link like this to get the metadata https://skynetpro.net/skynet/metadata/CACQM0csjp9iBrXBNTq3bjHMNw91RCc_IWJbU9DbcCnFAQ

Which results in

{
  "filename":"foo",

  "length":6165,

  "subfiles":{
    ".DS_Store":{
    "filename":".DS_Store",
    "contenttype":"application/octet-stream",
    "len":6148
    },
    "foo.txt":{
    "filename":"foo.txt",
    "contenttype":"text/plain",
    "offset":6148,
    "len":4
    }
    "bar/bar.txt":{
    "filename":"bar/bar.txt",
    "contenttype":"text/plain",
    "offset":6152,
    "len":4
    },
    "bar/tmp/tmp.txt":{
    "filename":"bar/tmp/tmp.txt",
    "contenttype":"text/plain",
    "offset":6156,
    "len":4
    },
    "bar2/bar2.txt":{
    "filename":"bar2/bar2.txt",
    "contenttype":"text/plain",
    "offset":6160,
    "len":5
    } 
  },

  "tryfiles":[
    "index.html"
  ]
 }

So you can use the old endpoint as a reference point.

Basically every file ends up in subfiles with the relative path as both the key and the filename. The first file .DS_Store has offset=0 which is dropped and a length of 6148. Then the next file foo.txt has offset 6148 and len 4 and so on.

The len of all subfiles adds up to the length one level up in the object.

tryfiles is currently set like this by default on the website. We can let users customize that in the sdk.