Skip to content

Uploading to Transloadit using tus

Jump to bottom
Marius edited this page Oct 22, 2018 · 17 revisions

Overview

Historically, beyond importing files from sources, Transloadit had only one way for getting files to its servers for processing them. This is directly integrated with the creation of an Assembly by sending a POST request to the /assemblies endpoint. Alongside with the access keys and the assembly instructions, the actual files had to be inside this request via multipart, to be parsed by node-formidable.

Using the tus protocol for uploading these files, the approach is a bit different. Basically, it is divided into two steps:

  1. An Assembly is created using the same endpoint as with the "traditional" way but instead of containing the actual files, you only define how many files you want to upload using tus, along with the (a reference to the) encoding instructions you would normally provide. An Assembly URL returned. This can be regarded as the encoding endpoint.
  2. The files themselves are now uploaded using tus. During its file creation you will pass along the Assembly URL that you have already received. The tusd server will 'feed' this encoding endpoint with references to the uploaded files, as they complete. The Assembly will know when it has received all file references from tus, or when it is still pending. The Assembly can however start processing files as soon as the first file upload completes.

This was the short version, now follows the more comprehensive!

Creating a Resumable Assembly

A POST request must be sent to the https://api2.transloadit.com/assemblies endpoint. As in the traditional approach (see https://transloadit.com/docs/api-docs/#create-a-new-assembly), its body must be multipart-encoded and must contain the params parameter which contains the usual information about authentication and instructions. However, additionally the num_expected_upload_files parameter must be included. Its value is the integer number of how many uploads will be performed using the tus protocol.

Following snippet contains an example HTTP request, which wants to perform two tus uploads:

POST /assemblies HTTP/1.1
Host: api2.transloadit.com
Content-Type:multipart/form-data; boundary=----WebKitFormBoundaryIAWBI8vxocZzsG03

------WebKitFormBoundaryIAWBI8vxocZzsG03
Content-Disposition: form-data; name="params"

{"auth":{"key":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"},"steps":{"encode":{"robot":"/image/resize"}}}
------WebKitFormBoundaryIAWBI8vxocZzsG03
Content-Disposition: form-data; name="num_expected_upload_files"

2
------WebKitFormBoundaryIAWBI8vxocZzsG03--

After we have received a successful response from the server, step #1 is completed and we can move on! Following code block shows part of the response body containing which would follow the above request

{
    "ok": "ASSEMBLY_UPLOADING",
    "assembly_id": "b841ea401e1a11e7b37d7bda1b503cdd",
    "assembly_url": "http://api2.freja.transloadit.com/assemblies/b841ea401e1a11e7b37d7bda1b503cdd",
    "assembly_ssl_url": "https://api2-freja.transloadit.com/assemblies/b841ea401e1a11e7b37d7bda1b503cdd",
    "websocket_url": "https://api2-freja.transloadit.com/ws20277",
    "tus_url": "https://api2-freja.transloadit.com/resumable/files/",
    ...
}

Note: The response will still contain the status_endpoint property which was used when we first developed the tus integration but is now deprecated. Use the assembly_ssl_url property instead.

Uploading using tus

Transloadit is running a tus server using the tusd software to which you can upload your files to. Its URL is provided by the tus_url property in the response to the POST request, e.g. https://api2-freja.transloadit.com/resumable/files/. You must use this URL as the endpoint for your tus client to upload the files for the specific assembly.

In order to tell the server to which Assembly a tus upload belongs, you have to add special metadata to the tus creation. In total, there are three values which must be presented in the metadata (if you miss one of them, the upload will silently be dropped without any error but we will probably change this in the future):

  • assembly_url: The Assembly's URL to which the upload belongs. You can obtain the value from the previous POST response value assembly_ssl_url
  • filename: The file's name which will be uploaded in this tus procedure.
  • fieldname: The equivalent to input field names in HTML forms.

Here's one example, in which are uploading a file called isaac.png to the Assembly 14b1b490447d11e6aba4756b3e9d3a0d with a field name of file-input:

POST /resumable/files/ HTTP/1.1
Content-Length: 0
Host: api2-freja.transloadit.com
Tus-Resumable: 1.0.0
Upload-Length: 24213
Upload-Metadata: assembly_url aHR0cHM6Ly9hcGkyLnRyYW5zbG9hZGl0LmNvbS9hc3NlbWJsaWVzLzE0YjFiNDkwNDQ3ZDExZTZhYmE0NzU2YjNlOWQzYTBk,filename aXNhYWMucG5n,fieldname ZmlsZS1pbnB1dA==

After that, most of the work is done. What's still to be done is that you have to upload the actual file content to the tus upload URL using the usual procedure using PATCH requests.

If one tus upload if finished, Transloadit will automatically process it using the parameters that you created the Assembly with - without requiring you to do anything special. Until all tus uploads have been finished the Assembly will be in the ASSEMBLY_UPLOADING state (although in reality, the first files may be processed already). If you are trying to add more tus uploads to an Assembly than you specified during the Assembly creation, it will be silently dropped. In the future, we want to error out hard on this.

Getting progress (and more) events over Web Sockets

When you do the POST request to https://api2.transloadit.com/assemblies, it also returns a property called websocket_url. For example https://api2-freja.transloadit.com/ws20277. A Socket.io server is listening at this URL, providing you with real-time information about your assembly when connecting to it.

After you have opened a connection to the Web Socket URL, you have to tell the API that you want events for a particular assembly by sending a "assembly_connect" event with {id: [[assembly_id]]} as the argument.

You will then receive the following events:

  • "assembly_uploading_finished",
  • "assembly_upload_meta_data_extracted"
  • "assembly_finished"
  • "assembly_upload_finished" (file)
  • "assembly_result_finished" (file)

Upload progress must currently be handled by the client. In the near future, we will also provide a "progress" websocket event.

Clone this wiki locally