Resumable Uploads for HTTP

HTTP clients often encounter interrupted data transfers as a result of canceled requests or dropped connections. Prior to interruption, part of a representation may have been exchanged. To complete the data transfer of the entire representation, it is often desirable to issue subsequent requests that transfer only the remainder of the representation. HTTP range requests support this concept of resumable downloads from server to client. This document describes a mechanism that supports resumable uploads from client to server using HTTP.¶

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress”.¶

This Internet-Draft will expire on April 21, 2024.¶

Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

HTTP clients often encounter interrupted data transfers as a result of canceled requests or dropped connections. Prior to interruption, part of a representation (see Section 3.2 of [HTTP]) might have been exchanged. To complete the data transfer of the entire representation, it is often desirable to issue subsequent requests that transfer only the remainder of the representation. HTTP range requests (see Section 14 of [HTTP]) support this concept of resumable downloads from server to client.¶

HTTP methods such as POST or PUT can be used by clients to request processing of representation data enclosed in the request message. The transfer of representation data from client to server is often referred to as an upload. Uploads are just as likely as downloads to suffer from the effects of data transfer interruption. Humans can play a role in upload interruptions through manual actions such as pausing an upload. Regardless of the cause of an interruption, servers may have received part of the representation before its occurrence and it is desirable if clients can complete the data transfer by sending only the remainder of the representation. The process of sending additional parts of a representation using subsequent HTTP requests from client to server is herein referred to as a resumable upload.¶

Connection interruptions are common and the absence of a standard mechanism for resumable uploads has lead to a proliferation of custom solutions. Some of those use HTTP, while others rely on other transfer mechanisms entirely. An HTTP-based standard solution is desirable for such a common class of problem.¶

This document defines an optional mechanism for HTTP than enables resumable uploads in a way that is backwards-compatible with conventional HTTP uploads. When an upload is interrupted, clients can send subsequent requests to query the server state and use this information to the send remaining data. Alternatively, they can cancel the upload entirely. Different from ranged downloads, this protocol does not support transferring different parts of the same representation in parallel.¶

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

The terms Byte Sequence, Item, String, Token, Integer, and Boolean are imported from [STRUCTURED-FIELDS].¶

The terms client and server are from [HTTP].¶

Resumable uploads are supported in HTTP through use of a temporary resource, an upload resource, that is separate from the resource being uploaded to (hereafter, the target resource) and specific to that upload. By interacting with the upload resource, a client can retrieve the current offset of the upload (Section 5), append to the upload (Section 6), and cancel the upload (Section 7).¶

The remainder of this section uses an example of a file upload to illustrate different interactions with the upload resource. Note, however, that HTTP message exchanges use representation data (see Section 8.1 of [HTTP]), which means that resumable uploads can be used with many forms of content -- not just static files.¶

When a resource supports resumable uploads, the first step is creating the upload resource. To be compatible with the widest range of resources, this is accomplished by including the Upload-Complete header field in the request that initiates the upload.¶

As a consequence, resumable uploads support all HTTP request methods that can carry content, such as POST, PUT, and PATCH. Similarly, the response to the upload request can have any status code. Both the method(s) and status code(s) supported are determined by the resource.¶

Upload-Complete MUST be set to false if the end of the request content is not the end of the upload. Otherwise, it MUST be set to true. This header field can be used for request identification by a server. The request MUST NOT include the Upload-Offset header field.¶

If the request is valid, the server SHOULD create an upload resource. Then, the server MUST include the Location header field in the response and set its value to the URL of the upload resource. The client MAY use this URL for offset retrieval (Section 5), upload append (Section 6), and upload cancellation (Section 7).¶

Once the upload resource is available, the target resource MAY send an informational response with a 104 (Upload Resumption Supported) status code to the client while the request content is being uploaded. In this informational response, the Location header field MUST be set to the upload resource.¶

The server MUST send the Upload-Offset header field in the response if it considers the upload active, either when the response is a success (e.g. 201 (Created)), or when the response is a failure (e.g. 409 (Conflict)). The Upload-Offset field value MUST be equal to the end offset of the entire upload, or the begin offset of the next chunk if the upload is still incomplete. The client SHOULD consider the upload failed if the response has a status code that indicates a success but the offset indicated in the Upload-Offset field value does not equal the total of begin offset plus the number of bytes uploaded in the request.¶

If the request completes successfully and the entire upload is complete, the server MUST acknowledge it by responding with a successful status code between 200 and 299 (inclusive). Servers are RECOMMENDED to use 201 (Created) unless otherwise specified. The response MUST NOT include the Upload-Complete header field with the value of false.¶

If the request completes successfully but the entire upload is not yet complete, as indicated by an Upload-Complete field value of false in the request, the server MUST acknowledge it by responding with the 201 (Created) status code and an Upload-Complete header value set to false.¶

If the request includes an Upload-Complete field value set to true and a valid Content-Length header field, the client attempts to upload a fixed-length resource in one request. In this case, the upload's final size is the Content-Length field value and the server MUST record it to ensure its consistency.¶

If the client received an informational response with the upload URL in the Location field value, it MAY automatically attempt upload resumption when the connection is terminated unexpectedly, or if a 5xx status is received. The client SHOULD NOT automatically retry if it receives a 4xx status code.¶

File metadata can affect how servers might act on the uploaded file. Clients can send representation metadata (see Section 8.3 of [HTTP]) in the request that starts an upload. Servers MAY interpret this metadata or MAY ignore it. The Content-Type header field (Section 8.3 of [HTTP]) can be used to indicate the MIME type of the file. The Content-Disposition header field ([RFC6266]) can be used to transmit a filename; if included, the parameters SHOULD be either filename, filename* or boundary.¶

If an upload is interrupted, the client MAY attempt to fetch the offset of the incomplete upload by sending a HEAD request to the upload resource.¶

The request MUST NOT include an Upload-Offset or Upload-Complete header field. The server MUST reject requests with either of these fields by responding with a 400 (Bad Request) status code.¶

If the server considers the upload resource to be active, it MUST respond with a 204 (No Content) status code. The response MUST include the Upload-Offset header field, with the value set to the current resumption offset for the target resource. The response MUST include the Upload-Complete header field; the value is set to true only if the upload is complete.¶

An upload is considered complete only if the server completely and successfully received a corresponding creation request (Section 4) or append request (Section 6) with the Upload-Complete header value set to true.¶

The client MUST NOT perform offset retrieval while creation (Section 4) or append (Section 6) is in progress.¶

The offset MUST be accepted by a subsequent append (Section 6). Due to network delay and reordering, the server might still be receiving data from an ongoing transfer for the same upload resource, which in the client perspective has failed. The server MAY terminate any transfers for the same upload resource before sending the response by abruptly terminating the HTTP connection or stream. Alternatively, the server MAY keep the ongoing transfer alive but ignore further bytes received past the offset.¶

The client MUST NOT start more than one append (Section 6) based on the resumption offset from a single offset retrieving (Section 5) request.¶

In order to prevent HTTP caching, the response SHOULD include a Cache-Control header field with the value no-store.¶

If the server does not consider the upload resource to be active, it MUST respond with a 404 (Not Found) status code.¶

The resumption offset can be less than or equal to the number of bytes the client has already sent. The client MAY reject an offset which is greater than the number of bytes it has already sent during this upload. The client is expected to handle backtracking of a reasonable length. If the offset is invalid for this upload, or if the client cannot backtrack to the offset and reproduce the same content it has already sent, the upload MUST be considered a failure. The client MAY cancel the upload (Section 7) after rejecting the offset.¶

The client SHOULD NOT automatically retry if a client error status code between 400 and 499 (inclusive) is received.¶

Upload appending is used for resuming an existing upload.¶

The request MUST use the PATCH method and be sent to the upload resource. The Upload-Offset field value (Section 8.1) MUST be set to the resumption offset.¶

If the end of the request content is not the end of the upload, the Upload-Complete field value (Section 8.2) MUST be set to false.¶

The server SHOULD respect representation metadata received during creation (Section 4) and ignore any representation metadata received from appending (Section 6).¶

If the server does not consider the upload associated with the upload resource active, it MUST respond with a 404 (Not Found) status code.¶

The client MUST NOT perform multiple upload transfers for the same upload resource in parallel. This helps avoid race conditions, and data loss or corruption. The server is RECOMMENDED to take measures to avoid parallel upload transfers: The server MAY terminate any creation (Section 4) or append (Section 6) for the same upload URL. Since the client is not allowed to perform multiple transfers in parallel, the server can assume that the previous attempt has already failed. Therefore, the server MAY abruptly terminate the previous HTTP connection or stream.¶

If the offset indicated by the Upload-Offset field value does not match the offset provided by the immediate previous offset retrieval (Section 5), or the end offset of the immediate previous incomplete successful transfer, the server MUST respond with a 409 (Conflict) status code.¶

The server MUST send the Upload-Offset header field in the response if it considers the upload active, either when the response is a success (e.g. 201 (Created)), or when the response is a failure (e.g. 409 (Conflict)). The value MUST be equal to the end offset of the entire upload, or the begin offset of the next chunk if the upload is still incomplete. The client SHOULD consider the upload failed if the status code indicates a success but the offset indicated by the Upload-Offset field value does not equal the total of begin offset plus the number of bytes uploaded in the request.¶

If the request completes successfully and the entire upload is complete, the server MUST acknowledge it by responding with a successful status code between 200 and 299 (inclusive). Servers are RECOMMENDED to use a 201 (Created) response if not otherwise specified. The response MUST NOT include the Upload-Complete header field with the value set to false.¶

If the request completes successfully but the entire upload is not yet complete indicated by the Upload-Complete field value set to false, the server MUST acknowledge it by responding with a 201 (Created) status code and the Upload-Complete field value set to true.¶

If the request includes the Upload-Complete field value set to true and a valid Content-Length header field, the client attempts to upload the remaining resource in one request. In this case, the upload's final size is the sum of the upload's offset and the Content-Length header field. If the server does not have a record of the upload's final size from creation or the previous append, the server MUST record the upload's final size to ensure its consistency. If the server does have a previous record, that value MUST match the upload's final size. If they do not match, the server MUST reject the request with a 400 (Bad Request) status code.¶

The client MAY automatically attempt upload resumption when the connection is terminated unexpectedly, or if a server error status code between 500 and 599 (inclusive) is received. The client SHOULD NOT automatically retry if a client error status code between 400 and 499 (inclusive) is received.¶

If the client wants to terminate the transfer without the ability to resume, it can send a DELETE request to the upload resource. Doing so is an indication that the client is no longer interested in continuing the upload, and that the server can release any resources associated with it.¶

The client MUST NOT initiate cancellation without the knowledge of server support.¶

The request MUST use the DELETE method. The request MUST NOT include an Upload-Offset or Upload-Complete header field. The server MUST reject the request with a Upload-Offset or Upload-Complete header field with a 400 (Bad Request) status code.¶

If the server successfully deactivates the upload resource, it MUST respond with a 204 (No Content) status code.¶

The server MAY terminate any in-flight requests to the upload resource before sending the response by abruptly terminating their HTTP connection(s) or stream(s).¶

If the server does not consider the upload resource to be active, it MUST respond with a 404 (Not Found) status code.¶

If the server does not support cancellation, it MUST respond with a 405 (Method Not Allowed) status code.¶

The 301 (Moved Permanently) and 302 (Found) status codes MUST NOT be used in offset retrieval (Section 5) and upload cancellation (Section 7) responses. For other responses, the upload resource MAY return a 308 (Permanent Redirect) status code and clients SHOULD use new permanent URI for subsequent requests. If the client receives a 307 (Temporary Redirect) response to an offset retrieval (Section 5) request, it MAY apply the redirection directly in an immediate subsequent upload append (Section 6).¶

The upload resource URL is the identifier used for modifying the upload. Without further protection of this URL, an attacker may obtain information about an upload, append data to it, or cancel it.¶

To prevent this, the server SHOULD ensure that only authorized clients can access the upload resource. In addition, the upload resource URL SHOULD be generated in such a way that makes it hard to be guessed by unauthorized clients.¶

Header field name: Upload-Offset, Upload-Complete¶

Applicable protocol: http¶

Status: standard¶

Author/change controller: IETF¶

Specification: This document¶

Related information: n/a¶

Code: 104 (suggested value)¶

Description: Upload Resumption Supported¶

Specification: This document¶

The server is allowed to respond to upload creation (Section 4) requests with a 104 (Upload Resumption Supported) intermediate response as soon as the server has validated the request. This way, the client knows that the server supports resumable uploads before the complete response is received. The benefit is the clients can defer starting the actual data transfer until the server indicates full support (i.e. resumable are supported, the provided upload URL is active etc).¶

On the contrary, support for intermediate responses (the 1XX range) in existing software is limited or not at all present. Such software includes proxies, firewalls, browsers, and HTTP libraries for clients and server. Therefore, the 104 (Upload Resumption Supported) status code is optional and not mandatory for the successful completion of an upload. Otherwise, it might be impossible in some cases to implement resumable upload servers using existing software packages. Furthermore, as parts of the current internet infrastructure currently have limited support for intermediate responses, a successful delivery of a 104 (Upload Resumption Supported) from the server to the client should be assumed.¶

We hope that support for intermediate responses increases in the near future, to allow a wider usage of 104 (Upload Resumption Supported).¶

This specification includes a section about feature detection (it was called service discovery in earlier discussions, but this name is probably ill-suited). The idea is to allow resumable uploads to be transparently implemented by HTTP clients. This means that application developers just keep using the same API of their HTTP library as they have done in the past with traditional, non-resumable uploads. Once the HTTP library gets updated (e.g. because mobile OS or browsers start implementing resumable uploads), the HTTP library can transparently decide to use resumable uploads without explicit configuration by the application developer. Of course, in order to use resumable uploads, the HTTP library needs to know whether the server supports resumable uploads. If no support is detected, the HTTP library should use the traditional, non-resumable upload technique. We call this process feature detection.¶

1. Avoid additional roundtrips by the client, if possible (i.e. an additional HTTP request by the client should be avoided).
2. Be backwards compatible to HTTP/1.1 and existing network infrastructure: This means to avoid using new features in HTTP/2, or features which might require changes to existing network infrastructure (e.g. nginx or HTTP libraries)
3. Conserve the user's privacy (i.e. the feature detection should not leak information to other third-parties about which URLs have been connected to)

Include a support statement in the SETTINGS frame. The SETTINGS frame is a HTTP/2 feature and is sent by the server to the client to exchange information about the current connection. The idea was to include an additional statement in this frame, so the client can detect support for resumable uploads without an additional roundtrip. The problem is that this is not compatible with HTTP/1.1. Furthermore, the SETTINGS frame is intended for information about the current connection (not bound to a request/response) and might not be persisted when transmitted through a proxy.¶

Include a support statement in the DNS record. The client can detect support when resolving a domain name. Of course, DNS is not semantically the correct layer. Also, DNS might not be involved if the record is cached or retrieved from a hosts files.¶

Send a HTTP request to ask for support. This is the easiest approach where the client sends an OPTIONS request and uses the response to determine if the server indicates support for resumable uploads. An alternative is that the client sends the request to a well-known URL to obtain this response, e.g. /.well-known/resumable-uploads. Of course, while being fully backwards-compatible, it requires an additional roundtrip.¶

Include a support statement in previous responses. In many cases, the file upload is not the first time that the client connects to the server. Often additional requests are sent beforehand for authentication, data retrieval etc. The responses for those requests can also include a header field which indicates support for resumable uploads. There are two options: - Use the standardized Alt-Svc response header field. However, it has been indicated to us that this header field might be reworked in the future and could also be semantically different from our intended usage. - Use a new response header field Resumable-Uploads: https://example.org/files/* to indicate under which endpoints support for resumable uploads is available.¶

Send a 104 intermediate response to indicate support. The clients normally starts a traditional upload and includes a header field indicate that it supports resumable uploads (e.g. Upload-Offset: 0). If the server also supports resumable uploads, it will immediately respond with a 104 intermediate response to indicate its support, before further processing the request. This way the client is informed during the upload whether it can resume from possible connection errors or not. While an additional roundtrip is avoided, the problem with that solution is that many HTTP server libraries do not support sending custom 1XX responses and that some proxies may not be able to handle new 1XX status codes correctly.¶

Send a 103 Early Hint response to indicate support. This approach is the similar to the above one, with one exception: Instead of a new 104 (Upload Resumption Supported) status code, the existing 103 (Early Hint) status code is used in the intermediate response. The 103 code would then be accompanied by a header field indicating support for resumable uploads (e.g. Resumable-Uploads: 1). It is unclear whether the Early Hints code is appropriate for that, as it is currently only used to indicate resources for prefetching them.¶

When an upload is created (Section 4), the Content-Type and Content-Disposition header fields are allowed to be included. They are intended to be a standardized way of communicating the file name and file type, if available. However, this is not without controversy. Some argue that since these header fields are already defined in other specifications, it is not necessary to include them here again. Furthermore, the Content-Disposition header field's format is not clearly enough defined. For example, it is left open which disposition value should be used in the header field. There needs to be more discussion whether this approach is suited or not.¶

However, from experience with the tus project, users are often asking for a way to communicate the file name and file type. Therefore, we believe it is help to explicitly include an approach for doing so.¶

• Are multipart requests supported? Yes, requests whose content is encoded using the multipart/form-data are implicitly supported. The entire encoded content can be considered as a single file, which is then uploaded using the resumable protocol. The server, of course, must store the delimiter ("boundary") separating each part and must be able to parse the multipart format once the upload is completed.

This document is based on an Internet-Draft specification written by Jiten Mehta, Stefan Matsson, and the authors of this document.¶

The tus v1 protocol is a specification for a resumable file upload protocol over HTTP. It inspired the early design of this protocol. Members of the tus community helped significantly in the process of bringing this work to the IETF.¶

The authors would like to thank Mark Nottingham for substantive contributions to the text.¶