Uploading User Files to S3: Keys, Multipart, and What to Retry

Uploading a file to object storage is three lines in any S3 SDK, and those three lines are correct. They keep being correct until a real user turns up: a 3 GB video over hotel wifi, a file called `Screenshot 2026-07-14 (1).png`, a laptop lid closing at 80%. What breaks then has nothing to do with the `PutObject` call, which is why the tutorial that taught you the call is no help.

This is about the part that comes after. How to name an object you'll never be able to rename, when a single request stops being enough, which failures you can repeat and which you can't, and why a storage bill can exceed the sum of everything visible in the bucket. It assumes the writing side is already settled: if you're still choosing between proxying uploads through your application and letting the browser write to storage directly, presigned URLs covers that decision and the security model behind it. For buckets, keys, and prefixes as concepts, how S3-compatible object storage works sets up the vocabulary used here.

The object key is the one decision you can't take back

In a filesystem you rename things. Object storage has no rename. A key is fixed when the object is written, and changing it means copying the object to the new key and deleting the old one, which reads and rewrites every byte. That falls out of a property the pillar article covers: objects are written whole, never edited in place. A server-side copy handles up to 5 GB in a single call on AWS as of writing, and larger objects have to be copied in parts.

With four objects in a bucket this is a footnote. With four million it's a migration, and the key is usually your URL as well, so anything already handed out breaks when you move it. Ten minutes of thought before the first upload is cheaper than any amount of thought after the hundred-thousandth.

Don't let the client name the object

Accepting the browser's filename as the key is the default in most tutorials and it fails in ordinary use, before anyone tries to attack you. Two users upload `avatar.jpg` and one overwrites the other. Keys are case-sensitive, so `Photo.JPG` and `photo.jpg` are two separate objects, which surprises everyone whose laptop says otherwise. Spaces and parentheses need escaping every time the key becomes a URL, and they will. Keys cap out at 1024 bytes of UTF-8, so a long name behind a long prefix can hit a ceiling nobody thought about. The security side of the same decision, including what a client-chosen key can reach in your bucket, is in the presigned URL risk table.

What a durable key looks like

• An opaque identifier from the server. A UUID or ULID, generated where the request is authorized. Random means unguessable, which matters because a key is half of a URL.
• A prefix carrying whatever you'll need to sweep by. Listing works on prefixes and nothing else, so `user/42/2026/07/` lets you find one tenant's month and `uploads/` lets you find nothing in particular.
• The extension, kept as a hint. Some clients infer a content type from it, so it's useful. It's a claim by the uploader, not a fact about the bytes.
• The display name stored somewhere else. Object metadata is specified as US-ASCII, so a Cyrillic or emoji filename needs encoding to survive there. The database row that tracks the file is the better home for it, and it's the row you'll join against anyway.

import datetime
import pathlib
import uuid


def upload_key(user_id: str, original_name: str) -> str:
    extension = pathlib.Path(original_name).suffix.lower()[:10]
    day = datetime.date.today().isoformat()
    return f"user/{user_id}/{day}/{uuid.uuid4().hex}{extension}"


s3.put_object(
    Bucket="uploads",
    Key=upload_key("42", "Screenshot 2026-07-14 (1).png"),
    Body=data,
    ContentType="image/png",
)

Serving the file back with its original name is then a `Content-Disposition` header at download time, set from the database row. The key stays boring forever.

One PUT, or many?

A single `PutObject` sends the whole body in one request, capped at 5 GB on AWS as of writing. The cap is rarely what stops you. What stops you is the failure mode: one dropped connection at 90% and the transfer restarts from zero, because a half-finished PUT leaves nothing behind to resume from.

Multipart usually gets explained as a speed feature. The parallelism is real, but throughput isn't what should decide it. A 2 GB upload over a mobile connection has a serious chance of dying somewhere in the middle, and the gap between losing 16 MB and losing 2 GB is the gap between a retry the user never notices and a feature they stop using.

How multipart actually works

A multipart upload is a single object sent as a set of independently uploaded parts, assembled by the storage into one object at the end. It's three operations, shaped like a transaction:

1. `CreateMultipartUpload` returns an upload ID. Nothing is stored at the key yet. The ID is the handle for everything that follows.
2. `UploadPart` sends one chunk, tagged with that upload ID and a part number. Parts can go in any order, over as many connections as you like, and each response carries an `ETag` you have to keep.
3. `CompleteMultipartUpload` sends the list of part numbers and their ETags. The storage assembles them. Until this call succeeds, no object exists at the key.

The limits are worth memorising because they constrain your design. As of writing, part numbers run from 1 to 10,000, every part except the last must be at least 5 MiB, and no part may exceed 5 GiB. One line of arithmetic follows: part size × 10,000 is the largest file that part size can carry. At the 5 MiB minimum you top out around 50 GB. Reaching the 5 TB object ceiling needs parts of at least 500 MiB.

Which makes part size a real decision rather than a constant to copy off a blog. Small parts mean cheap retries and more requests; large parts mean fewer requests and more to re-send when one fails. Somewhere between 8 and 64 MiB suits most application uploads, and the rule that matters more than the exact number is that ten thousand of your parts have to clear your largest plausible file.

from boto3.s3.transfer import TransferConfig

config = TransferConfig(
    multipart_threshold=64 * 1024 * 1024,  # one PutObject below this
    multipart_chunksize=16 * 1024 * 1024,  # part size above it
    max_concurrency=8,                     # parts in flight
)

# Chooses the strategy, splits the file, retries failed parts,
# and aborts the upload if it gives up.
s3.upload_file("render.mp4", "uploads", key, Config=config)

The parts you forgot about are still on the bill

`CompleteMultipartUpload` is what turns parts into an object. Nothing turns them back. An upload that stops halfway, because the process was killed or the client crashed or someone shipped a deploy, leaves its parts in the bucket under an upload ID nobody owns any more. They don't show up in a listing of objects, since there is no object. They do occupy storage, and AWS bills them as storage until the upload is aborted.

There are two ways to deal with it. `ListMultipartUploads` shows what's in flight and `AbortMultipartUpload` discards one, which is enough when you're cleaning up after a known incident. For the ongoing case, a lifecycle rule with `AbortIncompleteMultipartUpload` sweeps anything older than N days without you thinking about it. Lifecycle configuration is one of the areas where S3-compatible providers diverge, so it belongs in the compatibility checks you run against a live bucket rather than in your assumptions.

Worth keeping separate from transfer costs: zero-egress storage means moving bytes isn't metered, not that occupying space is free. Where transfer is metered, the arithmetic is its own problem and S3 egress fees works through it.

Where the SDK's help stops

The `upload_file` call above is doing more than it looks. It picks single-PUT or multipart based on the threshold, splits the file, runs parts concurrently, retries the ones that fail, and aborts the upload if it eventually gives up. boto3 calls this a managed transfer; the JavaScript SDK has the same thing in `Upload` from `@aws-sdk/lib-storage`. It's the right answer whenever your own code holds the credentials, which covers a server accepting a proxied upload, a worker moving a render into storage, or a scheduled job pushing a database dump.

The constraint hides in that clause. A managed transfer constructs and signs every request itself, so it needs the secret key. Browser code never has one, which is the whole premise of direct uploads with presigned URLs. The helper is unavailable exactly where a large file upload hurts most, and nobody mentions this until you're halfway through the feature.

Multipart straight from a browser therefore looks different. Your server does the signing, one part at a time, and the client runs the state machine:

1. The client asks to start an upload. Your server authorizes it, generates the key, calls `CreateMultipartUpload`, and returns the upload ID.
2. For each part the client asks for a signed URL, or collects a batch up front. The server signs `upload_part` for that key, upload ID, and part number.
3. The client PUTs each part to its own URL and keeps the `ETag` from the response, retrying individual parts on failure.
4. The client sends back the part numbers and ETags it collected, and your server calls `CompleteMultipartUpload`.

# Your server signs one part at a time. The browser never sees a key.
part_url = s3.generate_presigned_url(
    "upload_part",
    Params={
        "Bucket": "uploads",
        "Key": key,
        "UploadId": upload_id,
        "PartNumber": part_number,
    },
    ExpiresIn=900,
)

That's four endpoints and a client-side loop where the server-side version was one function call. Reading the `ETag` off each part response also means the bucket's CORS configuration has to list it as an exposed header, or the browser hides it from your JavaScript even though it arrived. Keeping bytes out of your application costs you this loop, which is worth knowing before you promise someone resumable uploads.

What's safe to retry

Uploads fail in ways that are mostly retryable. The useful question is which operations you can repeat without reasoning about them first.

For 5xx responses and throttling, exponential backoff with jitter is standard and every AWS SDK does it by default up to a configurable attempt count. What the SDK won't decide is how many times your application should try before it tells the user something went wrong.

The ETag is not a checksum for multipart objects

Retries protect you against transfers that fail loudly. They say nothing about a transfer that succeeded with the wrong bytes, and the usual homegrown answer to that is quietly broken.

For an object uploaded in a single PUT without SSE-C or SSE-KMS encryption, the ETag happens to be the MD5 of the body, and plenty of verification code depends on that coincidence. It stops holding the moment the object arrives in parts. A multipart object's ETag is a hash over the concatenated part hashes, followed by a dash and the part count, so `-14` at the end tells you the file came in fourteen pieces and nothing at all about its contents. It also shifts with the part size used, meaning the same file uploaded with different settings produces different ETags.

Integrity you can act on has to be requested. `Content-MD5` on a single PUT makes the storage reject a body that doesn't match what you declared, and the additional checksum algorithms (CRC32, CRC32C, SHA-1, SHA-256 as of writing) work per part as well as per object. Support for the newer ones varies across S3-compatible providers. The approach that works everywhere is computing your own checksum and storing it in the row that tracks the file.

Progress, cancel, and resume in the browser

A progress bar is the thing users notice missing. `fetch()` reports progress on the response, not the request: there's no upload progress event as of writing, and streaming request bodies with `duplex: "half"` are supported unevenly enough that you can't rely on them. So a progress bar in a browser still means `XMLHttpRequest` and its `upload.onprogress`. Multipart hands you a second, coarser signal at no cost, since parts completed over parts total is already a percentage.

const xhr = new XMLHttpRequest();

xhr.upload.onprogress = (event) => {
  if (event.lengthComputable) setProgress(event.loaded / event.total);
};

xhr.open("PUT", presignedUrl);
xhr.setRequestHeader("Content-Type", file.type);
xhr.send(file);

Cancelling is `xhr.abort()` or an `AbortController`, and it stops the request without cleaning up after it. A cancelled multipart upload still needs an `AbortMultipartUpload` from your server, and a cancelled single PUT may well have landed anyway.

Resuming an interrupted upload is possible only with multipart, and only if the client kept its state: the upload ID, the part numbers that succeeded, and their ETags. Persist that in `localStorage` or IndexedDB and a page reload can pick up where the connection died, as long as the upload ID hasn't been swept by your lifecycle rule in the meantime. A single PUT has no resume story to offer. It starts from zero, every time, forever.

After the bytes land

Once the browser writes to storage directly, your application is out of the byte path and learns about the file only if you arrange for it. The shape of that arrangement is a database row in a `pending` state, a callback from the client, and a `HeadObject` that checks what actually arrived against what you signed for before the row becomes `ready`. It's worked through step by step in the presigned URL article, so it isn't repeated here.

Two jobs stay yours regardless of who did the uploading. The content type on the object was declared by the uploader, so when the distinction matters, read the first bytes and check them against the extension instead of trusting either. And serving files back is a separate decision from storing them: assets meant for everyone belong in a public bucket behind a CDN, while anything private wants a short-lived presigned GET.

How Runsite handles it

Runsite's S3-compatible object storage speaks the standard API, so everything above is the same code with a different endpoint. `upload_file` with a `TransferConfig` in boto3, `Upload` from `@aws-sdk/lib-storage`, `aws s3 cp` at the terminal: they work once the endpoint and access keys point at Runsite, with multipart the recommended path for anything above 100 MB. Objects go up to 5 TB, and prefixes are how you organise them, since `uploads/2026/07/` is a naming convention rather than a directory. Creating a bucket, issuing a key pair, and pointing an SDK at the endpoint is walked through in the Runsite docs.

Direct browser uploads are covered by presigned URLs and per-bucket CORS configuration. Access keys carry a read, write, delete, or admin scope and rotate without downtime, so signing uploads with a write-scoped key keeps the blast radius of a leaked URL inside a boundary you drew yourself. Private buckets stay private; public ones are served through a global CDN with automatic cache headers.

Transfer isn't metered in either direction. Accepting an upload costs nothing whatever its size, and re-sending a 2 GB transfer that died at 90% costs nothing either, which takes the sting out of the retry logic above. Stored gigabytes are the only variable on the invoice, at €0.025 per GB per month after 5 GB free as of writing. Objects are encrypted at rest with AES-256 and in transit with TLS, they live in the Frankfurt, Germany region, and a signed GDPR Data Processing Agreement comes with every plan. User uploads from an app running on Runsite Web Services land in the same account behind the same keys, which is the common case this whole article describes.

The short version

Generate the object key on the server, keep it opaque, and put the user's filename in your database instead, because renaming an object means copying it. Use a single PUT for small predictable files and multipart above roughly 100 MB, remembering that part size × 10,000 caps the file you can send, and that you split a file to make failures cheap rather than to go faster. Abort incomplete uploads, by lifecycle rule if your provider supports one, or pay to store parts you can't see. Managed transfer helpers need credentials, so direct browser multipart means your server signs each part and your client tracks them. Retries are safe for whole objects and individual parts, an ETag ending in `-N` proves nothing about content, and the object only counts as real once your server has confirmed it with a `HeadObject`. To push a file through the whole path before committing to it, 5 GB is free, which is enough to sign a URL, upload in parts, and watch the key appear.