abelcastro.dev

The booking form on thebest.ink asks a visitor for the usual things, a description of the tattoo they want, where on the body it goes, how to reach them, and then it asks for photos. Up to five reference images of the style they like, and up to five photos of the body part where it goes. Ten images at most, and any one of them can be up to five megabytes.

For a long time this was one POST. The browser packed everything into a multipart form, sent it to the API, and the API validated the images, wrote them to object storage, and created the row. One request, one response, nothing clever. It worked, which is why it survived as long as it did.

It does not scale, and the reason is more interesting than "uploads are big".

The arithmetic

Every byte crossed the server twice. Once on the way in from the visitor's phone, then again on the way out to the bucket. The server did nothing to those bytes in between. It was an expensive piece of wire.

The slow half of that trip is the phone. People fill this form in on mobile data, and a worst case submission is fifty megabytes going up a connection that was never built for it. For the entire length of that upload, one of my worker threads is occupied and doing nothing. It is not computing. It is holding a connection open while a phone trickles bytes through it.

So capacity was never really about requests per second. It was about how many people could be uploading at the same moment, and that number was small. The reverse proxy had to buffer request bodies of up to fifty megabytes, and to keep memory from running out I had to cap how many of those could be in flight at once. That cap was the ceiling on the whole feature. Timeouts had to be long enough to tolerate a bad connection, so one stalled phone held its slot for minutes.

A load test made the ceiling easy to see. The bottleneck was not the database, not the image handling, not anything I could rewrite. It was the decision to put the bytes on the request path at all, and there is no tuning my way out of that. The only fix is to move the bytes.

The shape of the fix

The browser uploads each image directly to object storage. The API never sees a single byte of image data.

The flow becomes three steps instead of one. The browser asks the API for permission to upload, describing what it wants to send. The API answers with a short lived, narrowly scoped credential per file. The browser uploads straight to the bucket. Then the form submits as it always did, except the request body is now a small JSON document naming the things that were uploaded, rather than carrying them.

A submission that could weigh fifty megabytes now weighs about two kilobytes. The proxy buffering limits and the in flight cap stopped being load bearing. The generous timeout became unnecessary. None of that was an optimisation in the usual sense, where the same work is made to run faster. The work simply stopped happening on that machine.

There is a second win that falls out for free. Ten images used to be one request that failed as a unit, so a single flaky upload at the end of a long form lost everything. Now each image is its own transfer with its own progress and its own retry, and the form submission is a small request that either works or does not. The failure modes got smaller and more honest.

What the server keeps

The instinct that makes people nervous about this is correct: it hands an anonymous browser a credential that writes to my bucket. The answer is that the credential is not general. The server decides everything about what that upload is allowed to be, and signs those decisions in.

The server picks the destination, so the holder cannot write anywhere else. The server pins the access level, so nothing lands publicly readable in a bucket that also serves public files. The server pins the content type and the exact size, both of which the browser already knows before it asks. The credential expires in minutes.

The same principle governs the second phase. When the form is finally submitted, the browser sends back a reference to what it uploaded, and the temptation is to let it name a storage location directly. That is the mistake. The server hands out a signed token naming the location, and only accepts that token back. Without it, a visitor can submit a form pointing at somebody else's uploaded files, and the performance improvement has quietly become a data leak.

The rule I ended up with: the client can move bytes, but the client never gets to choose facts. Every fact stays server side, signed, and short lived.

What it costs

This is not free, and the costs are the part that is missing from most write ups.

The server stops seeing the bytes, so it stops being able to prove anything about them. The old code knew an uploaded file was genuinely an image, because it had the file in hand. Pinning a content type is not the same thing: it constrains what a client claims, not what it sent. That check has to move to a background job after the fact, or be given up. The thing worth getting right is making that an explicit decision rather than something that silently stops happening.

One request became two phases, and phases leak. A visitor picks four images, they upload, and then the visitor closes the tab. Those files are now in the bucket, referenced by nothing, and no code path will ever look at them again. This is not an edge case, it is ordinary behaviour on a form that people abandon. Uploads that are never claimed need an expiry policy, and that policy lives in the storage provider rather than in the application.