Latest / Logosofia no Dia a Dia / Private Buckets, Pre-Signed URLs: Self-Hosting Without the Compromise

Transcript

Private Buckets, Pre-Signed URLs: Self-Hosting Without the Compromise · Logosofia no Dia a Dia

• [Prompt] custom: I talked in a previous episode about setting up an image processing pipeline for a home inventory sy | Hosts: corn, herman [Response] Corn: Daniel sent us this one — he's been building out a fork of Homebox for his home inventory system, and he hit a wall that I think a lot of self-hosters hit. You've got a server packed in a box during a move, you've got thousands of inventory photos, and the local-first philosophy that made sense when the server was plugged in suddenly becomes the thing that makes the whole system useless exactly when you need it most. His question is basically: can you have cloud image storage with actual authentication — not security by obscurity — plus server-side image processing, plus a local backup, without handing your data over to some opaque service? Herman: The answer is yes, and the mechanism is pre-signed URLs on a private bucket, which sounds arcane until you realize it's about five lines of Go. But the interesting thing here is the tension Daniel's naming. The self-hosting impulse is totally reasonable — you don't want photos of serial numbers and shipping labels sitting in someone else's infrastructure with a guessable URL. And the usual compromise is either a public bucket with UUID filenames, which is just hoping nobody finds them, or proxying every image through your app server, which is burning CPU and bandwidth for no good reason. Corn: The UUID approach being what I'd call the "please don't look at my stuff" school of security. Herman: And that falls apart the moment a URL leaks in a log file or a browser history. So the real question Daniel's asking is whether you can decouple image storage from the physical server without decoupling it from your control. And the architecture that does that — private R2 bucket, pre-signed URLs with short expiry, Workers for image processing, rclone for local backup — that's worth walking through step by step, because it flips the whole local-versus-cloud framing on its head. Corn: It's not a compromise. It's actually better security than most fully local setups, because access is authenticated and time-limited instead of just... being on a hard drive in your closet. Herman: And the timing matters here. Two things have changed in the last couple years that make this viable for self-hosters in a way it wasn't before. One, Cloudflare R2 launched with zero egress fees — so pulling down a full local backup costs nothing beyond the fifteen cents per gigabyte per month for storage. Two, the Homebox ecosystem has enough forks and variants now that people are actually running these inventory systems in production, not just tinkering. The "server in a box during a move" problem isn't hypothetical anymore. Corn: Which is the moment you realize your careful self-hosted setup is just a very expensive paperweight in a cardboard box in the back of a moving truck. Herman: That's the contradiction Daniel's naming. The local-first philosophy says keep everything on premises. But the practical reality of mobility — moving apartments, traveling, even just wanting to check your inventory from your phone while you're at a store — says you need some cloud component. The question is whether you can do it without becoming the product. Corn: Let's walk through the actual mechanism. Daniel mentioned pre-signed URLs, and I think for a lot of people who've used S3 or R2 in the basic sense — upload a file, get a public URL — the idea that you can have a completely private bucket that still serves images to authenticated users sounds like magic. Herman: It basically is magic if you've only ever used public buckets. The mental model most people have is: I upload a file, I get back a URL, anyone with that URL can see the file. End of story. But S3-compatible storage — and R2 implements the full S3 API — has a completely different mode where the bucket itself is locked down, no public access at all, and you generate temporary access credentials on a per-object basis. Corn: Instead of the bucket saying "come on in," your application server says "I'll walk you to the door and the key stops working in fifteen minutes. Herman: And the reason this solves Daniel's PII problem is that photos of serial numbers and shipping labels are never sitting at a publicly accessible URL. The only way to fetch an image is to first authenticate with the inventory app, which then generates a signed URL that's valid for a narrow window. Someone intercepting that URL gets access to one image for a few minutes, not the entire bucket forever. Corn: Which is a fundamentally different threat model than "hope nobody guesses the UUID. Herman: And the false dichotomy Daniel was stuck in — either fully local or fully cloud — dissolves once you realize the cloud component doesn't have to be public. You can have a private bucket that behaves like an extension of your local storage, with your application server as the sole gatekeeper. The images live in R2, but access is mediated by your own authentication logic. Corn: The processing piece fits into this same architecture. Instead of running a separate image server to generate thumbnails, you put a Worker in front of the bucket that handles resizing on the fly. Herman: Which means your app just requests the image with a width parameter, the Worker fetches the original from R2, resizes it at the edge, caches the result, and serves it. No dedicated processing infrastructure, no pre-generated thumbnails taking up storage, and the full-resolution original is always available when you need to zoom in on a serial number. Corn: The architecture in one sentence: private bucket, authenticated gateway, edge processing, local backup. Four pieces that together give you cloud convenience without cloud surrender. Herman: Let's get into the mechanism, because the pre-signed URL is the piece that makes the whole thing work. The bucket is private — no public access, no anonymous reads. Your application server holds the access key and secret key. When a user requests an image, the server doesn't serve the image itself. It generates a URL that's cryptographically signed with those credentials, and that signature is what R2 validates before serving the object. Corn: The signature is time-bound. Herman: AWS Signature V4 — which R2 fully supports because it implements the S3 API — includes a timestamp and an expiration in the signed URL itself. So you generate a URL that says "this is valid for the next fifteen minutes, and here's the cryptographic proof that I, the bucket owner, authorized it." R2 checks the signature, checks the timestamp, and either serves the object or returns a four-oh-three. Corn: Which means the application server is the gatekeeper, but it's not the delivery mechanism. The heavy lifting of actually transferring the image file happens directly between the client and R2's edge. Herman: That's the key efficiency win. In a proxy model, every image request flows through your app server — the client asks the server, the server fetches from storage, the server streams it back to the client. You're paying for bandwidth twice and burning CPU on your server for what's essentially a pass-through operation. With pre-signed URLs, the server does a tiny amount of work — generate a signature, return a redirect — and then steps out of the way entirely. Corn: You get the authentication benefit of a proxy without the resource cost. The server says "here's your ticket" and the client goes directly to the source. Herman: The S3 API compatibility is what makes this not a Cloudflare-specific trick. R2 implements the same API that S3 uses, which means you're writing code against the AWS SDK for Go — specifically the S3 client — not against some proprietary Cloudflare interface. If you ever wanted to move to Backblaze B2 or Wasabi or straight S3, you change the endpoint URL and the credentials. The pre-signed URL generation code doesn't change. Corn: That's the vendor lock-in concern that Daniel didn't even raise but is implicitly solved. He's using R2 because it's cost-efficient, but the architecture isn't married to it. Herman: And the code is genuinely minimal. In Go, using aws-sdk-go-v2, you create an S3 client pointed at your R2 endpoint, then you use s3.NewPresignClient to get a presigner. Calling PresignGetObject on that presigner with the bucket name, object key, and a time.Duration for expiry — that's it. Three to five lines. The SDK handles the signature generation, the timestamp encoding, all of it. Corn: The flow in practice: user opens the inventory app, clicks on an item photo. The browser sends a request to your Go server — something like slash images slash item four two seven. The server checks the session cookie, confirms this user is authenticated, then calls PresignGetObject with a fifteen-minute expiry. It gets back a signed URL and returns an HTTP three-oh-two redirect to that URL. The browser follows the redirect and fetches the image directly from R2's nearest edge node. Herman: That fifteen-minute window is configurable. You could set it to five minutes if you're paranoid about serial numbers, or an hour if it's just a photo of a bookshelf. The point is that even if someone intercepts that URL — through a compromised browser extension, a man-in-the-middle attack, whatever — they have a narrow window to access one specific image. They can't enumerate the bucket, they can't list objects, they can't access anything else. Corn: Compare that to the UUID approach. Public bucket, every image has a random filename like "a three f seven b nine d two dot jpeg." The theory is nobody can guess that. But the moment that URL appears in a browser history, a server log, a shared link, a screenshot — it's permanently accessible. There's no expiration, no authentication, no audit trail. You're one accidental paste away from exposing every photo whose filename someone saw. Herman: The UUID approach doesn't scale in terms of trust. Every person you ever share a link with now has permanent access to that image. With pre-signed URLs, sharing an image means generating a new signed URL for that specific recipient, potentially with a different expiry. You can revoke access by simply not generating new URLs. You can't un-see a UUID. Corn: The PII angle is what makes this more than a theoretical preference. Daniel's photographing shipping labels with addresses, serial numbers that could be used for warranty fraud, maybe credit card receipts stapled to manuals. That stuff shouldn't be sitting at a publicly reachable URL, even one with a random name. A private bucket with pre-signed URLs means those images are never publicly accessible, period. The only path to them runs through your authentication system. Herman: Here's the thing that flips the usual assumption: this is arguably more secure than storing those images on a local server with no authentication at all. A lot of self-hosted setups serve static files from the local filesystem — if you're on the local network, you can access them. There's often no per-image access control, no expiry, no audit log. Pre-signed URLs on a private bucket give you finer-grained control than "well, I trust everyone on my home network. Corn: Which is the real "best of both worlds" Daniel was circling. You're not trading security for convenience. You're getting better security and better availability in the same move. Herman: That's the authentication mechanism — private bucket, pre-signed URLs, your app server as the gatekeeper. But once those images are in R2, there's a whole other layer Daniel raised: processing. You don't want to serve full-resolution photos to a phone screen. And you definitely don't want to pre-generate thumbnails and store them separately. Corn: Because now you're maintaining two sets of files — originals and thumbnails — and hoping they stay in sync, and paying for storage twice, and writing cleanup logic for when an item gets deleted. It's the kind of thing that starts simple and turns into a background headache you never quite fix. Herman: The alternative is doing the processing at the edge, on demand. Cloudflare Workers sit between the client and R2. When a request comes in for an image with query parameters — say slash images slash item four two seven dot jpeg question mark width equals two hundred — the Worker intercepts it before it hits the bucket. Corn: The Worker is a programmable middleman that runs on Cloudflare's edge network, not on your server. Herman: The Worker receives the request, parses the width parameter, then fetches the original full-resolution image from the private R2 bucket. It runs the image through the Workers runtime's built-in image manipulation APIs — resize, compress, convert format — and returns the optimized version to the client. And here's the part that makes this actually viable in production: the result gets cached at the edge. Corn: The first request for a two-hundred-pixel-wide thumbnail is a little slow, and every subsequent request is served from cache instantly. Herman: The Worker isn't reprocessing the same image on every request. Cloudflare's cache stores the resized version, and subsequent requests for the same dimensions skip the Worker entirely — they're served directly from the edge cache. You're only paying the processing cost once per unique size per image. Corn: Which means your app can just request whatever dimensions it needs — a tiny thumbnail for the list view, a medium image for the detail page, full resolution for zooming in on a serial number — and the Worker handles all of it from a single stored original. Herman: This is where the architecture gets elegant. You upload the full-resolution photo once. That's it. No preprocessing step, no thumbnail generation queue, no separate storage for resized variants. The Worker becomes your entire image processing pipeline, and it only does work when a new size is requested for the first time. Corn: The processing infrastructure is effectively zero — no dedicated server, no background jobs, no cron task that generates thumbnails. It's just a Worker script deployed to Cloudflare's edge. Herman: The Worker itself is surprisingly simple. You check for the width parameter in the URL, you fetch the object from R2 using the Workers runtime's fetch API — Workers have first-class R2 bindings, so you don't even need to worry about authentication between the Worker and the bucket — then you call the image manipulation API with the desired dimensions, set the cache headers, and return the response. Corn: The R2 binding being the thing that keeps this secure. The Worker has authorized access to the private bucket, but the outside world never sees that access path. The client only sees the Worker's URL. Herman: Now, the backup piece is where Daniel's architecture really closes the loop. All of this cloud infrastructure is great until your internet goes down during a move, or Cloudflare has an outage, or you just want the peace of mind of having your data physically in your possession. Corn: This is where R2's pricing model changes the calculus entirely. Zero egress fees means pulling down every image in your bucket costs nothing beyond the storage itself. With S3, a full backup of a large bucket would incur meaningful egress charges. With R2, it's free. Herman: The backup strategy is rclone. You configure R2 as a remote type — rclone supports it natively, same as it supports S3, Backblaze, Google Drive, everything else. Then you run a sync command that pulls the entire bucket down to local storage. Corn: The command is basically one line. Herman: Rclone sync r2 colon inventory hyphen images slash data slash inventory hyphen backup dash dash progress dash dash checksum. The checksum flag is the important part — instead of comparing timestamps or file sizes, rclone compares actual file checksums. If the checksum matches, the file hasn't changed and it skips it. You're only downloading new or modified images. Corn: The daily cron job runs, checksums get compared, and ninety-nine percent of the time nothing transfers because you haven't added new inventory photos. But when you have, only those new files come down. Herman: R2 supports object versioning, which means your backup tool can detect not just new objects but modified ones. If you replace an image — say you retake a blurry photo of a serial number — the version changes, the checksum changes, and rclone pulls the new version. Corn: The offline scenario is where this pays off. Server's in a box, internet's not set up yet at the new place, but you need to check which box has the kitchen knives. Your app detects that the CDN is unreachable and falls back to the local backup path. Same file structure, same filenames, just served from slash data slash inventory hyphen backup instead of through the pre-signed URL mechanism. Herman: Because rclone preserves the object keys as file paths, the fallback is transparent. The database stores relative paths or object keys, and your app just switches the base URL from the Worker endpoint to a local file server. No database migration needed when you go offline. Corn: The whole thing becomes a self-healing system. Cloud goes down, you run local. Internet comes back, rclone syncs any changes that happened while you were offline — though realistically during a move you're probably not adding inventory items — and you're back to the full cloud architecture without any manual intervention. Herman: The cost reality makes this practical for a home inventory. R2 charges one point five cents per gigabyte per month for storage. If Daniel's got, say, ten gigabytes of inventory photos — that's thousands of images — he's paying fifteen cents a month. The write operations cost thirty-six cents per gigabyte, but you only pay that once when you upload. The backup sync costs nothing in egress. The Worker invocations for image processing are minimal because of edge caching. We're talking maybe a dollar a month total. Corn: For a system that's available during a move, has authenticated access control, processes images on the fly, and maintains a complete local backup. That's the "best of both worlds" Daniel was reaching for — and it's not a compromise. Each piece reinforces the others. Herman: Let's pull this together into something you can actually build this weekend. The architecture is four pieces: private R2 bucket, pre-signed URLs for authenticated access, a Cloudflare Worker for on-the-fly image processing, and rclone for local backup. That's it. Each piece solves one specific problem, and none of them require you to surrender control of your data. Corn: The implementation order matters. Start with the bucket — create it in the Cloudflare dashboard, mark it private, generate an access key and secret key from the R2 console. Those keys never leave your application server. They're what let your Go code sign URLs, and they're what rclone uses to authenticate the backup sync. Herman: Step two is the pre-signed URL generation in your app. Add aws-sdk-go-v2 to your Go module, configure the S3 client with your R2 endpoint and those credentials, then wrap image requests through s3.You're replacing whatever currently serves static image files with a three-line function that returns a redirect to a signed URL with a fifteen-minute expiry. Corn: Step three, deploy the Worker. It intercepts image requests, checks for width or format parameters, fetches the original from R2 using the Worker's R2 binding, resizes on the fly, and caches the result. This is maybe thirty lines of JavaScript or TypeScript deployed through the Cloudflare dashboard or wrangler CLI. Herman: Step four, configure rclone. Set up R2 as a remote type, point it at your bucket, and schedule a daily cron job with rclone sync and the checksum flag. That's your safety net. If Cloudflare has an outage, if you're mid-move with no internet, your app falls back to the local copy and you're still functional. Corn: The migration path for someone already running Homebox or a fork: upload your existing image directory to R2 using the S3 API or rclone copy, update your database to store object keys instead of local file paths, flip the switch to pre-signed URL mode. No downtime, no data loss, and you can run both paths in parallel during the transition. Herman: The cost reality bears repeating because it's what makes this not just technically elegant but actually practical. Ten gigabytes of inventory photos — that's thousands of images — costs fifteen cents a month in R2 storage. The write operations when you upload are a one-time cost of a few cents per gigabyte. The backup sync is free. The Worker invocations are pennies. You're looking at well under a dollar a month for a system that's available during moves, has proper authentication, and maintains a complete local backup. Corn: That's the number that should make self-hosters reconsider the reflexive "cloud bad" instinct. For less than the cost of a single coffee per year, you solve the server-in-a-box problem, you get better access control than most local setups, and you still have all your data on a hard drive in your house. Herman: The open question, though — and this is where I'd want to think harder before calling it done — is failover detection. You've got the local backup sitting there, but how does your app know when to switch? Corn: That's the part most tutorials skip. The happy path works great until Cloudflare has an outage and your app just... Herman: You can't just check whether R2 is reachable on every request — that adds latency to every image load. But you could run a lightweight health check in the background, something that pings the Worker endpoint every thirty seconds, and if it fails three times in a row, flip a flag that tells the image handler to serve from local storage instead. Corn: The flip back is trickier than it sounds. You don't want to thrash between local and cloud if the connection is flaky. You need hysteresis — switch to local fast, but wait until the cloud endpoint has been stable for a solid minute or two before switching back. Herman: That's a interesting systems problem in what looks like a simple home inventory app. And it's the kind of thing that makes me wonder whether this architecture generalizes beyond Daniel's use case. Paperless-ngx has the same tension — documents you need to access anywhere, but you don't want your tax returns in a public bucket. Nextcloud has it. Any self-hosted app where the data matters and the server isn't guaranteed to be online. Corn: The pre-signed URL pattern plus local backup plus edge processing — that's not an inventory app architecture. That's a template for self-hosted applications that need to survive the physical world. Herman: The physical world is the part the cloud-native crowd and the self-hosting purists both tend to forget. Internet goes down. Boxes get packed. The architecture that wins isn't the one that's purest on paper — it's the one that still works when you're standing in an empty apartment holding a phone. Corn: Daniel's real insight wasn't about image processing or pre-signed URLs specifically. It was recognizing that the false choice between local and cloud was the actual problem, and then finding the mechanisms that dissolve it. And now: Hilbert's daily fun fact. Hilbert: In early Renaissance Iceland, fish was preserved by burying it in volcanic ash for six to eight weeks — the ash's alkalinity chemically "cooked" the fish in a process similar to lutefisk, meaning a family's winter food supply was measured in cubic feet of fermented shark, which converts to roughly seven gallons of edible protein per cubic foot. Corn:...I need to sit with that one. Herman: Seven gallons of fermented shark per cubic foot. This has been My Weird Prompts. If you enjoyed this episode, leave us a review wherever you listen — it helps more people find the show. We're back next week.