Tom Clark
1 min readJun 14, 2021

--

Hi Harith, no I never got an actual answer. However, I was fishing around (for another reason) in django-storages the other week and I noticed an explicit statement about using your key to generate pre-signed URLs *without* having to roundtrip a request to GCP, which gave me some hope... but looking deep in the client I'm not sure it was true; there's a request to IAM built right in there here:

https://github.com/googleapis/python-storage/blob/0dbbb8ac17a4b632707485ee6c7cc15e4670efaa/google/cloud/storage/_signing.py#L677

However, what you could do is create a cloud load balancer on (say) uploads.yourdomain.com for your uploads, whose backend is a Google Cloud Function. Your server could use `itsdangerous` or an equivalent library to create many upload tokens in a fast loop (encrypted on some secret key, containing destination data) that it hands out to a client. Your client uploads to that endpoint. The cloud function (to which you've given the same key you used to encrypt) gets the file and request parameters, uses the key to decrypt the token. On success it shoves the file to its end destination, on failure returns a 403.

That way you have a scalable way of handling millions of uploads, without making yourself vulnerable to infinite uploads. You can also time limit the tokens or bake in whatever other data might be useful.

--

--

Tom Clark
Tom Clark

Written by Tom Clark

Fluid Dynamicist at the core, lover of chaos theory. Experienced scientist, developer and team lead working in wind energy — from startups to heavy industry.

No responses yet