Hi Neil,

This is really great, thank you. This is the most relevant article I've found recently, I hope you don't mind a long-running question!

At octue.com, we're doing something very similar (not using airflow, but essentially a set of steps in a DAG), using pubsub to broker long-running requests. But we're in the tall weeds when it comes to actually hosting that request:

- creating a serverless / low-maintenance target for the long running process to execute on seems tough (we got Cloud Run to work but the timeouts are too limited)

- deliver-once execute-once is important, especially if your long running process is costly or changes state somewhere, and we're making horrible botches to control this with pubsub and Cloud Run (we tried dataflow to solve the deliver-once problem but support for python in apache beam is extremely limited - no logging, limited dependencies)

It's fairly important to have a serverless solution for us, because we're fostering a network of scientific applications from different organisations - many services, each running a specific analysis occasionally. A server for each service would be overwhelmingly expensive.

Have you come across such a usage pattern before? And if so are there any resources you can point me to please?