7

I'm trying to find a way gracefully end my jobs, so as not to lose any data, streaming from PubSub and writing to BigQuery.

A possible approach I can envision is to have the job stop pulling new data and then run until it has processed everything, but I don't know if/how this is possible to implement.

4

2 回答 2

3

It appears this feature was added in the latest release.

All you have to do now is select the drain option when cancelling a job.

Thanks.

于 2016-08-22T22:45:37.020 回答
2

I believe this would be difficult (if not impossible) to do on your own. We (Google Cloud Dataflow team) are aware of this need and are working on addressing it with a new feature in the coming months.

于 2016-02-05T19:26:51.417 回答