Google Dataflow stalled after BigQuery outage -
i have google dataflow job running. dataflow job reading messages pub/sub, enrich , write enriched data bigquery.
dataflow processing approximately 5000 messages per second. using 20 workers run dataflow job.
yesterday seems there bigquery outage. writing data in bigquery part failed. after time, dataflow stopped working. see 1000 errors below
(7dd47a65ad656a43): exception: java.lang.runtimeexception: com.google.api.client.googleapis.json.googlejsonresponseexception: 400 bad request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "the project xx-xxxxxx-xxxxxx has not enabled bigquery.", "reason" : "invalid" } ], "message" : "the project xx-xxxxxx-xxxxxx has not enabled bigquery.", "status" : "invalid_argument" } com.google.cloud.dataflow.sdk.util.bigquerytableinserter.insertall(bigquerytableinserter.java:285) com.google.cloud.dataflow.sdk.util.bigquerytableinserter.insertall(bigquerytableinserter.java:175) com.google.cloud.dataflow.sdk.io.bigqueryio$streamingwritefn.flushrows(bigqueryio.java:2728) com.google.cloud.dataflow.sdk.io.bigqueryio$streamingwritefn.finishbundle(bigqueryio.java:2685) com.google.cloud.dataflow.sdk.util.dofnrunnerbase.finishbundle(dofnrunnerbase.java:159) com.google.cloud.dataflow.sdk.runners.worker.simplepardofn.finishbundle(simplepardofn.java:194) com.google.cloud.dataflow.sdk.runners.worker.forwardingpardofn.finishbundle(forwardingpardofn.java:47) com.google.cloud.dataflow.sdk.util.common.worker.pardooperation.finish(pardooperation.java:65) com.google.cloud.dataflow.sdk.util.common.worker.maptaskexecutor.execute(maptaskexecutor.java:77) com.google.cloud.dataflow.sdk.runners.worker.streamingdataflowworker.process(streamingdataflowworker.java:719) stack trace truncated. please see cloud logging entire trace.
please note dataflow not working bigquery started working. had restart dataflow job make work.
this causes data loss. not @ time of outage, until notice error , restart dataflow job. there way configure retry option dataflow job not go stale on these cases?
Comments
Post a Comment