missing_t | missing I18n translations in your Rails project | Internationalization library

I have a streaming pipeline that takes messages from pub/sub, parses them, and writes to BigQuery. The challenge is that each message goes to a different event table based on the event property in the message, and they are not ordered.

This means (I believe) that the WriteToBigQuery method cannot efficiently batch the writes, I am seeing it basically write each message one at a time, and hence it is running too slowly. I have also tried adding a 60-second window and adding a GroupByKey/FlatMap to try to reorder them, with only minimal success at speeding it up.

Using the FILE_LOADS method in WriteToBigQuery with a 60+ second triggering frequency, it APPEARS to work, sending load jobs, which then (at least sometimes) succeed and I see the data go into the correct tables. BUT, the temporary tables that were created never get deleted, so I have hundreds of tables getting created (with names like beam_bq_job_LOAD_AUTOMATIC_JOB_NAME_LOAD_NAME_STEP_756_37417blahblahblah)...which is obviously not sustainable.

Writing via STREAMING_INSERTS works fine, just slowly, this is an attempt to make it more efficient.

If anybody could help me figure out why the tables aren't getting deleted that would I think give me a working, efficient pipeline. I've tried longer triggering frequencies (up to 1 hour) but the same behavior happens.

Here is my main pipeline - again, I don't have any issues with the rest of it, just providing for context.