Old NRT pipeline decommission

What

This is our near real time pipeline that write tracking data from Google Pub/Sub in near real-time using Google Cloud Dataflow to BigQuery table. The pipeline itself run on near-real-time-pipeline project using Dataflow SDK 1.9.0

Before we decommission this old pipeline, we've released the new version of the pipeline at tvlk-realtime project using Apache Beam 2.4.x ( latest version is 2.6.0). We've also asked respective team who own the table produced by the old pipeline to migrate their queries to the new table at tvlk-realtime project

You can see more detail on the new pipeline at https://bqmigration.page.link/spec

Why

Google will drop its support on Dataflow pipeline that use 1.9.x (link)
Explanation from the Google support (link)

You can still run AND update existing 1.9 pipelines in existing project(s) on or after August 15, 2018.

But you won't be able to do this with new projects created on or after August 15, 2018, even though you are passing in the same pipeline option parameters.

On October 16, 2018 = existing 1.9 pipeline will completely unsupported and the pipeline might start failing, regardless of when the project(s) are created.

In short, if you are not creating any new GCP projects, you can continue using the SDK until October 16, 2018.

When

We're going to stop our NRT dataflow pipeline at near-real-time-pipeline project at 15th October 2018 at 14.00 13:00 UTC +7

Who's affected

Anyone who's still accessing BigQuery nrtprod dataset from near-real-time-pipeline project

How

If you think you have a table on near-real-time-pipeline project under nrtprod dataset and never been contacted by us to migrate your table then please DM me (@zaka). We believe we have contacted each table owner and they almost finish their migration, you can see the progress at https://bqmigration.page.link/status