bigtable | Abstraction layer for big tables

I am connecting to my GCP BigTable instance using python (google-cloud-bigtable library), after setting up "GOOGLE_APPLICATION_CREDENTIALS" in my environment variables. I'm successful at doing this.

However, my requirement is that I want to pass the credentials during the run-time while creating the BigTable Client object as shown below:

I need to store user interactions for 7 days in an existing BigTable table whose row key is user identifier. There are two types of interactions and we should be able to retrieve interaction history of each user in the order of time. It's obvious that the column family should have 7days as TTL and the column should contain type of interaction. I'm thinking about two options for the column, {interaction_type}:{timestamp} with the latest 1 cell and {interaction_type} with multiple cells. As the GCP bigtable doc doesn't recommend too many columns in a row, so the latter looks more reasonable. However, the column should be retrieved along with other existing columns designed by the former schema (including timestamp in column and the latest 1 cell), if I choose the latter one, the query should use interleaved filters due to the different number of cells for columns. So I wondered which one would show better read performance. Also wondered implications of one column with multi cells vs multi columns with one cell and chain filter vs interleaved filter in terms of performance in BigTable.

I have a pretty big code base written in Java. I have a lot of integration tests with both Kafka and Bigtable using JUnits ExternalResource. I have introduced fetching of secrets from GCP Secret Manager in my code. I now want to write integration tests for that as well.

So my scenario is that I want to create a mock username/password, create a secret of that username/password in my mock GCP Secret Manager, access the secrets and then use it to connect to my mock-service that requires that username/password. So, in reality, I'm connecting to a Kafka broker in my test with SSL and I want to simulate the entire flow with fetching of secrets.

The problem is, I can't find any documentation on how to do it. Google has great other documentation on how to emulate Bigtable, but I can't find any documentation on how to emulate/mock a Secret Manager. Has anyone ran into something similar?

I've tried to insert 100 values into a column test_col in column family test_cf in a row key test-123

The problem is that I successfully inserted 100 values into Bigtable.

However, The number of values in the test_col column in test_cf is less than 100 and It appears to be randomly inserted.

The code I wrote is below.

I need to to trigger a Data Fusion pipeline located on a GCP project called myDataFusionProject through a Data Fusion operator (CloudDataFusionStartPipelineOperator) inside a DAG whose Cloud Composer instance is located on another project called myCloudComposerProject.

I have used the official documentation as well as the source code to write the code that roughly resembles the below snippet:

I'm writing a Dataflow pipeline using Apache beam to add large batches of rows of data to Bigtable.

• apache-beam==2.24.0
• google-cloud-bigtable==2.4.0

I have the following method used in my pipeline to create the Bigtable row(s) prior to writing to Bigtable:

We are considering using Beam/Dataflow for stateful processing:

• Real-time aggregation of metrics on global windows (every 1min)
• Real-time aggregation on a high number of parallel sessions (> 1 mio)

Example: get max price article bought for each 1 mio clients since registered on a portal

Also, we would also like to access those calculated aggregates while not interfering with the real-time job.

Design question : can it be covered by the current state back-end - Windmill/Persistent Disks [1] - or would the use of a database (like BigTable) be a better fit ?

Thanks !

[1] Dataflow - State persistence?

I am aware that BigTable supports operations append and increment using ReadModifyWriteRow requests, but I'm wondering if there is support or an alternative way to use more generic mapping functions where the value from the cell can be accessed and modified within some sort of closure? For instance, bitwise ANDing a long value in a cell:

I need to view the data in a BigTable table, but I can't find a data browser in the web console. (Dynamo has a nice browser in the AWS web console.) Is there a data browser for BigTable, or am I limited to the cbt command line?