dlp | Deep learning platform based on tensorflow | Machine Learning library

If your records are stored separated by newlines in a text file I would recommend the following approach by opening the file, parsing the records, and adding them to a dict which you can later dump with the native json library.

It's a bug (harmless) in definition of the google provider 2.2.0 in fact:

In provider.yaml:

airflow.providers.google.common.hooks.leveldb.LevelDBHook

should be:

airflow.providers.google.leveldb.hooks.LevelDBHook

This was fixed in https://github.com/apache/airflow/pull/15453 and will be available in next version of google provider.

We no longer populate row_index for bigquery as it's not meaningful since BQ is unordered. If you want to identify the row where the finding came from, I suggest using identifyingFields which lives in BigQueryOptions when you create your job.

https://cloud.google.com/dlp/docs/creating-job-triggers#job-identifying-fields

var datasetIdList = List.empty[String] defines a mutable variable pointing to a immutable list.

val datasetidsFeeder = datasetIdList.map(datasetId => Map("datasetId" -> datasetId)).iterator uses the immutable list. Further changes to datasetIdList is irrelevant to datasetidsFeeder.

Mutating a global variable with your virtual user is usually not a good idea. You can save the value into the user's session instead.

In the exec block, you can write:

When you want Project Y (where your data fusion is in) to use resources from Project X (where the DLP is in) is to add the data fusion and compute engine service accounts of Project Y to Project X.

Notes:

• Data Fusion service account: service-xxxxxxx@gcp-sa-datafusion.iam.gserviceaccount.com
• Default compute engine service account: xxxxx-compute@developer.gserviceaccount.com

Project Y:

1. Go to IAM & Admin -> IAM
2. Click View by: "Members"
3. Tick checkbox "Include Google-provided role grants"
4. Look for service-(project number of Project Y)@gcp-sa-datafusion.iam.gserviceaccount.com and (project number of Project Y)-compute@developer.gserviceaccount.com
5. Add role "DLP Administrator" for service-(project number of Project Y)@gcp-sa-datafusion.iam.gserviceaccount.com

Project X:

1. Go to IAM & Admin -> IAM
2. Click Add
3. Under New Members, put service-(project number of Project Y)@gcp-sa-datafusion.iam.gserviceaccount.com
4. Grant role of "DLP Admininistrator"
5. Repeat step 2 to step 4 but this time put in (project number of Project Y)-compute@developer.gserviceaccount.com

Now that you are able to set the permissions, Go back to Project Y and update your Redact to point to Project X.

1. Go to Data Fusion -> Studio
2. Click Redact-> Properties
3. Put the template ID you created in Project X, in my sample it is "test_template"
4. Under Project ID, put the Project ID of Project X
5. Run your Data Fusion pipeline

Images are only supported in the RedactImage method.

Python doesn't use camel casing, but instead uses snake case.

See

https://cloud.google.com/dlp/docs/samples/dlp-deidentify-masking#dlp_deidentify_masking-python

so that should be cloud_storage_path

In order to further contribute to the community I am posting the answer based on my comment above.

Firstly, you should try to upgrade the packages using the command:

pip install --upgrade pandas-gbq 'google-cloud-bigquery[bqstorage,pandas]'

Then, instead of using the to_dataframe() method you can use the read_gbq(), which loads data from BigQuery using the environment's default project, as follows:

Issue Found:

Environmental variable was set using quotes. It should be without quotes.

set GOOGLE_APPLICATION_CREDENTIALS="path\to\secret\key"

Confusing debug message

google.auth.exceptions.DefaultCredentialsError: File "path\to\secret\key" was not found.

Resolution:

set GOOGLE_APPLICATION_CREDENTIALS=path\to\secret\key