datacatalog | Data Catalog is a service

If you use Scala Play, for each case class you should have an companion object which will help you a lot with read/write object in/from json:

Thought I should close this question off with what I eventually discovered. The proper question isn't what role is assigned an instance of a resource, but what users have been allowed to use the resource and with what role.

The proper call is GetIamPolicy which is available in the APIs for all of the resources that I've been working with. The problem was that I wasn't seeing anything due to no user accounts being assigned to the resource. I updated the Terraform script to assign a user to the resource with the required roles. When calling GetIamPolicy, it returns an array in the Bindings that lists roles and users that are assigned. This was the information I needed. Going down the path of using TestIamPermissions was unneeded.

Here's an example my use of this:

It looks like the S3 bucket has multiple nested folders inside it. For Glue to read these folders you need to add a flag adding additional_options = {"recurse": True} to your from_catalog(). This will help to recursively read records from s3 files.

Move load_args inside dataset

If you want to create a tag template and attach a tag along with the values to the BigQuery table programmatically using Python API, refer to this quickstart.

If you are facing the same error (PermissionDenied : 403 Permission denied on resource project) as the OP, refer to the answer by the OP himself.

If you want to do the same through console, follow the below steps. For more information, refer to this doc.

• Create a new tag template by navigating through Data Catalog -> Tag templates -> Create tag templates (Provide the “Template id”, “Location” and “Fields” based on your requirements and click create)

• To attach a tag to a table in your dataset, navigate through Data Catalog -> Search, enter your dataset/table name in the Search box, then click search. From the results, select your table which you want to attach a tag.

• After selecting the table, click “Attach Tags” and select your table name in “Choose what to tag”, then choose the tag template (which you have newly created) in “Choose the tag templates” and add values based on the display name and type as per your requirement in “Fill in tag values” and click save. The “Attach Tags” page will appear as shown below:

• You can check the created tag with values under the “Tags” on the table page in Data Catalog as below:

To be able to specifically update the metadata only you need to create a custom role. To do this you can follow the steps below:

• Open your Google Cloud console

• Select "IAM & Admin" -> Roles

• Click "+ Create Role"

• Edit "Title" to provide a descriptive role title. I used "Custom BigQuery metadata update"

• Click "+ Add Permissions"

• At the filter bar, put bigquery.tables.update and click "ADD"

  • Permission bigquery.tables.update allows you to update the TABLE METADATA ONLY. See permission list for reference.

• Click "Create"

Once done, the created custom role that contains only bigquery.tables.update should be searchable when assigning roles in IAM.

It looks like the coordinates of World Bank barrier JSON don't have a winding order that vega-lite interprets properly. I suspect this makes it think some islands are lakes, and puts the land outside them rather than inside.

Using https://github.com/mapbox/geojson-rewind fixed it:

It's a bug (harmless) in definition of the google provider 2.2.0 in fact:

In provider.yaml:

airflow.providers.google.common.hooks.leveldb.LevelDBHook

should be:

airflow.providers.google.leveldb.hooks.LevelDBHook

This was fixed in https://github.com/apache/airflow/pull/15453 and will be available in next version of google provider.

It was a silly mistake because I was not creating the Kedro session. To load an item of the catalog it can be done with the following code: