dataflow | experimental self-hosted Observable notebook editor

I have a Python Apache Beam streaming pipeline running in Dataflow. It's reading from PubSub and writing to GCS. Sometimes I get errors like "Error in _start_upload while inserting file ...", which comes from:

My dataflow job has both source & sink as synapse database.

I have a source query with joins & transformations in the dataflow while extracting data from the synapse database.

As we know, dataflow under the hood will spin up the databricks cluster to execute the dataflow code.

My question here, the source query I am using in the data flow will that be executed on the synapse db/databricks cluster?

Getting the following error when I try to launch a Dataflow SQL job:

Failed to start the VM, launcher-____, used for launching because of status code: INVALID_ARGUMENT, reason: Error: Message: Invalid value for field 'resource.networkInterfaces[0].network': 'global/networks/default'. The referenced network resource cannot be found. HTTP Code: 400.

This issue just started today.

Just to get it right, I would like to have your opinion if I am right with my imagination of how the dataflow is between a C# programm calling a C++ dll with delegates as parameter.

1. The System gives memory to the C# program
2. The C# Program loads the .dll and gives some of its space to the C++ dll. In this space there will be no C# Garbage Collection, only if the .dll is unloaded and then there can be freed the whole space.
3. A C++ function is called. The specific Function has a delegate as parameter. We dive into the C++ memory area and declare some variables. The C++ function will somewhere in its Code call the C# delegate.
4. The C# delegate operates on the C# Memory and will have a copy of its input parameters in the C# memory, if they are native types or a reference to the variables in the C++ memory, if it is a complex type. If we have native types I can just save it into the C# world and all will be fine. But if it is a reference and I just save it into my C# memory, I will get undefined behaviour, if I end my C++ function, because the variables will get out of scope and will be destroyed.
5. The C# function ends and we get the returnvalue in C++ as copy (or a pointer to the returnvalue, if it is a complex type, the pointer will point into the C# memory)
6. the C++ function ends and the used memory of the C++ function is released

Am I right with this?

I am trying to run a beam job on dataflow using the python sdk.

My directory structure is :

I'm using Apache Beam 2.28.0 on Google Cloud DataFlow (with Scio SDK). I have a large input PCollection (bounded) and I want to limit / sample it to a fixed number of elements, but I want to start the downstream processing as soon as possible.

Currently, when my input PCollection has e.g. 20M elements and I want to limit it to 1M by using https://beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/transforms/Sample.html#any-long-

I am trying to set up a new custom condition for Azure monitor alert rule, but when I enter my kql query it doesn't show the expected data. When I run the same query in Logs it outputs 9 rows that fulfil my condition, but for some reason, no data are shown in the Monitor Alerts.

I can see that the problem is in the last condition | where Anomaly has "1" as I get data when I delete this condition - but I need to have it included in the query (or at least a similar version of it). Any suggestions? (I have tried also contains and == but it gives the same problem)

With Dataflow SQL I would like to read a Pub/Sub topic, enrich the message and write the message to a Pub/Sub topic.

Which Dataflow SQL query will create my desired output message?

Pub/Sub input message: {"event_timestamp":1619784049000, "device":{"ID":"some_id"}}

Desired Pub/Sub output message: {"event_timestamp":1619784049000, "device":{“ID":"some_id", “NAME”:”some_name”}}

What I get is: {"event_timestamp":1619784049000, "device":{"ID":"some_id"}, "NAME":"some_name" }

but I need the NAME inside the “device” attribute.

I am using a Python POST request to geocode the addresses of my company's branches, but I'm getting wildly inaccurate results.

I looked at this answer, but the problem is that some results aren't being processed. My problem is different in that all of my results are inaccurate, even ones with Confidence="High". And I do have an enterprise account.

Here's the documentation that shows how to create a geocode Job and upload data:
https://docs.microsoft.com/en-us/bingmaps/spatial-data-services/geocode-dataflow-api/create-a-geocode-job-and-upload-data

here's a basic version of my code to upload: