seneca | A microservices toolkit for Nodejs | Runtime Evironment library

Temporal joins using the AS OF syntax you're using require:

• an append-only table with a valid event-time attribute
• an updating table with a primary key and a valid event-time attribute
• an equality predicate on the primary key

When Flink SQL's temporal operators are applied to event time streams, watermarks play a critical role in determining when results are produced, and when the state is cleared.

When performing a temporal join:

• rows from the append-only table are buffered in Flink state until the current watermark of the join operator reaches their timestamps
• for the versioned table, for each key the latest version whose timestamp precedes the join operator's current watermark is kept in state, plus any versions from after the current watermark
• whenever the join operator's watermark advances, new results are produced, and state that's no longer relevant is cleared

The join operator tracks the watermarks it receives from its input channels, and its current watermark is always the minimum of these two watermarks. This is why your join stalls, and only makes progress when the flow_rate is updated.

One way to fix this would be to set the watermark for the TransportNetworkEdge_Kafka table like this:

The string version of a data type hint only works with SQL types. For POJOs and other classes, you can use @DataTypeHint(bridgedTo = AQIAccumulator.class).

Alternatively, you can simply override getTypeInference and provide all components programmatically.

But for your example Flink should be smart enough to derive all types automatically using reflection. No hints required.

I think you need to either group by the area or compute some aggregation over the areas included in the given sensor and window.

They are two different things.

philosophers::iterator is a function which when invoked, takes 0 arguments and returns an Iterator. As it happens, this is compatible with the definition of Iterable, since that interface has a single abstract method which does just that.

philosophers.iterator() does not return a function; it returns an Iterator. An Iterator is not an Iterable so the cast fails.

It is worth nothing that there's no good reason to do either of these. The first is basically only working by accident. That's a very unintuitive way to use a method reference. Just do

Below is a working version to adapt further for your needs. One overall recommendation is to start with a small working example before adding in more components/complexity.

Some of your errors came from how the data was being filtered. For example, you have:

To plot a pandas.DataFrame on a matplotlib subplot you need to:

1. Store the axis returned by plt.subplot() call to a variable that you can then
2. Pass this axis to pandas.DataFrame.plot() call

Like:

Try switching the CREATE statements for MERGE statements. In Cypher, MERGE will check to see if a pattern exists and create it if it does not. CREATE will create a new pattern every time, even if a pattern like it already exists in the graph.

I have been able to reproduce your issue, here are the steps I have followed:

1. Created a service account and assign Storage Object Creator permission.
2. Ran gcloud iam service-accounts keys create [FILE_NAME].json --iam-account [NAME]@[PROJECT_ID].iam.gserviceaccount.com to get account credentials file.
3. Erased lines 7 to 9 of your script and changed the filename of line 11 to match the name of the downloaded credentials file.
4. Ran the script. Here I get the same error as you.

The reason behind this error is that on line 14 you're getting your bucket object through get_bucket method. This method queries cloud Storage, requiring get permissions on your bucket but the Storage Object Creator role does not include get permissions.

To solve your issue just change line 14 with this code bucket = client.bucket(bucket_name) which directly creates a bucket object without interacting with Cloud Storage and therefore it does not raise the permission error, see corresponding reference.

Another solution might be to change the service account role to Storage Object Admin because it includes get permissions.