aggregateD | dogstatsD inspired metrics and event aggregation daemon | Monitoring library

I have a big problem when I want to make a view.

I have a Dataframe that can have multiple columns of Array type like "Array1", "Array2" ... etc. These array columns would have same number of elements. I need to compute a new column of Array type which will be the sum of arrays element wise. How can I do it ?

Spark version = 2.3

For Ex:

Input:

I am probably having a failry easy question but cannnot figure it out.

I am having a dataset that has two variables, both factors. It looks like this:

When reading about CQRS it is often mentioned that the write model should not depend on any read model (assuming there is one write model and up to N read models). This makes a lot of sense, especially since read models usually only become eventually consistent with the write model. Also, we should be able to change or replace read models without breaking the write model.

However, read models might contain valuable information that is aggregated across many entities of the write model. These aggregations might even contain non-trivial business rules. One can easily imagine a business policy that evaluates a piece of information that a read model possesses, and in reaction to that changes one or many entities via the write model. But where should this policy be located/implemented? Isn't this critical business logic that tightly couples information coming from one particular read model with the write model?

When I want to implement said policy without coupling the write model to the read model, I can imagine the following strategy: Include a materialized view in the write model that gets updated synchronously whenever a relevant part of the involved entities changes (when using DDD, this could be done via domain events). However, this denormalizes the write model, and is effectively a special read model embedded in the write model itself.

I can imagine that DDD purists would say that such a policy should not exist, because it represents a business invariant/rule that encompasses multiple entities (a.k.a. aggregates). I could probably agree in theory, but in practice, I often encounter such requirements anyway.

Finally, my question is simply: How do you deal with requirements that change data in reaction to certain conditions whose evaluation requires a read model?

I have this data

I want to build functionality for selected objects and shown models in my Aggregated View. I can't seem to figure out how to use the "getSelection" method that is available to the Viewer3D (which Aggregated View is built on?). I can getModel easy enough though:

I am working with a nested data structure which needs to be flattened. The values need to be aggregated so totals are produced across each level of the nested data. I'm trying to do this recursively but it's not clear how best to achieve this?

The following is an example of the data I'm working with.

Spring Integration - Producer Queue capacity limitations

We are using Remote partitioning with MessageChannelPartitionHandler to send partition messages to Queue(ActiveMQ) for workers to be pick and process. The job has huge data to process , many partition messages are publishing to queue and the aggregator of response from replyChannnel is failing with timeout of messages as all messages cant be processed in a given time. We also tried to limit messages published to queue by using queue capacity which resulted into server crash with heap dump generated due to memory issues of holding all these partition messages in internal memory.

We wanted to control the creation of StepExecution split itself , so that memory issue doesn’t occur. Example case is around 4k partition messages are being published to queue and whole job takes around 3hrs.

Can we control the publishing of messages to QueueChannel?

I have two dataframes df1 and df2