aggregator | Code , docs and discussion related to the Aggregator | Socket library

Spring Integration - Producer Queue capacity limitations

We are using Remote partitioning with MessageChannelPartitionHandler to send partition messages to Queue(ActiveMQ) for workers to be pick and process. The job has huge data to process , many partition messages are publishing to queue and the aggregator of response from replyChannnel is failing with timeout of messages as all messages cant be processed in a given time. We also tried to limit messages published to queue by using queue capacity which resulted into server crash with heap dump generated due to memory issues of holding all these partition messages in internal memory.

We wanted to control the creation of StepExecution split itself , so that memory issue doesn’t occur. Example case is around 4k partition messages are being published to queue and whole job takes around 3hrs.

Can we control the publishing of messages to QueueChannel?

So, I'm using FlatFileItemWriter to write a csv file from data that I can successfully read from a database.

I'm struggling with how to write an integer number (i.e., row counter) corresponding to the row that I'm writing to the file. Seems like an easy thing to do, but quite simply I am stumped.

Everything is working (file is being produced from the data being read from a database). But I just can't seem to figure out how to implement my getCount() method in a way that gets me the corresponding row's count. I'm thinking it has something to do with leveraging the ChunkContext, but I can't seem to figure it out.

So I have the following in bean in my job configuration.

I have a Spark dataframe on which I am doing certain operations as follows. I wanted to know how do I skip processing certain records going through all the operations

Grateful for your help. I found this sample script to extract a PDF to a text file: https://gist.github.com/vinovator/c78c2cb63d62fdd9fb67

This works, and it is probably the most accurate extraction I've found. I would like to edit it to loop through multiple PDFs and write them to multiple text files, all with the same name as the PDF they were created from. I'm struggling to do so and keep either only writing one text file, or overwriting the PDFs I'm trying to extract from. Anyone able just to help me with a loop that will loop through all PDFs in a single folder and extract them to individual text files of the same name as the PDF?

Thanks in advance for your help!

I'm trying to implement something similar to Akka Streams statefulMapConcat... Basically I have a Flux of scores something like this:

Score(LocalDate date, Integer score)

I want to take these in and emit one aggregate per day:

ScoreAggregate(LocalDate date, Integer scoreCount, Integer totalScore)

So I've got an aggregator that keeps some internal state that I set up before processing, and I want to flatmap over that aggregator which returns a Mono. The aggregator will only emit a Mono with a value if the date changes so you only get one per day.

I have compiled a smart contract that is supposed to take bets from 2 addresses, a bet creator and a bet taker. The bet is on the price of ETH/USD (via ChainLink).

What would be the best way to for the smart contract to listen to the price of ETH/USD constantly, so that whenever the price would reach one or the other side of the bet, the contract would generateBetOutcome() automatically?

is it possible to test if eventAggregator.publish was executed during another publish?

I'm solving a MILP problem with CPLEX called from Julia JuMP package. In CPLEX log the number of solutions is displayed to be more than 3000, but there is the parameter CPXPARAM_MIP_Limits_Solutions set to 55, so the solver should return when the number of solution is more than 55. The explosion of the number of solution causes an out of memory error, therefore the Linux kernel kills the process.

This is the log:

I have a local Kubernetes install based on Docker Desktop. I have a Kubernetes Service setup with Cluster IP on top of 3 Pods. I notice when looking at the Container logs the same Pod is always hit.

Is this the default behaviour of Cluster IP? If so how will the other Pods ever be used or what is the point of them using Cluster IP?

The other option is to use a LoadBalancer type however I want the Service to only be accessible from within the Cluster.

Is there a way to make the LoadBalancer internal?

If anyone can please advise that would be much appreciated.

UPDATE:

I have tried using an LoadBalancer type and the same Pod is being hit all the time also.

Here is my config: