tasklet | ️ A task scheduling library written in Rust | Job Scheduling library

I am trying to implement a Spring batch job where in order to process a record , it require 2-3 db calls which is slowing down the processing of records(size is 1 million).If I go with chunk based processing it would process each record separately and would be slow in performance. So, I need to process 1000 records in one go as bulk processing which would reduce the db calls and performance would increase. But my question is If I implement Tasklet then I would lose the functionality of restartability and retrial/skip features too and if implemented using AggregateInputReader I am not sure what would be the impact on restartability and transaction handling. As per the below thread AggregateReader should work but not sure its impact on transaction handling and restartability in case of failure:

Spring batch: processing multiple record at once

I am trying to write to an Azure Storage using Spring.

I am configuring the resource inside a Bean instead of Autowiring it from the Class.

How can I create a tasklet class to make a custom select query from DB and pass the data to the next tasklet? I have to use tasklet (no jdbcReader or any reader)

Code Sample:

Looking for any suggestions on possible solutions for running a Spring Batch app deployed in Kubernetes to access directories on a server, run commands, etc.

This app has two Jobs & uses a host of Tasklets to perform work using linux commands on the server. The Tasklets replace the existing script files.

Job A : take the daily file located in a directory on the server, move the file between different directories(prep the file), finally encrypt the file on the server & SFTP the file to a vendor.

Job B : Retrieve an acknowledgment file from the vendor : when the ack file is available from the vendor, we retrieve the file via SFTP, move it around some directories on the server.

Seems to be a fairly straight forward process but how an application in Kubernetes accesses directories & runs commands on a server has not been so straight forward based on the research we have done.

Thanks in advance for any suggestions.

In my Spring batch job, I'm trying to share data between steps using JobExecutionContext, which works only if i keep the steps single threaded as follows:

I have a spring batch job. It processes a large number of items. For each item, it calls an external service (assume a stored procedure or a REST service. This does some business calculations and updates a database. These results are used to generate some analytical reports.). Each item is independent, so I am partitioning the external calls in 10 partitions in the same JVM. For example, if there are 50 items to process, each partition will have 50/10 = 5 items to process. This external service can result a SUCCESS or FAILURE return code. All the business logic is encapsulated in this external service and therefore worker step is a tasklet which just calls the external service and receives a SUCCESS/FAILURE flag. I want to store all the SUCCESS/FAILURE flag for each item and get them when job is over. These are the approaches I can think of:

1. Each worker step can store the item and its SUCCESS/FAILURE in a collection and store that in job execution context. Spring batch persists the execution context and I can retrieve it at the end of the job. This is the most naïve way, and causes thread contention when all 10 worker steps try to access and modify the same collection.
2. The concurrent exceptions in 1st approach can be avoided by using a concurrent collection like CopyOnWriteArrayList. But this is too costly and the whole purpose of partitioning is defeated when each worker step is waiting to access the list.
3. I can write the item ID and success/failure to an external table or message queue. This will avoid the issues in above 2 approaches but we are going out of spring bath framework to achieve this. I mean we are not using spring batch job execution context and using an external database or message queue.

Are there any better ways to do this?

I am new to spring batch development. I have the following requirement. There will be a s3 source with zip files and each of the zipfiles will contain multiple pdf files and xml files.[Eg:100 pdfs and 100 xml files] (xml files will contain data about the pdf) Batch needs to read the pdf files and its associated xml file and push these to rest service/db.

When I looked at examples, most of it all covered how to read a line from the file and process it. here I have the items itself as file. I want to read one pdf file(as bytes) + xml file(converted into pojo) as set and push this to rest service one by one.

Right now, I am doing all the reading and processing inside a single tasklet. but I am sure there will be better solution to implement it. Please suggest and thank you.

I have a Spring batch step executing a tasklet that polls for files on a remote server:

I'm working with spring batch and have a job with two steps the first step (tasklet) validation the header CSV and the second step reads an CSV file and Write to another CSV file like this: