spring-batch | Spring Integration ActiveMQ Spring-Data H2 | Model View Controller library

I am using spring-batch in spring-boot application. The Spring Boot version is 2.3.3.RELEASE.

What I intend to achieve

I have to read a xml file containing thousands of Transactions with header tag (fileInformation). Do some business logic on transaction and then write the file back with the updated values in transaction. I am using StaxEventItemReader for reading the file and StaxEventItemWriter for writing to the file. Then i have couple of ItemProcessors for handling the business logic. Xml file looks like :

I am doing a study on the feasibility of a Spring Batch composed of two datasources. A SQL datasource for the Spring Batch metadata and a MongoDB datasource (with transactional use) for the business data. The transactional aspect raises several questions here.

The following topic: Spring batch with MongoDB and transactions and related resources provide a number of answers to my questions. The answer mentions the use of Spring's JtaTransactionManager to manage distributed transactions on the two datasources. This technique uses the 2PC protocol. It is also the most robust solution if I understood correctly. https://www.infoworld.com/article/2077963/distributed-transactions-in-spring--with-and-without-xa.html?page=2

On the other hand, I found some resources about Spring's ChainedTransactionManager. This technique uses the best effort 1PC protocol. This solution is less robust, if I understand correctly the system can be in an inconsistent state in case of a problem in the infrastructure (network failure for example). The ChainedTransactionManager has the advantage of being easier to implement and offers better performance. I saw that it is deprecated https://github.com/spring-projects/spring-data-commons/issues/2232.

What are the concrete risks of using the ChainedTransactionManager in a Spring Batch? In case of an error, can I have inconsistencies between the Spring batch metadata and the business data in Mongo? I imagine there are also considerations to take into account with retry or chunk skip strategies?

Thanks a lot for your help.

Basically I have a Spring Batch that queries a Database and implements Partitioner to get the Jobs, and assign the Jobs to a ThreadPoolTaskExecutors in a SlaveStep.

The Reader reads (Job) from the Database. The Writer loads the data into a csv file in an Azure Blob Storage.

The Job Partitioner and Reader works fine. The Writer writes to one file, then it closes, and the other jobs cannot finish because the stream is closed. I get the following error:

I am trying to implement a Spring batch job where in order to process a record , it require 2-3 db calls which is slowing down the processing of records(size is 1 million).If I go with chunk based processing it would process each record separately and would be slow in performance. So, I need to process 1000 records in one go as bulk processing which would reduce the db calls and performance would increase. But my question is If I implement Tasklet then I would lose the functionality of restartability and retrial/skip features too and if implemented using AggregateInputReader I am not sure what would be the impact on restartability and transaction handling. As per the below thread AggregateReader should work but not sure its impact on transaction handling and restartability in case of failure:

Spring batch: processing multiple record at once

Recently we run a test which created different Spring batch job instances concurrently (e.g 10 threads in parallel, job names are similar but different, e.g with the same prefix). And it's fairly easy to trigger deadlockerror reported from MySQL exception is

org.springframework.dao.DeadlockLoserDataAccessException: PreparedStatementCallback; SQL [INSERT into BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION) values (?, ?, ?, ?)]; Deadlock found when trying to get lock; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:267) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1443) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:633) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:862) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:917) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:922) at org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.createJobInstance(JdbcJobInstanceDao.java:120) at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:140) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

We searched for existing reports regarding to deadlock and find that some are specific to SQLServer like this: (https://github.com/spring-projects/spring-batch/issues/1448). After analysis of the isolation level used for creating jobs(SERIALIZABLE) and the operation sequence, we think the deadlock could be trigged as following:

1、before creating a job instance, the code will first query batch_job_instance table to check if the instance already exists(about 3 times), under SERIALIZABLE mode, this will hold shared next-key lock (https://dev.mysql.com/doc/refman/5.7/en/innodb-next-key-locking.html) in MySQL which lock records that are in scope related to the job name.

2、thread 2 want to create job2 and insert a row in batch_job_instance and thread 3 want to do the same thing, as both threads hold the same read next-key lock and the rows that need to be inserted are also in the key scope, the deadlock will happen.

Refer to the link here(https://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/core/repository/support/AbstractJobRepositoryFactoryBean.html#setIsolationLevelForCreate-java.lang.String-), we tried changing the isolation level to REPEATABLE_READ and this worked without any deadlock.

So the key question here is :

Is setting isolation level to REPEATABLE_READ the recommended solution here and is there any side effect of this solution as it's not set as default option ?

Thanks a lot!

we plan to use Spring Batch for mission critical production batching system. And I am wondering which version we should choose now?

From the version history in github(https://github.com/spring-projects/spring-batch/tags): version 4.2.7 is marked with .RELEASE, while 4.3.3 is not.

So the questions are:

1、what's the difference regarding the versions with or without .RELEASE ending, is the ".RELEASE" ending verison more stable?

2、which version is recommended for production now ?
Thanks a lot!

I am trying to write to an Azure Storage using Spring.

I am configuring the resource inside a Bean instead of Autowiring it from the Class.

In my Spring batch job, I'm trying to share data between steps using JobExecutionContext, which works only if i keep the steps single threaded as follows:

I need to upgrade my maven project to Java 11, so I am wondering:

1. What is the min version of Spring Framework and spring-batch version supported in Java 11?
2. What to be change in the pom?

As of now I know:

1. My project is not a spring-boot.
2. I know the Spring version should be at least 5.1.
3. ...