gs-batch-processing | Batch Service : : Learn how to create a basic batch | BPM library

 by   spring-guides Java Version: 2.1.6.RELEASE License: Apache-2.0

kandi X-RAY | gs-batch-processing Summary

gs-batch-processing is a Java library typically used in Automation, BPM, Docker applications. gs-batch-processing has no vulnerabilities, it has a Permissive License and it has low support. However gs-batch-processing has 1 bugs and it build file is not available. You can download it from GitHub.
This guide walks you through the process of creating a basic batch-driven solution.
    Support
      Quality
        Security
          License
            Reuse
            Support
              Quality
                Security
                  License
                    Reuse

                      kandi-support Support

                        summary
                        gs-batch-processing has a low active ecosystem.
                        summary
                        It has 189 star(s) with 373 fork(s). There are 35 watchers for this library.
                        summary
                        It had no major release in the last 6 months.
                        summary
                        There are 1 open issues and 15 have been closed. On average issues are closed in 263 days. There are 4 open pull requests and 0 closed requests.
                        summary
                        It has a neutral sentiment in the developer community.
                        summary
                        The latest version of gs-batch-processing is 2.1.6.RELEASE
                        gs-batch-processing Support
                          Best in #BPM
                            Average in #BPM
                            gs-batch-processing Support
                              Best in #BPM
                                Average in #BPM

                                  kandi-Quality Quality

                                    summary
                                    gs-batch-processing has 1 bugs (0 blocker, 0 critical, 0 major, 1 minor) and 2 code smells.
                                    gs-batch-processing Quality
                                      Best in #BPM
                                        Average in #BPM
                                        gs-batch-processing Quality
                                          Best in #BPM
                                            Average in #BPM

                                              kandi-Security Security

                                                summary
                                                gs-batch-processing has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
                                                summary
                                                gs-batch-processing code analysis shows 0 unresolved vulnerabilities.
                                                summary
                                                There are 0 security hotspots that need review.
                                                gs-batch-processing Security
                                                  Best in #BPM
                                                    Average in #BPM
                                                    gs-batch-processing Security
                                                      Best in #BPM
                                                        Average in #BPM

                                                          kandi-License License

                                                            summary
                                                            gs-batch-processing is licensed under the Apache-2.0 License. This license is Permissive.
                                                            summary
                                                            Permissive licenses have the least restrictions, and you can use them in most projects.
                                                            gs-batch-processing License
                                                              Best in #BPM
                                                                Average in #BPM
                                                                gs-batch-processing License
                                                                  Best in #BPM
                                                                    Average in #BPM

                                                                      kandi-Reuse Reuse

                                                                        summary
                                                                        gs-batch-processing releases are not available. You will need to build from source code and install.
                                                                        summary
                                                                        gs-batch-processing has no build file. You will be need to create the build yourself to build the component from source.
                                                                        summary
                                                                        Installation instructions are not available. Examples and code snippets are available.
                                                                        summary
                                                                        It has 257 lines of code, 17 functions and 8 files.
                                                                        summary
                                                                        It has low code complexity. Code complexity directly impacts maintainability of the code.
                                                                        gs-batch-processing Reuse
                                                                          Best in #BPM
                                                                            Average in #BPM
                                                                            gs-batch-processing Reuse
                                                                              Best in #BPM
                                                                                Average in #BPM
                                                                                  Top functions reviewed by kandi - BETA
                                                                                  kandi has reviewed gs-batch-processing and discovered the below as its top functions. This is intended to give you an instant insight into gs-batch-processing implemented functionality, and help decide if they suit your requirements.
                                                                                  • bean data reader .
                                                                                    • Verify that a BatchJob completes .
                                                                                      • Converts the person into a person object .
                                                                                        • Import user job .
                                                                                          • String representation .
                                                                                            • Entry point for the BatchProcessing application .
                                                                                              • Set the first name .
                                                                                                Get all kandi verified functions for this library.
                                                                                                Get all kandi verified functions for this library.

                                                                                                gs-batch-processing Key Features

                                                                                                What You Will build
                                                                                                What You Need
                                                                                                Business Data
                                                                                                Starting with Spring Initializr Manual Initialization (optional)
                                                                                                Create a Business Class
                                                                                                Create an Intermediate Processor
                                                                                                Put Together a Batch Job
                                                                                                Make the Application Executable
                                                                                                Summary
                                                                                                See also

                                                                                                gs-batch-processing Examples and Code Snippets

                                                                                                No Code Snippets are available at this moment for gs-batch-processing.
                                                                                                Community Discussions

                                                                                                Trending Discussions on gs-batch-processing

                                                                                                Method org.postgresql.jdbc.PgConnection.createClob() is not yet implemented if hibernate dependency is absent
                                                                                                chevron right
                                                                                                FlatFileItemReader tab delimiter not working
                                                                                                chevron right
                                                                                                FlatFileItemWriter cannot create new file or update existing file
                                                                                                chevron right
                                                                                                Unable to avoid job triggering at Start up
                                                                                                chevron right

                                                                                                QUESTION

                                                                                                Method org.postgresql.jdbc.PgConnection.createClob() is not yet implemented if hibernate dependency is absent
                                                                                                Asked 2019-Aug-07 at 16:15

                                                                                                I've created small hello world project for spring batch:

                                                                                                build.gradle:

                                                                                                buildscript {
                                                                                                    repositories {
                                                                                                        mavenCentral()
                                                                                                    }
                                                                                                    dependencies {
                                                                                                        classpath("org.springframework.boot:spring-boot-gradle-plugin:2.1.6.RELEASE")
                                                                                                    }
                                                                                                }
                                                                                                
                                                                                                apply plugin: 'java'
                                                                                                apply plugin: 'eclipse'
                                                                                                apply plugin: 'idea'
                                                                                                apply plugin: 'org.springframework.boot'
                                                                                                apply plugin: 'io.spring.dependency-management'
                                                                                                
                                                                                                bootJar {
                                                                                                    baseName = 'gs-batch-processing'
                                                                                                    version = '0.1.0'
                                                                                                }
                                                                                                
                                                                                                repositories {
                                                                                                    mavenCentral()
                                                                                                }
                                                                                                
                                                                                                sourceCompatibility = 1.8
                                                                                                targetCompatibility = 1.8
                                                                                                
                                                                                                dependencies {
                                                                                                    compile("org.springframework.boot:spring-boot-starter-batch")
                                                                                                    compile("org.postgresql:postgresql")
                                                                                                    compile("org.springframework.boot:spring-boot-starter-data-jpa")
                                                                                                    //to fix exception on startup
                                                                                                    //compile('org.hibernate:hibernate-core:5.4.2.Final')
                                                                                                
                                                                                                    testCompile("junit:junit")
                                                                                                }
                                                                                                

                                                                                                configuration:

                                                                                                @Configuration
                                                                                                @EnableBatchProcessing
                                                                                                public class BatchConfiguration {
                                                                                                    @Autowired
                                                                                                    public JobBuilderFactory jobBuilderFactory;
                                                                                                
                                                                                                    @Autowired
                                                                                                    public StepBuilderFactory stepBuilderFactory;
                                                                                                
                                                                                                    @Autowired
                                                                                                    private DbPersonWriter dbPersonWriter;
                                                                                                
                                                                                                    @Autowired
                                                                                                    private ToLowerCasePersonProcessor toLowerCasePersonProcessor;
                                                                                                
                                                                                                    @Value("${app.users-location}")
                                                                                                    Resource csvResource;
                                                                                                
                                                                                                    @Bean
                                                                                                    public Job job() {
                                                                                                        return jobBuilderFactory.get("myJob")
                                                                                                                .incrementer(new RunIdIncrementer())
                                                                                                                .flow(csvToDataBaseStep())
                                                                                                                .end()
                                                                                                                .build();
                                                                                                    }
                                                                                                
                                                                                                    private Step csvToDataBaseStep() {
                                                                                                        return stepBuilderFactory.get("csvToDatabaseStep")
                                                                                                                .chunk(100)
                                                                                                                .reader(csvPersonReader())
                                                                                                                .processor(toLowerCasePersonProcessor)
                                                                                                                .writer(dbPersonWriter)
                                                                                                                .build();
                                                                                                
                                                                                                    }
                                                                                                

                                                                                                on startup I see following exception:

                                                                                                java.lang.reflect.InvocationTargetException: null
                                                                                                    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]
                                                                                                    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111]
                                                                                                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
                                                                                                    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
                                                                                                    at org.hibernate.engine.jdbc.env.internal.LobCreatorBuilderImpl.useContextualLobCreation(LobCreatorBuilderImpl.java:113) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.engine.jdbc.env.internal.LobCreatorBuilderImpl.makeLobCreatorBuilder(LobCreatorBuilderImpl.java:54) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentImpl.(JdbcEnvironmentImpl.java:271) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:114) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:94) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.id.factory.internal.DefaultIdentifierGeneratorFactory.injectServices(DefaultIdentifierGeneratorFactory.java:152) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.service.internal.AbstractServiceRegistryImpl.injectDependencies(AbstractServiceRegistryImpl.java:286) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:243) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.(InFlightMetadataCollectorImpl.java:179) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:119) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:904) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:935) [hibernate-core-5.3.10.Final.jar:5.3.10.Final]
                                                                                                    at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:57) [spring-orm-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365) [spring-orm-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:390) [spring-orm-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:377) [spring-orm-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341) [spring-orm-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1837) [spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1774) [spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) [spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) [spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) [spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) [spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1105) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                                                                                                    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:742) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
                                                                                                    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:389) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
                                                                                                    at org.springframework.boot.SpringApplication.run(SpringApplication.java:311) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
                                                                                                    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1213) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
                                                                                                    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1202) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
                                                                                                    at spring.boot.hello.world.MyApplication.main(MyApplication.java:9) ~[main/:na]
                                                                                                Caused by: java.sql.SQLFeatureNotSupportedException: Method org.postgresql.jdbc.PgConnection.createClob() is not yet implemented.
                                                                                                    at org.postgresql.Driver.notImplemented(Driver.java:688) ~[postgresql-42.2.5.jar:42.2.5]
                                                                                                    at org.postgresql.jdbc.PgConnection.createClob(PgConnection.java:1269) ~[postgresql-42.2.5.jar:42.2.5]
                                                                                                    ... 43 common frames omitted
                                                                                                

                                                                                                I have to say that everything is working as expected and this trace just litter my logs. But I want to avoid it.

                                                                                                I've found workaround to fix it( thx this topic How to resolve org.postgresql.jdbc.PgConnection.createClob() is not yet implemented):

                                                                                                just added dependency and excption disappeared:

                                                                                                compile('org.hibernate:hibernate-core:5.4.2.Final')
                                                                                                

                                                                                                Is there more elegant way? I don't think that workaround is a good solution

                                                                                                P.S.

                                                                                                To play with source code you can follow my github repo

                                                                                                ANSWER

                                                                                                Answered 2019-Aug-07 at 16:15

                                                                                                This was a Hibernate issue. If you are using Spring Boot latest version from 2.0.x to 2.1.x includes Hibernate 5.3.10.final you can take a look here but this issue was fixed on Hibernate version 5.4.0.CR1 then you need to add that dependency or if it is possible the latest version:

                                                                                                For Gradle:

                                                                                                compile('org.hibernate:hibernate-core:5.4.2.Final')
                                                                                                

                                                                                                For Maven:

                                                                                                
                                                                                                    org.hibernate
                                                                                                    hibernate-core
                                                                                                    5.4.2.Final
                                                                                                
                                                                                                

                                                                                                UPDATED "SPRING BOOT 2.2.0.M(1-4)"

                                                                                                In addition Spring boot v2.2.0.Mx includes now Hibernate v5.4.x then this issue was fixed for these versions.

                                                                                                Source https://stackoverflow.com/questions/57361157

                                                                                                QUESTION

                                                                                                FlatFileItemReader tab delimiter not working
                                                                                                Asked 2019-Feb-13 at 10:20

                                                                                                I checkout this project from spring: https://github.com/spring-guides/gs-batch-processing

                                                                                                Source: https://spring.io/guides/gs/batch-processing/

                                                                                                I replace the ',' with 'tab' in 'sample-data.csv:

                                                                                                Jill    Doe
                                                                                                Joe Doe
                                                                                                Justin  Doe
                                                                                                Jane    Doe
                                                                                                John    Doe
                                                                                                

                                                                                                Then I add the new Delimiter to the reader:

                                                                                                @Bean
                                                                                                public FlatFileItemReader reader() {
                                                                                                    return new FlatFileItemReaderBuilder()
                                                                                                        .name("personItemReader")
                                                                                                        .resource(new ClassPathResource("sample-data.csv"))
                                                                                                        .delimited()
                                                                                                        .delimiter(DelimitedLineTokenizer.DELIMITER_TAB) // NEW DELIMITER
                                                                                                        .names(new String[]{"firstName", "lastName"})
                                                                                                        .fieldSetMapper(new BeanWrapperFieldSetMapper() {{
                                                                                                            setTargetType(Person.class);
                                                                                                        }})
                                                                                                        .build();
                                                                                                }
                                                                                                

                                                                                                When I launch I get this error:

                                                                                                Caused by: org.springframework.batch.item.file.transform.IncorrectTokenCountException: Incorrect number of tokens found in record: expected 2 actual 1
                                                                                                    at org.springframework.batch.item.file.transform.AbstractLineTokenizer.tokenize(AbstractLineTokenizer.java:142) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
                                                                                                    at org.springframework.batch.item.file.mapping.DefaultLineMapper.mapLine(DefaultLineMapper.java:43) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
                                                                                                    at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:180) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
                                                                                                    ... 50 common frames omitted
                                                                                                

                                                                                                I have tried with '@' delimiter -> it works. For some reason, I can't make it work with the tab delimiter...

                                                                                                Of course in my real project, I have an input file with 'tab' separators...

                                                                                                Any solution here?

                                                                                                ANSWER

                                                                                                Answered 2018-Aug-08 at 07:55

                                                                                                You cant't set the tab delimiter that way. Since tab ('\t') doesnt contain any actual text it is ignored by the DelimitedLineTokenizer in static DelimitedBuilder class in FlatFileItemReaderBuilder.java . Any non-whitespace delimiter can be set using above code that you have given in the question.

                                                                                                FlatFileItemReaderBuilder sourceCode

                                                                                                This is how the LineTokenizer instance is built in FlatFileItemReaderBuilder.java.

                                                                                                public DelimitedLineTokenizer build() {
                                                                                                        Assert.notNull(this.fieldSetFactory, "A FieldSetFactory is required.");
                                                                                                        Assert.notEmpty(this.names, "A list of field names is required");
                                                                                                
                                                                                                        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
                                                                                                
                                                                                                        tokenizer.setNames(this.names.toArray(new String[this.names.size()]));
                                                                                                
                                                                                                        // the hasText ignores the tab delimiter.
                                                                                                
                                                                                                        if(StringUtils.hasText(this.delimiter)) {
                                                                                                            tokenizer.setDelimiter(this.delimiter);
                                                                                                        }
                                                                                                // more code
                                                                                                

                                                                                                So to fix this issue, you need to provide bean of Type DelimitedLineTokenizer explicitly configured with tab delimiter.

                                                                                                use below code in your spring configuration file to set the tab delimiter:

                                                                                                @Bean
                                                                                                public FlatFileItemReader reader() {
                                                                                                    return new FlatFileItemReaderBuilder().name("personItemReader")
                                                                                                            .resource(new ClassPathResource("sample-data.csv"))
                                                                                                            .lineMapper(lineMapper()).build();
                                                                                                }
                                                                                                
                                                                                                @Bean
                                                                                                public DefaultLineMapper lineMapper(){
                                                                                                      DefaultLineMapper lineMapper = new DefaultLineMapper<>();
                                                                                                      lineMapper.setLineTokenizer(lineTokenizer());
                                                                                                      lineMapper.setFieldSetMapper(new BeanWrapperFieldSetMapper() {
                                                                                                                {
                                                                                                                    setTargetType(Person.class);
                                                                                                                }
                                                                                                            });
                                                                                                      return lineMapper;
                                                                                                }
                                                                                                
                                                                                                @Bean
                                                                                                public DelimitedLineTokenizer lineTokenizer() {
                                                                                                    DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer(DelimitedLineTokenizer.DELIMITER_TAB);
                                                                                                    tokenizer.setNames(new String[] { "firstName", "lastName" });
                                                                                                    return tokenizer;
                                                                                                }
                                                                                                

                                                                                                Source https://stackoverflow.com/questions/51738718

                                                                                                QUESTION

                                                                                                FlatFileItemWriter cannot create new file or update existing file
                                                                                                Asked 2018-Jun-25 at 19:55

                                                                                                [New to SpringBatch] Using Spring Boot, I am trying to create a job which reads names from MongoDB, converts to lowercase, and outputs to CSV file. My reader and processor are working but the writer isn't.

                                                                                                My code is as follows.

                                                                                                Configuration file:

                                                                                                package bbye;
                                                                                                
                                                                                                import java.util.HashMap;
                                                                                                import java.util.Map;
                                                                                                
                                                                                                import javax.sql.DataSource;
                                                                                                
                                                                                                import org.springframework.batch.core.Job;
                                                                                                import org.springframework.batch.core.Step;
                                                                                                import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
                                                                                                import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
                                                                                                import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
                                                                                                import org.springframework.batch.core.launch.support.RunIdIncrementer;
                                                                                                import org.springframework.batch.item.data.MongoItemReader;
                                                                                                import org.springframework.batch.item.data.builder.MongoItemReaderBuilder;
                                                                                                import org.springframework.batch.item.file.FlatFileItemWriter;
                                                                                                import org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor;
                                                                                                import org.springframework.batch.item.file.transform.DelimitedLineAggregator;
                                                                                                import org.springframework.batch.item.file.transform.FieldExtractor;
                                                                                                import org.springframework.batch.item.file.transform.LineAggregator;
                                                                                                import org.springframework.beans.factory.annotation.Autowired;
                                                                                                import org.springframework.context.annotation.Bean;
                                                                                                import org.springframework.context.annotation.Configuration;
                                                                                                import org.springframework.core.io.ClassPathResource;
                                                                                                import org.springframework.data.domain.Sort;
                                                                                                import org.springframework.data.mongodb.core.MongoTemplate;
                                                                                                
                                                                                                import hello.Person;
                                                                                                
                                                                                                @Configuration
                                                                                                @EnableBatchProcessing
                                                                                                public class BatchConfigProcessing {
                                                                                                
                                                                                                
                                                                                                    @Autowired
                                                                                                    public JobBuilderFactory jobBuilderFactory;
                                                                                                
                                                                                                    @Autowired
                                                                                                    public StepBuilderFactory stepBuilderFactory;
                                                                                                
                                                                                                    @Autowired
                                                                                                    private MongoTemplate mongoTemplate;
                                                                                                
                                                                                                    private String readQuery = "{}";
                                                                                                
                                                                                                    // tag::readerwriterprocessor[]
                                                                                                    @Bean
                                                                                                    public MongoItemReader readMongo(DataSource dataSource) {
                                                                                                        return new MongoItemReaderBuilder()
                                                                                                                .name("mongoDocReader")
                                                                                                                .jsonQuery(readQuery)
                                                                                                                .targetType(Person.class)
                                                                                                                .sorts(sort())
                                                                                                                .template(mongoTemplate)
                                                                                                                .collection("people")
                                                                                                                .build();
                                                                                                    }
                                                                                                
                                                                                                    @Bean
                                                                                                    public PersonDocProcessor processor() {
                                                                                                        return new PersonDocProcessor();
                                                                                                    }
                                                                                                
                                                                                                
                                                                                                    @Bean
                                                                                                    public FlatFileItemWriter writer() {
                                                                                                        /*FlatFileItemWriterBuilder writePerson =  new FlatFileItemWriterBuilder();
                                                                                                            writePerson.name("personDocWriter");
                                                                                                            writePerson.resource(new ClassPathResource("PersonExtracted.csv"));
                                                                                                            writePerson.lineAggregator(new DelimitedLineAggregator());
                                                                                                            writePerson.shouldDeleteIfExists(true);
                                                                                                            writePerson.build();*/
                                                                                                
                                                                                                     FlatFileItemWriter fileWriter = new FlatFileItemWriter<>();
                                                                                                     fileWriter.setName("csvWriter");
                                                                                                     fileWriter.setResource(new ClassPathResource("PersonExtracted.csv"));
                                                                                                     fileWriter.setLineAggregator(lineAggregator());
                                                                                                     fileWriter.setForceSync(true);
                                                                                                     fileWriter.close();   
                                                                                                     return fileWriter; 
                                                                                                    }
                                                                                                    // end::readerwriterprocessor[]
                                                                                                
                                                                                                    // tag::jobstep[]
                                                                                                
                                                                                                    @Bean
                                                                                                    public Job exportUserJob(FileUploadNotificationListener listener, Step step1) {
                                                                                                        return jobBuilderFactory.get("exportUserJob")
                                                                                                            .incrementer(new RunIdIncrementer())
                                                                                                            .listener(listener)
                                                                                                            .flow(step1)
                                                                                                            .end()
                                                                                                            .build();
                                                                                                    }
                                                                                                
                                                                                                    @Bean
                                                                                                    public Step step2(MongoItemReader reader) {
                                                                                                        return stepBuilderFactory.get("step2")
                                                                                                            . chunk(10)
                                                                                                            .reader(reader)
                                                                                                            .processor(processor())
                                                                                                            .writer(writer())
                                                                                                            .build();
                                                                                                    }
                                                                                                
                                                                                                    // end::jobstep[]
                                                                                                
                                                                                                    public FieldExtractor fieldExtractor()
                                                                                                    {
                                                                                                        BeanWrapperFieldExtractor extractor = new BeanWrapperFieldExtractor<>();
                                                                                                        extractor.setNames( new String[] { "firstName",
                                                                                                                                            "lastName"});
                                                                                                        return extractor;
                                                                                                
                                                                                                    }
                                                                                                
                                                                                                
                                                                                                    public LineAggregator lineAggregator() {
                                                                                                        DelimitedLineAggregator la = new DelimitedLineAggregator();
                                                                                                        la.setDelimiter(",");
                                                                                                        la.setFieldExtractor(fieldExtractor());
                                                                                                        return la;
                                                                                                    }
                                                                                                
                                                                                                    public Map sort(){
                                                                                                        String firstName = "firstName";
                                                                                                        Map sortMap = new HashMap();
                                                                                                        sortMap.put(firstName, Sort.DEFAULT_DIRECTION);
                                                                                                        return sortMap;
                                                                                                    }
                                                                                                
                                                                                                
                                                                                                }
                                                                                                

                                                                                                Processor file

                                                                                                package bbye;
                                                                                                
                                                                                                import org.slf4j.Logger;
                                                                                                import org.slf4j.LoggerFactory;
                                                                                                import org.springframework.batch.item.ItemProcessor;
                                                                                                import org.springframework.stereotype.Component;
                                                                                                
                                                                                                import hello.Person;
                                                                                                
                                                                                                @Component
                                                                                                public class PersonDocProcessor implements ItemProcessor {
                                                                                                
                                                                                                    private static final Logger log = LoggerFactory.getLogger(PersonDocProcessor.class);
                                                                                                
                                                                                                    @Override
                                                                                                    public Person process(final Person person) throws Exception {
                                                                                                        final String firstName = person.getFirstName().toLowerCase();
                                                                                                        final String lastName = person.getLastName().toLowerCase();
                                                                                                
                                                                                                        final Person transformedPerson = new Person(firstName, lastName);
                                                                                                
                                                                                                        log.info("Converting (" + person + ") into (" + transformedPerson + ")");
                                                                                                
                                                                                                        return transformedPerson;
                                                                                                    }
                                                                                                }
                                                                                                

                                                                                                Listener

                                                                                                package bbye;
                                                                                                
                                                                                                import org.springframework.batch.core.JobExecution;
                                                                                                import org.springframework.batch.core.JobExecutionListener;
                                                                                                import org.springframework.stereotype.Component;
                                                                                                
                                                                                                @Component
                                                                                                public class FileUploadNotificationListener implements JobExecutionListener {
                                                                                                
                                                                                                    @Override
                                                                                                    public void beforeJob(JobExecution jobExecution) {
                                                                                                        System.out.println("===== listening for job - mongoReader - fileWriter ====");
                                                                                                
                                                                                                    }
                                                                                                
                                                                                                    @Override
                                                                                                    public void afterJob(JobExecution jobExecution) {
                                                                                                        System.out.println("==== file write job completed =====");
                                                                                                
                                                                                                    }
                                                                                                }
                                                                                                

                                                                                                Here Person is a simple POJO. The stack trace with and without manual file creation is as follows:

                                                                                                1. If the file is not present under src/main/resources FlatFileItemWriter does not create a new file and throws
                                                                                                org.springframework.batch.item.ItemStreamException: Could not convert resource to file: [class path resource [PersonExtracted.csv]]
                                                                                                    at org.springframework.batch.item.file.FlatFileItemWriter.getOutputState(FlatFileItemWriter.java:399) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
                                                                                                    at org.springframework.batch.item.file.FlatFileItemWriter.open(FlatFileItemWriter.java:337) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
                                                                                                ........
                                                                                                Caused by: java.io.FileNotFoundException: class path resource [PersonExtracted.csv] cannot be resolved to URL because it does not exist
                                                                                                
                                                                                                1. If I create the PersonExtracted.csv file manually, the program runs without errors but does not write anything to the file. In fact, a blank file is returned. The stack trace is as below.
                                                                                                :: Spring Boot ::        (v2.0.2.RELEASE)
                                                                                                
                                                                                                2018-06-19 11:35:17.663  INFO 25136 --- [           main] hello.Application                        : Starting Application on MyPC with PID 25136 (C:\eclipse-workspace\gs-batch-processing-master\complete\target\classes started by shristi in C:\eclipse-workspace\gs-batch-processing-master\complete)
                                                                                                2018-06-19 11:35:17.666  INFO 25136 --- [           main] hello.Application                        : No active profile set, falling back to default profiles: default
                                                                                                2018-06-19 11:35:17.689  INFO 25136 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@15bb6bea: startup date [Tue Jun 19 11:35:17 EDT 2018]; root of context hierarchy
                                                                                                2018-06-19 11:35:18.135  INFO 25136 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
                                                                                                2018-06-19 11:35:18.136  WARN 25136 --- [           main] com.zaxxer.hikari.util.DriverDataSource  : Registered driver with driverClassName=org.hsqldb.jdbcDriver was not found, trying direct instantiation.
                                                                                                2018-06-19 11:35:18.282  INFO 25136 --- [           main] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)
                                                                                                2018-06-19 11:35:18.284  INFO 25136 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
                                                                                                2018-06-19 11:35:18.293  INFO 25136 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from URL [file:/C:/eclipse-workspace/gs-batch-processing-master/complete/target/classes/schema-all.sql]
                                                                                                2018-06-19 11:35:18.297  INFO 25136 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from URL [file:/C:/eclipse-workspace/gs-batch-processing-master/complete/target/classes/schema-all.sql] in 4 ms.
                                                                                                2018-06-19 11:35:18.518  INFO 25136 --- [           main] org.mongodb.driver.cluster               : Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
                                                                                                2018-06-19 11:35:18.552  INFO 25136 --- [localhost:27017] org.mongodb.driver.connection            : Opened connection [connectionId{localValue:1, serverValue:140}] to localhost:27017
                                                                                                2018-06-19 11:35:18.554  INFO 25136 --- [localhost:27017] org.mongodb.driver.cluster               : Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 0]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1438717}
                                                                                                2018-06-19 11:35:18.723  INFO 25136 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: HSQL
                                                                                                2018-06-19 11:35:18.770  INFO 25136 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
                                                                                                2018-06-19 11:35:18.778  INFO 25136 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
                                                                                                2018-06-19 11:35:18.781  INFO 25136 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 3 ms.
                                                                                                2018-06-19 11:35:18.870  INFO 25136 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
                                                                                                2018-06-19 11:35:18.871  INFO 25136 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'dataSource' has been autodetected for JMX exposure
                                                                                                2018-06-19 11:35:18.873  INFO 25136 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
                                                                                                2018-06-19 11:35:18.880  INFO 25136 --- [           main] hello.Application                        : Started Application in 1.357 seconds (JVM running for 2.284)
                                                                                                2018-06-19 11:35:18.881  INFO 25136 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: []
                                                                                                2018-06-19 11:35:18.908  INFO 25136 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=exportUserJob]] launched with the following parameters: [{run.id=1}]
                                                                                                ===== listening for job - mongoReader - fileWriter ====
                                                                                                2018-06-19 11:35:18.917  INFO 25136 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [step2]
                                                                                                2018-06-19 11:35:18.995  INFO 25136 --- [           main] org.mongodb.driver.connection            : Opened connection [connectionId{localValue:2, serverValue:141}] to localhost:27017
                                                                                                2018-06-19 11:35:19.022  INFO 25136 --- [           main] bbye.PersonDocProcessor                  : Converting (firstName: ALICE, lastName: WONDERLAND) into (firstName: alice, lastName: wonderland)
                                                                                                2018-06-19 11:35:19.022  INFO 25136 --- [           main] bbye.PersonDocProcessor                  : Converting (firstName: FIRSTNAME, lastName: LASTNAME) into (firstName: firstname, lastName: lastname)
                                                                                                2018-06-19 11:35:19.022  INFO 25136 --- [           main] bbye.PersonDocProcessor                  : Converting (firstName: JANE, lastName: DOE) into (firstName: jane, lastName: doe)
                                                                                                2018-06-19 11:35:19.022  INFO 25136 --- [           main] bbye.PersonDocProcessor                  : Converting (firstName: JOHN, lastName: DOE) into (firstName: john, lastName: doe)
                                                                                                2018-06-19 11:35:19.022  INFO 25136 --- [           main] bbye.PersonDocProcessor                  : Converting (firstName: MARK, lastName: WINN) into (firstName: mark, lastName: winn)
                                                                                                ==== file write job completed =====
                                                                                                2018-06-19 11:35:19.031  INFO 25136 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=exportUserJob]] completed with the following parameters: [{run.id=1}] and the following status: [COMPLETED]
                                                                                                
                                                                                                2018-06-19 11:35:19.032  INFO 25136 --- [       Thread-2] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@15bb6bea: startup date [Tue Jun 19 11:35:17 EDT 2018]; root of context hierarchy
                                                                                                2018-06-19 11:35:19.033  INFO 25136 --- [       Thread-2] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
                                                                                                2018-06-19 11:35:19.034  INFO 25136 --- [       Thread-2] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans
                                                                                                2018-06-19 11:35:19.035  INFO 25136 --- [       Thread-2] org.mongodb.driver.connection            : Closed connection [connectionId{localValue:2, serverValue:141}] to localhost:27017 because the pool has been closed.
                                                                                                2018-06-19 11:35:19.036  INFO 25136 --- [       Thread-2] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
                                                                                                2018-06-19 11:35:19.037  INFO 25136 --- [       Thread-2] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.
                                                                                                

                                                                                                ANSWER

                                                                                                Answered 2018-Jun-25 at 15:04

                                                                                                I think we should use FileSystemResource instead of ClassPathResource. Could you please try and let us know.

                                                                                                Source https://stackoverflow.com/questions/50948796

                                                                                                QUESTION

                                                                                                Unable to avoid job triggering at Start up
                                                                                                Asked 2018-Feb-08 at 11:42

                                                                                                I am using a batch process and i want it to run at a specific cron scheduled time. However, the job is getting triggered at the start up and again triggering at the scheduled time. I am trying to avoid the former however failing to do so. This is a sample repository which reproduces the same issue: https://github.com/ppanigrahi02/BatchJobWithScheduler. i am using the spring guide example https://github.com/spring-guides/gs-batch-processing and added a scheduler on top of it. I will really appreciate any leads.

                                                                                                ANSWER

                                                                                                Answered 2018-Feb-08 at 11:42

                                                                                                The @Scheduled annotation used here will start the job immediately at startup and every 600000 milliseconds. If you want to specify an initial delay to wait before the first execution, you can use the initialDelay attribute of the annotation, something like:

                                                                                                @Scheduled(initialDelay = 600000, fixedRate = 600000)

                                                                                                Another option is to use the cron attribute and provide a cron expression. For more details about this annotation, you can check the reference documentation here: https://docs.spring.io/spring/docs/5.0.3.RELEASE/spring-framework-reference/integration.html#scheduling-annotation-support-scheduled

                                                                                                Source https://stackoverflow.com/questions/48683209

                                                                                                Community Discussions, Code Snippets contain sources that include Stack Exchange Network

                                                                                                Vulnerabilities

                                                                                                No vulnerabilities reported

                                                                                                Install gs-batch-processing

                                                                                                You can download it from GitHub.
                                                                                                You can use gs-batch-processing like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the gs-batch-processing component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

                                                                                                Support

                                                                                                For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .
                                                                                                Find more information at:
                                                                                                Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
                                                                                                Find more libraries
                                                                                                Explore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits​
                                                                                                Save this library and start creating your kit
                                                                                                CLONE
                                                                                              • HTTPS

                                                                                                https://github.com/spring-guides/gs-batch-processing.git

                                                                                              • CLI

                                                                                                gh repo clone spring-guides/gs-batch-processing

                                                                                              • sshUrl

                                                                                                git@github.com:spring-guides/gs-batch-processing.git

                                                                                              • Share this Page

                                                                                                share link

                                                                                                Explore Related Topics

                                                                                                Consider Popular BPM Libraries

                                                                                                Try Top Libraries by spring-guides

                                                                                                tut-spring-security-and-angular-js

                                                                                                by spring-guidesTypeScript

                                                                                                gs-rest-service

                                                                                                by spring-guidesJava

                                                                                                tut-react-and-spring-data-rest

                                                                                                by spring-guidesJavaScript

                                                                                                gs-spring-boot

                                                                                                by spring-guidesJava

                                                                                                tut-spring-boot-oauth2

                                                                                                by spring-guidesJava

                                                                                                Compare BPM Libraries with Highest Support

                                                                                                airflow

                                                                                                by apache

                                                                                                dagster

                                                                                                by dagster-io

                                                                                                azkaban

                                                                                                by azkaban

                                                                                                conductor

                                                                                                by Netflix

                                                                                                prefect

                                                                                                by PrefectHQ

                                                                                                Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
                                                                                                Find more libraries
                                                                                                Explore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits​
                                                                                                Save this library and start creating your kit