CSVImporter | Import CSV files line by line with ease | CSV Processing library

A JGraphT graph consists of vertex and edge objects. When importing a graph from a text file, the importer must somehow create vertex objects for every vertex it encounters in the text file. These objects must be of the same type you defined in the graph. To generate these objects, JGraphT uses vertex suppliers.

Various examples of how to use the CSV importer can be found in the corresponding test class CSVImporterTest.

There are two different ways to create a graph with a vertex supplier. Either you use the GraphTypeBuilder, or you use one of the graph constructors. Here's an example for a directed graph.

The new I/O package was redesigned from scratch and indeed it changes the semantics during graph creation. This is mainly the reason for switching package names from org.jgrapht.io to org.jgrapht.nio.

During the last years vertex/edge creation has been improved in the graphs using graph vertex and edge suppliers. The new I/O importers switch behavior and call Graph#addVertex() whenever a new vertex is required, which in turn uses the provided graph vertex supplier to create the vertex.

Unfortunately this leads to your observed behavior. The actual vertex identifiers from the input file are still accessible as they are considered vertex attributes and are reported during import using a key of "ID".

See also Import graph with 1.4.0 which is a very similar case and uses this functionality to create a second graph with the exact same identifiers.

On the other hand, the old behavior makes sense. It is natural to expect that the import retains your vertex identifiers (at least for most importers). There is already a fix for this by providing a method #setVertexFactory(Function). This method allows the user to bypass vertex creation by providing a custom vertex factory method. The factory method is responsible to create a new graph vertex given the vertex identifier read from the input file.

The fix will be available in the next release (probably 1.4.1) and is already available in the snapshot build (1.4.1-SNAPSHOT). See https://github.com/jgrapht/jgrapht#using-via-maven on how to use the snapshot build.

In order to retain the old behavior you should build your importer like:

So I have tried to reproduce what I think you need into a simple example

Let's say we have this command

You might be able to solve the problem by loading the CSV data into a DataTable, adding a FileId column with a default value to the table, and passing the DataTable into the SqlBulkCopy. It doesn't look like your current solution loads the whole file into memory, so you should monitor memory usage if you try this approach. You might be able to get your current solution to work if you dig through the documentation of the Columns property of the CsvReader. It looks like it does not behave the way you are trying to use it.

Here is an example of you you might load the file using a DataTable:

Fixed the issue by just using CSwiftV instead of CSVImporter. https://github.com/Daniel1of1/CSwiftV

Neo4j's batch import logic tries to break up large input data into smaller chunks so that multiple parsers can process them simultaneously. The logic looks for newlines to determine where to separate the chunks. If your data has very infrequent or no newlines, then chunking fails with the error you are seeing.

You may need to reformat your input data so that it contains more frequent newlines.

On your computer, quite outside of your app, write a script or app that recodes your big data as a SQLite database. Include that database in your app. Now when the app runs the data is available instantly via SQLIte with no time spent converting and no strain on memory (because data fetches are performed from disk as needed).

According to my understanding, since you are using then in the promise chain, it is also meant to return a promise and hence you are getting this error. If you intend not to return promise from your next step, you can directly use done after firstly.

Use below chain if you want to return Promise from then

The @EnableBatchProcessing automatically imports the StepScope, so you don't need to declare it as a bean in your application context. The issue you are linking to happens when there is a mix between XML and Java Config. In your case here, I see only Java Config so the issue should not happen.

I am trying to scope the multiResourceItemReader below

All I know is that stepScope() has to be in a @Configuration file, other than that, I'm thoroughly confused as to what needs to be done.

Just declaring the step scope is not enough, you need to add the @StepScope annotation on the bean definition.

You can find more details about the StepScope in the reference documentation here: https://docs.spring.io/spring-batch/4.0.x/reference/html/step.html#step-scope