neo4j-spark-connector | Neo4j Connector for Apache Spark

I've run into a technical challenge around Neo4j usage that has had me stumped for a while. My organization uses Neo4j to model customer interaction patterns. The graph has grown to a size of around 2 million nodes and 7 million edges. All nodes and edges have between 5 and 10 metadata properties. Every day, we export data on all of our customers from Neo4j to a series of python processes that perform business logic.

Our original method of data export was to use paginated cypher queries to pull the data we needed. For each customer node, the cypher queries had to collect many types of surrounding nodes and edges so that the business logic could be performed with the necessary context. Unfortunately, as the size and density of the data grew, these paginated queries began to take too long to be practical.

Our current approach uses a custom Neo4j procedure to iterate over nodes, collect the necessary surrounding nodes and edges, serialize the data, and place it on a Kafka queue for downstream consumption. This method worked for some time but is now taking long enough so that it is also becoming impractical, especially considering that we expect the graph to grow an order of magnitude in size.

I have tried the cypher-for-apache-spark and neo4j-spark-connector projects, neither of which have been able to provide the query and data transfer speeds that we need.

We currently run on a single Neo4j instance with 32GB memory and 8 cores. Would a cluster help mitigate this issue?

Does anyone have any ideas or tips for how to perform this kind of data export? Any insight into the problem would be greatly appreciated!

I am using Neo4j/Cypher , my data is about 200GB , so i thought of scalable solution "spark".

Two solutions are available to make neo4j graphs with spark :

1) Cypher for Apache Spark (CAPS)

2) Neo4j-Spark-Connector

I used the first one ,CAPS . The pre-processed CSV got two "geohash" informations : one for pickup and another for drop off for each row what i want is to make a connected graph of geohash nodes.

CAPS allow only to make a graph by mapping nodes : If node with id 0 is to be connected to node with id 1 you need to have a relationship with start id 0 and end id 1.

A very simple layout would be:

I am trying to test on Intellij IDEA a scala maven project

when i run

mvn test

i get this error :

I have defined project dependencies and merge strategies for them to generate the jar file with the sbt-assembly plugin. Alos, the main class in the future jar was defined too. After I create the jar file and try to run it using the bash script I get an error:

I have multi-project with the main module called root, consumer and producer modules with dependencies which depend on the core module. The core modules hold configuration related classes. I would like to build 2 separate jars for consumer and producer with separate main classes with sbt-assembly. However, when I try to build them individually like this sbt consumer/assembly or altogether by running sbt assemblyI get such an error and sbt cannot compile the whole project:

I am fetching neo4j data into spark dataframe using neo4j-spark connector. I am able to fetch it successfully as I am able to show the dataframe. Then I register the dataframe with createOrReplaceTempView() method. Then I try running spark sql on it, but it gives exception saying

I was running scala code in spark-shell using this: