graphx | Former GraphX development repository

According to the documentation available at http://ampcamp.berkeley.edu/big-data-mini-course/graph-analytics-with-graphx.html:

"The GraphX API is currently only available in Scala but we plan to provide Java and Python bindings in the future."

However, you should look at GraphFrames (https://github.com/graphframes/graphframes), which wraps GraphX algorithms under the DataFrames API and it provides Python interface.

Here is a quick example from https://graphframes.github.io/graphframes/docs/_site/quick-start.html, with slight modification so that it works.

First, start pyspark with the graphframes pkg loaded.

pyspark --packages graphframes:graphframes:0.1.0-spark1.6

python code:

Yes, you need to add the jar in the spark-submit command :

I was in your same situation, but fortunately I found a solution to get back a Dataframe.

Solution code is commented at steps [1], [2] and [3].

Okay, So it seems some kind of compatibility issue between spark2.4 and bahir2.4. I fixed it by rolling back both of them to ver 2.3.

Here is my build.sbt

These 2 lines contain errors:

The Graph Data Science library pathfinding algorithms are designed to find the shortest weighted paths and use algorithms similar to Dijkstra to find them. In your case, it seems that you are dealing with a directed unweighted graph and you could use the native cypher allShortestPath procedure:

An example would be:

You can use a case class as edge attribute and another as the vertex property. MyEdgesLabel is already ok for the edges, to crete the edge RDD, simply do:

The main problem is that you use two different types for the messages. The initial messages are of type mutable.HashMap[VertexId, (Double,VertexId)] but after merging two (with the mergeMessage function) the type becomes mutable.HashMap[VertexId, List[(Double,VertexId)]]. The problem here is that now the merged message can't merge with another message since the type is wrong.

There are two ways to solve this:

1. Change the message type to mutable.HashMap[VertexId, List[(Double,VertexId)]], making sure the initial messages matches this.
2. Keep the message type as mutable.HashMap[VertexId, (Double,VertexId)] and change the output type of mergeMessage to match.

Below are some sketches on possible solutions for both options. There could be some errors inside them since the actual wanted logic is not very clear (there are some unused variables in the code, etc.) Both options can run when combined with the rest of the code and will return a new graph.

Solution 1:

You need to adjust sendMessage, mergeMessage and initialMessage to handle Lists. This could be done as follows:

The problem is here: you have a graph that its nodes property is (VertexId,Double) but you have defined the e: EdgeTriplet[VertexId, ED] which needs to be corrected and it should be e: EdgeTriplet[(VertexId,Double), ED]. This means that your vertexes or your nodes property in Graph have (VertexId,Double)as their property.