learning-spark | Tidy up Spark and Hadoop tutorials

I'm trying to read a stream from a Kafka source containing JSON records using a pattern from the book Learning Spark:

I have a docker container running on my laptop with a master and three workers, I can launch the typical wordcount example by entering the ip of the master using a command like this:

I have an RDD in pyspark of the form (key, other things), where "other things" is a list of fields. I would like to get another RDD that uses a second key from the list of fields. For example, if my initial RDD is:

(User1, 1990 4 2 green...)
(User1, 1990 2 2 green...)
(User2, 1994 3 8 blue...)
(User1, 1987 3 4 blue...)

I would like to get (User1, [(1990, x), (1987, y)]),(User2, (1994 z))

where x, y, z would be an aggregation on the other fields, eg x is the count of how may rows I have with User1 and 1990 (two in this case), and I get a list with one tuple per year.

I am looking at the key value functions from: https://www.oreilly.com/library/view/learning-spark/9781449359034/ch04.html

But don't seem to find anything that will give and aggregation twice: once for user and one for year. My initial attempt was with combineByKey() but I get stuck in getting a list from the values.

Any help would be appreciated!

Can anyone explain me this?

The flipside, however, is that for transformations that cannot be guaranteed to pro‐ duce a known partitioning, the output RDD will not have a partitioner set. For example, if you call map() on a hash-partitioned RDD of key/value pairs, the function passed to map() can in theory change the key of each element, so the result will not have a partitioner. Spark does not analyze your functions to check whether they retain the key. Instead, it provides two other operations, mapValues() and flatMap Values(), which guarantee that each tuple’s key remains the same.

Source Learning Spark by Matei Zaharia, Patrick Wendell, Andy Konwinski, Holden Karau.

I try to get a basic regression run with Zeppelin 0.7.2 and Spark 2.1.1 on Debian 9. Both zeppelin are "installed" in /usr/local/ that means /usr/local/zeppelin/ and /usr/local/spark. Zeppelin also knows the correct SPARK_HOME. First I load the data:

I'm learning Spark by working through some of the examples in Learning Spark: Lightning Fast Data Analysis and then adding my own developments in.

I created this class to get a look at basic transformations and actions.

I'm new to Spark so am trying to setup a project from the book Learning Spark: Lightning-Fast Big Data Analysis. The book uses version 1.3 but I've only got 2.1.1 so am trying to work around a few differences.

All the Spark related jars that I'm importing into my Java project have a "import org.apache cannot be resolved". I know it's because the project cannot find the jar files specified.

I can manually add each by going to Build Path > Configure Build path and adding them to the Libraries section but I think I shouldn't need to do this. The project uses Maven so I believe if I have the Spark dependencies configured correctly in my pom.xml it should work. Is this correct?

I also set the following environment variables: