BigDL | Fast, distributed, secure AI for Big Data

The directory paths in the error message are wrong. The Python 3.5 kernel on DSX specifies a build directory for pip by setting the environment variable PIP_BUILD. The multiple dist-info directories are there:

If your notebooks are backed by an Apache Spark as a Service instance in DSX, installing BigDL is simple. But you have to collect some version information first.

1. Which Spark version? Currently, 2.1 is the latest supported by DSX.
   With Python, you can only install BigDL for one Spark version per service.
2. Which BigDL version? Currently, 0.3.0 is the latest, and it supports Spark 2.1.
   If in doubt, check the download page. The Spark fixlevel does not matter.

With this information, you can determine the URL of the required BigDL JAR file in the Maven repository. For the example versions, BigDL 0.3.0 with Spark 2.1, the download URL is
https://repo1.maven.org/maven2/com/intel/analytics/bigdl/bigdl-SPARK_2.1/0.3.0/bigdl-SPARK_2.1-0.3.0-jar-with-dependencies.jar

For other versions, replace 0.3.0 and 2.1 in that URL as required. Note that both versions appear twice, once in the path and once in the filename.

You need the JAR, and the matching Python package. The Python package depends only on the version of BigDL, not on the Spark version. The installation steps can be executed from a Python notebook:

1. Install the JAR.

The python script is trying to create a bigdl.log file in the /usr/local/lib/python3.5/dist-packages/bigdl/bigdl.log which is a protected directory in linux accessible through root-access only.

You can specify a log file path to redire_spark_logs function something like this redire_spark_logs(log_path='/home/bigdl-projects'). Look here for more details.

Try to add this while(true) Thread.sleep(1000) in your code, to keep the server running then check the sparks task in the browser. Normally you should see your application running.

If you generalize the idea and think of it as a monad (polymorphic thing working for arbitrary type params) then you won't be able to implement a tail recursive implementation.

Trampolines try to solve this very problem by providing a way to evaluate a recursive computation without overflowing the stack. The general idea is to create a stream of pairs of (result, computation). So at each step you'll have to return the computed result up to that point and a function to create the next result (aka thunk).

From Rich Dougherty’s blog:

A trampoline is a loop that repeatedly runs functions. Each function, called a thunk, returns the next function for the loop to run. The trampoline never runs more than one thunk at a time, so if you break up your program into small enough thunks and bounce each one off the trampoline, then you can be sure the stack won't grow too big.

In the categorical sense, the theory behind such data types is closely related to Cofree Monads and fold and unfold functions, and in general to Fixed point types.

See this fantastic talk: Fun and Games with Fix Cofree and Doobie by Rob Norris which discusses a use case very similar to your question.

This article about Free monads and Trampolines is also related to your first question: Stackless Scala With Free Monads.

See also this part of the Matryoshka docs. Matryoshka is a Scala library implementing monads around the concept of FixedPoint types.