fast-cluster | fast-cluster - Cluster documents using LSH in linear time

In the original Spark RDD model, RDDs described distributed collections of Java objects or pickled Python objects. However, SparkSQL "dataframes" (including Dataset) represent queries against one or more sources/parents.

To evaluate a query and produce some result, Spark does need to process records and fields, but these are represented internally in a binary, language-neutral format (called "encoded"). Spark can decode these formats to any supported language (e.g., Python, Scala, R) when needed, but will avoid doing so if it's not explicitly required.

For example: if I have a text file on disk, and I want to count the rows, and I use a call like:

spark.read.text("/path/to/file.txt").count()

there is no need for Spark to ever convert the bytes in the text to Python strings -- Spark just needs to count them.

Or, if we did a spark.read.text("...").show() from PySpark, then Spark would need to convert a few records to Python strings -- but only the ones required to satisfy the query, and show() implies a LIMIT so only a few records are evaluated and "decoded."

In summary, with the SQL/DataFrame/DataSet APIs, the language you use to manipulate the query (Python/R/SQL/...) is just a "front-end" control language, it's not the language in which the actual computation is performed nor does it require converting original data sources to the language you are using. This approach allows higher performance across all language front ends.