sparklyr | R interface for Apache Spark

I am using RStudio Cloud and I want to connect to Spark using sparklyr package. I tried a local master and a yarn master. The code is as below.

New in Azure Databricks environment, I discover the packages SparkR and sparklyr.

From my notebooks with SparkR, I manage to connect to a database :

Background

I'm doing some data manipulation (joins, etc.) on a very large dataset in R, so I decided to use a local installation of Apache Spark and sparklyr to be able to use my dplyr code to manipulate it all. (I'm running Windows 10 Pro; R is 64-bit.) I've done the work needed, and now want to output the sparklyr table to a .csv file.

The Problem

Here's the code I'm using to output a .csv file to a folder on my hard drive:

I'm using: R version 4.1.1 sparklyr version ‘1.7.2’

I'm connected to my databricks cluster with databricks-connect and trying to read an avro file using the following code:

I have a Spark dataframe in a SparklyR interface, and I'm trying to extract elements from an array column.

I have a spark dataframe I'm manipulating using sparklyr that looks like the following:

Background

This may be my lack of skill showing, but as I'm working on data manipulation in R, using RStudio, I'm fond of clicking into dataframes in the "Environments" section of the GUI (for me it's in the top-right of the screen) to see how my joins, mutates, etc. are changing the table(s) as I move through my workflow. It acts as a visual sanity check for me; when it comes to tables and dataframes I'm a very visual thinker, and I like to see my results as I code. As an example, I click on this:

And see something like this:

The Problem

Lately, because of a very large dataset (~200m rows), I've needed to do some of my dplyr work inside sparklyr, using a local instance of Apache Spark to work through some data manipulation. It's working mostly fine, but I lose my ability to have little previews of the data because spark dataframe objects look like lists in the Environment pane:

Besides clicking, is there a way I can "preview" my Spark dataframes inside RStudio as I work on them?

What I've tried

So your first thought might be "just use head()" -- and you'd be right! Except that running head(d1, 5) on a local Spark df with 200 million rows takes ... a long time.

Anything I may be missing?

Im trying to convert the following to a 24 hour time using sparklyr:

I'm trying to develop a simple logistic regression model using Tidymodels with the Spark engine. My code works fine when I specify set_engine = "glm", but fails when I attempt to set the engine to spark. Any advice would be much appreciated!