pythonutils | Miscellaneous utility functions in Python , mostly game

Can you avoid using queries and just use the table option?

Probably something with the way I supplied my credentials via hadoopConfiguration().set() in the python code was wrong. But there is another way of configuring flintrock (and more generally EC2 instances) to be able to access S3 without supplying credentials in the code (this is actually a recomded way of doing this when dealing with temporary credentials from AWS). The following helped:

• The flintrock docu, which says "Setup an IAM Role that grants access to S3 as desired. Reference this role when you launch your cluster using the --ec2-instance-profile-name option (or its equivalent in your config.yaml file)."
• This AWS documentation page that explains step-by-step how to do it.
• Another useful AWS docu page.
• Please note: If you create the above role via AWS Console then the respective instance profile with the same name is created automatically, otherwise (if you use awscli or AWS API) you have to create the desired instance profile manually as an extra step.

The problem is with your url.. In order to read data from github you have to pass the raw url instead.

On the data page click on raw and then copy that url to get the data

Please assign the SparkSession.builder result to a variable:

I just had a fresh pyspark installation on my Windows device and was having the exact same issue. What seems to have helped is the following:

Go to your System Environment Variables and add PYTHONPATH to it with the following value: %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j--src.zip:%PYTHONPATH%, just check what py4j version you have in your spark/python/lib folder.

The reason why I think this works is because when I installed pyspark using conda, it also downloaded a py4j version which may not be compatible with the specific version of spark, so it seems to package its own version.

Some config params are required to recognize "gs" as a distributed filesystem.

Use this setting for google cloud storage connector, gcs-connector-hadoop2-latest.jar

The udf function will be executed by multiple spark tasks in parallel. Those tasks run in completely isolated python processes and they are scheduled to physically different machines. Hence each data, those functions reference, must be on the same node. This is the case for everything created within the udf.

Whenever you reference any object outside of the udf from the function, this data structure needs to be serialised (pickled) to each executor. Some object state, like open connections to a socket, cannot be pickled.

You need to make sure, that connections are lazily opened each executor. It must happen only on the first function call on that executor. The connection pooling topic is covered in the docs, however only in the spark streaming guide (though it also applies for normal batch jobs).

Normally one can use the Singleton Pattern for this. But in python people use the Borgh pattern.

You can do the following

I am not expert in DataBricks or Spark, but pickling functions from the local notebook context is always problematic when you are touching complex objects like the service object. In this particular case, I would recommend removing the dependency on the azureML service object and just use requests to call the service.

Pull the key from the service: