Python-Libraries

You can use bootstraps to install additional modules while creating your EMR https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-bootstrap.html

You have to check if the link indeed has the "href" attribute:

To avoid writing any files, you can just save your image to BytesIO object. For example:

In Qubole are two ways to upgrade/install a package for the python environment. Currently there is no interface available inside notebook to install new packages.

New and Recommended Way (via Package Mangement) : User can enable Package Management functionality for an account and add new packages to a cluster via UI. There are lot of advantages of using package management over cluster versions in terms of performance and usability. Refer to https://docs.qubole.com/en/latest/user-guide/package-management/index.html for further details.

Old Way (via bootstrap) : User can configure a bootstrap which is basically a shell script executed on each node when the cluster starts and or upscales (more nodes are getting added to cluster). This can be configured via clusters UI and need a cluster start for every change. This is what is instructed in link you shared.

Found this question

AWS Glue Python

Which suggested using this:

For some reason, the fastavro package works as default on the same file. I ended up using the code below. Still not sure if there is a way to address this problem directly using the avro library, or to deserialise the output posted in the question above.

You can try with np.diagonal:

In comparing R packages like RPostgreSQL and RPostgres to SQL command-line or database management application, you're not limiting yourself at all. The R packages can provide the same command-line-like interface that the other tools do, such as by enabling dbExecute() through DBI and you can issue whatever statements are needed. Where differences may come in are in higher-level wrapper functions that may have trouble interfacing between R and SQL. Such issues might look like class/data type interpretation (eg, I believe R packages do not know how to resolve postgresql array data types as a specific example). This can affect upsert type commands, for example, or calls to append full data.frames onto database tables, since data types may mismatch when higher-level functions attempt to auto-map these incorrectly; similar issues may also exist in having to map NA to NULL or explicitly cast data types. There may also be issues in writing temporary tables and things of this nature. I'd consider these more as quirks or annoyances, since there are work-arounds that allow the same level of functionality. Such as by using lower-level function calls that require to write-out SQL more comprehensively; or by additional work to map data types that will translate between native SQL and R environments. (I replied to another SO question elsewhere with an example of such a work-around to use intermediate tables between R and the database: How to insert R dataframe into existing table in SQL Server). Stepping away from transaction level commands to fuller DBA needs, like managing users, permissions, underlying configurations and so forth may find some limitations as well. I'll also note that super popular though it may be, dplyr package, which provides very high level convenience functions for interacting with SQL is more likely to impose limitations than writing statements directly and using lower-level package calls. dplyr auto-builds SQL statements using the R function calls, but trying to do anything beyond relatively simple select/insert/update/join style calls is likely to require a fair amount of effort to enable the so-called shortcuts to perform as expected.

Before doing spark-submit try going to python shell and try importing the modules. Also check which python shell (check python path) is opening up by default.

If you are able to successfully import these modules in python shell (same python version as you trying to use in spark-submit), please check following:

In which mode are you submitting the application? try standalone or if on yarn try client mode. Also try adding export PYSPARK_PYTHON=(your python path)