rdd | Python tools for regression discontinuity designs | Testing library

For casting a map to a json part: after asking a colleague, I understood that such casting couldn't work, simply because map type is key value one without any specific schema not like struct type. Because more information is needed, map to struct cast can't work.

For the loading a json part: I managed to solve the json issue after removing the json loading and using the "failfast" mode to load the json:

Honestly I have never used it, but I guess it is there because some transformations need an RDD as argument, whether it is empty or not. Suppose you need to perform an outer join and the RDD you are joining against depends on a condition that could determine its emptyness, like:

Manually push a context:

As @ewertonvsilva mentioned, this was related to import error. specifically ->

I would add the following option if you want to use jks

It seems that you were missing a VPC Endpoint for DynamoDB, since your Glue Jobs run in a private VPC when you write to Redshift.

Assuming you input RDD is something like:

There are a couple of things you can do to improve your computation:

Improve parallelism

As Nithish mentioned in the comments, you don't have enough partitions in your input data frames to make use of all your CPU cores. You're not using all your CPU capability and this will slow you down.

To increase your parallelism, repartition dfc to at least your number of cores:

dfc = dfc.repartition(dfc.sql_ctx.sparkContext.defaultParallelism)

You need to do this because your crossJoin is run as a BroadcastNestedLoopJoin which doesn't reshuffle your large input dataframe.

Separate your computation stages

A Spark dataframe/RDD is conceptually just a directed action graph (DAG) of operations to run on your input data but it does not hold data. One consequence of this behavior is that, by default, you'll rerun your computations as many times as you reuse your dataframe.

In your fuzzy_match_approve function, you run 2 separate filters on your df, this means you rerun the whole cross-join operations twice. You really don't want this !

One easy way to avoid this is to use cache() on your fuzzy_match result which should be fairly small given your inputs and matching criteria.

One of the issues is that partition is an overloaded term in Spark world and you're looking at 2 different kind of partitions:

• your dataset is organized as a Hive-partitioned table, where each partition is a separate directory named with = that may contain many data files inside. This is only useful for dynamically pruning the set of input files to read and has no effect on the actual RDD processing

• when Spark loads your data and creates a DataFrame/RDD, this RDD is organized in splits that can be processed in parallel and that are also called partitions.

df.rdd.getNumPartitions() returns the number of splits in your data and that is completely unrelated to your input table partitioning. It's determined by a number of config options but is mostly driven by 3 factors:

• computing parallelism: spark.default.parallelism in particular is the reason why you have 2 partitions in your RDD even though you don't have enough data to fill the first
• input size: spark will try to not create partitions bigger than spark.sql.files.maxPartitionBytes and thus may split a single multi-gigabyte parquet file into many partitions)
• shuffling: any operation that need to reorganize data for correct behavior (for example join or groupBy) will repartition your RDD with a new strategy and you will end up with many more partitions (governed by spark.sql.shuffle.partitions and AQE settings)

On the whole, you want to preserve this behavior since it's necessary for Spark to process your data in parallel and achieve good performance. When you use df.coalesce(1) you will coalesce your data into a single RDD partition but you will do your processing on a single core in which case simply doing your work in Pandas and/or Pyarrow would be much faster.

If what you want is to preserve the property on your output to have a single parquet file per Hive-partition attribute, you can use the following construct: