fuzzy_match | haystack using string similarity | Natural Language Processing library

There are a couple of things you can do to improve your computation:

Improve parallelism

As Nithish mentioned in the comments, you don't have enough partitions in your input data frames to make use of all your CPU cores. You're not using all your CPU capability and this will slow you down.

To increase your parallelism, repartition dfc to at least your number of cores:

dfc = dfc.repartition(dfc.sql_ctx.sparkContext.defaultParallelism)

You need to do this because your crossJoin is run as a BroadcastNestedLoopJoin which doesn't reshuffle your large input dataframe.

Separate your computation stages

A Spark dataframe/RDD is conceptually just a directed action graph (DAG) of operations to run on your input data but it does not hold data. One consequence of this behavior is that, by default, you'll rerun your computations as many times as you reuse your dataframe.

In your fuzzy_match_approve function, you run 2 separate filters on your df, this means you rerun the whole cross-join operations twice. You really don't want this !

One easy way to avoid this is to use cache() on your fuzzy_match result which should be fairly small given your inputs and matching criteria.