Multiprocessing-Example | Read full explaination on Medium

Let's say I have the following function:

I have a case where I am using PySpark (or Spark if I can't do it with Python and instead need to use Scala or Java) to pull data from several hundred database tables that lack primary keys. (Why Oracle would ever create an ERP product that contains tables with primary keys is a different subject... but regardless, we need to be able to pull the data and save the data from each database table into a Parquet file.) I originally tried using Sqoop instead of PySpark, but due to a number of issues we ran into, it made more sense to try using PySpark/Spark instead.

Ideally, I'd like to have each task node in my compute cluster: take the name of a table, query that table from the database, and save that table as a Parquet file (or set of Parquet files) in S3. My first step is to get it working locally in standalone mode. (If I had a primary key for each given table, then I could partition the query and file saving process across different sets of rows for the given table and distribute the row partitions across the task nodes in the compute cluster to perform the file saving operation in parallel, but because Oracle's ERP product lacks primary keys for the tables of concern, that's not an option.)

I'm able to successfully query the target database with PySpark, and I'm able to successfully save the data into a parquet file with multithreading, but for some reason, only a single thread does anything. So, what happens is that only a single thread takes a tableName, queries the database, and saves the file to the desired directory as a Parquet file. Then the job ends as if no other threads were executed. I'm guessing that there may be some type of locking issue taking place. If I correctly understood the comments here: How to run multiple jobs in one Sparkcontext from separate threads in PySpark? then what I'm trying to do should be possible unless there are specific issues related to executing parallel JDBC SQL queries.

Edit: I'm specifically looking for a way that allows me to use a thread pool of some type so that I don't need to manually create a thread for each one of the tables that I need to process and manually load-balance them across the task nodes in my cluster.

Even when I tried setting:

In the Logging Cookbook on Python, there are the following comments in the section "A more elaborate multiprocessing example":

I am planning to ingest scientific measurement data into my 6-node cassandra cluster using python script.

I have checked various posts and articles on bulk loading data into cassandra. But unfortunately, none of the state-of-the.-art as discussed fits my use-case [1][2]. However, I found this post on Stack Overflow which seemed quite helpful.

Considering that post and my billion of records data, I would like to know if the combination of using PreparedStatement (instead of Simple Statement) and execute_async is a good practice.

I have a code that works with Thread in python, but I wanna switch to Process as if I have understood well that will give me a speed-up. Here there is the code with Thread: