jdf | 🛠️JD front-end automated construction tool

I want to do a pct_change() at group level and I can get one value column done. But I am wondering how to do it at multiple value columns and bring them back to the dataframe with a prefix name like pc_.

I have a Structured Streaming pyspark program running on GCP Dataproc, which reads data from Kafka, and does some data massaging, and aggregation. I'm trying to use withWatermark(), and it is giving error.

Here is the code :

I've two dataframes, I need to update records in df1 based on new updates available in df2 in pyspark.

DF1:

I am not able to assign a series as a new column to a koalas dataframe. Below is the codebase that I am using:

I'm trying to find missing and null values from my dataframe but I'm getting an exception. I have included only the initial few schema below:

I've searched through many solutions here but can't get exactly what I'm looking for in this situation.

I pull a JSON file from an API and get as far as get it into a list. The data is basically a list of values for a day in 5 minute intervals. The values are in a list of length 288 (as there are 288 5 minute intervals in a day) and there is a startTime object that starts at date-00:00:00 and then another of intervalLength '00:05:00' but these are not lists.

I'm new to pyspark. I'd like to be able to read values from my kafka topic. To do so, I've created a schema for messages in my topic.

Here is example message in my kafka topic:

I have a json with nested lists @ "ManyActionDateTimes" and @ "Comments" like this:

I have created a membership system in my application. I create record with Firebase Auth and store some information in firebase realtime . However, I saw that 1% of the users did not print their data to firebase realtime during record creation. If users' data is not written to firebase realtime, my application will crash. I haven't been able to figure out how to fix this problem for weeks. I would be glad if you help. The table where I store user information: users

Regard.

This my code;