spark | Lightning-fast cluster computing in Java , Scala and Python

This seems like a bug introduced by a bug fix in this ticket. The result was wrong for outer joins. Hence the need to add a Project node (packing of the struct) before the Join node.

However, we end up with this kind of query plan:

you could make Workflow implement Serializeble and SparkSession as @transient

do some query work with Spark SQL

You wouldn't use a ForEachWriter for that

You can use sum function, casting your event as an int:

You are using scalatest version 2.2.6:

Support for _PIP_ADDITIONAL_REQUIREMENTS environment variable has not been released yet. It is only supported by the developer/unreleased version of the docker image. It is planned that this feature will be available in Airflow 2.1.1. For more information, see: Adding extra requirements for build and runtime of the PROD image.

For the older version, you should build a new image and set this image in the docker-compose.yaml. To do this, you need to follow a few steps.

1. Create a new Dockerfile with the following content:

You need to explicitly create the object for that test suite & execute it. In IDE you're relying on specific runner, but it doesn't work in the notebook environment.

You can use either the .execute function of create object (docs):

Yes.

If your topic has 1 partition means it is in one broker. So, If you increase producer rate for the topic, then that broker becomes busy. But if you have multiple partitions, your Kafka cluster shared those partitions into different brokers and those production rate shared within multiple brokers. So, Writing the same dataset in one topic with 4 partitions is more fast than writing in a topic with 1 partition.

This not only production rate. In Kafka brokers, There is multiple processes like compactions, compressions, segmentations etc... So with number of messages, that work load becomes high. But with multiple partitions in multiple brokers, it will be distributed.

However, you don’t necessarily want to use more partitions than needed because increasing partition count simultaneously increases the number of open server files and leads to increased replication latency.

from kafka documentation

Distribution The partitions of the log are distributed over the servers in the Kafka cluster with each server handling data and requests for a share of the partitions. Each partition is replicated across a configurable number of servers for fault tolerance. Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster.