hudi | Upserts , Deletes And Incremental Processing on Big Data

I am new to apace hudi and trying to write my dataframe in my Hudi table using spark shell. For type first time i am not creating any table and writing in overwrite mode so I am expecting it will create hudi table.I am Writing below code.

I am trying to integrate Hudi with Kafka topic.

Steps followed :

1. Created Kafka topic in Confluent with schema defined in schema registry.
2. Using kafka-avro-console-producer, I am trying to produce data.
3. Running Hudi Delta Streamer in continuous mode to consume the data.

Infrastructure :

1. AWS EMR
2. Spark 2.4.4
3. Hudi Utility ( Tried with 0.6.0 and 0.7.0 )
4. Avro ( Tried avro-1.8.2, avro-1.9.2 and avro-1.10.0 )

I am getting the below error stacktrace. Can someone please help me out with this?

I just started using delta lake so my mental model might be off - I'm asking this question to validate/refute it.

My understanding of delta lake is that it only stores incremental changes to data (the "deltas"). Kind of like git - every time you make a commit, you're not storing an entire snapshot of the codebase - a commit only contains the changes you made. Similarly, I would imagine that if I create a Delta table and then I attempt to "update" the table with everything it already contains (i.e. an "empty commit") then I would not expect to see any new data created as a result of that update.

However this is not what I observe: such an update appears to duplicate the existing table. What's going on? That doesn't seem very "incremental" to me.

(I'll replace the actual UUID values in the filenames for readability)

I am currently doing a POC on deltalake where I came across this framework called Apache Hudi. Below is the data I am trying to write using apache spark framework.

I am trying to setup Apache Hudi on an Ubuntu 16.04 server. I cloned the repo https://github.com/apache/incubator-hudi.git and then build it as

Im using spark to write my json data to s3. However, I keep getting the below error. We are using apache hudi for updates. This only happens for some data, everything else works fine.

I have a requirement where i need to write the stream using structured streaming to Hudi dataset. I found there is a provision to do this over Apache Hudi Jira issues but wanted to know if anyone successfully implemented this and have an example. I am trying to structure stream the data from AWS Kinesis Firehose to Apache Hudi using spark structured streaming

Quick help is appreciated.

In our data pipelines ,we ingest CDC events from data-sources and write these changes into "incremental data" folder in AVRO format.

Then periodically, we run Spark jobs to merge this "incremental data" with our current version of the "snapshot table" (ORC format) to get the latest version of the upstream snapshot.

During this merge logic :

1) we load the "incremental data" as an DataFrame df1

2) load the current "snapshot table" as an DataFrame df2

3) merge df1 and df2 de-duplicating ids and taking the latest version of the rows (using update_timestamp column)

This logic loads the entire data for both "incremental data" and current "snapshot table" into Spark memory which can be quite huge depend on the database.

I noticed that in Delta Lake, similar operation is done using following code:

In order to have a clean and DRY ionic project, I just wanted my navbar code to be written in one place, instead of writing the whole HTML in every Ionic page.

For this purpose, I created an Angular component (not an Ionic page) named navbar, and I inject it in my pages. To keep a clean layout with no additional stuff in the DOM, I created a component with the brackets notation, like this: