hadoop-hdfs | Hadoop分布式文件系统hdfs代码分析

I was doing a POC of flink CDC + iceberg. I followed this debezium tutorial to send cdc to kafka - https://debezium.io/documentation/reference/1.4/tutorial.html. My flink job was working fine and writing data to hive table for inserts. But when I fired an update/delete query to the mysql table, I started getting this error in my flink job. I have also attached the output of retract stream

Update query - UPDATE customers SET first_name='Anne Marie' WHERE id=1004;

I'm troubleshooting YARN application failures that happen when nodes are LOST, so I'm trying to recreate this scenario. But I'm only able to force nodes to be SHUTDOWN instead of LOST. I'm using AWS EMR, and I've tried:

• logging into a node and doing a shutdown -h now
• logging into a node and doing sudo stop hadoop-yarn-nodemanager and sudo stop hadoop-hdfs-datanode
• killing the NodeManager with a kill -9

Those result in SHUTDOWN nodes but not LOST nodes.

How do I create a LOST node in AWS EMR?

I installed Cloudera Quickstart VM 5.13 on virtualbox and I'm trying to start hadoop server with the command sudo service hadoop-hdfs-namenode start but 'm getting this two errors: log4j:ERROR Could not find value for key log4j.appender.RFA and log4j:ERROR Could not instantiate appender named "RFA". Can anyone can help me with this issue?

HDFS has a feature called storage types/policies - it makes possible to store files on stores with different properties (fast SSD or slow but cheap archival storage).

I wonder if it's possible to use this feature through HBase?

My use case is that I have some data that is "hot" and is expected to be accessed frequently so I want to put it in "hot" (SSD) storage and some data is "cold" and is accessed infrequently so I want to put it on a cheaper storage. And I'm trying to find out how to organize this with HBase/HDFS.

i'm trying to write simple data into the table by Apache Iceberg 0.9.1, but error messages show. I want to CRUD data by Hadoop directly. i create a hadooptable , and try to read from the table. after that i try to write data into the table . i prepare a json file including one line. my code have read the json object, and arrange the order of the data, but the final step writing data is always error. i've changed some version of dependency packages , but another error messages are show. Are there something wrong on version of packages. Please help me.

this is my source code:

I read Hadoop ver 3 document about disk balancer and it said

"Diskbalancer is a command line tool that distributes data evenly on all disks of a datanode.
This tool is different from Balancer which takes care of cluster-wide data balancing."

I really dont know whats difference between 'balancer' and 'disk balancer' yet.
Could you explain what is it?
Thank you!

IntelliJ IDEA cannot import Spark mllib, when I added dependency in maven. With other parts of Spark no problems. In project Structure -> Libraries spark mllib is present.

Here I am trying to execute Structured Based Streaming with Apache Kafka. But in here not working and execute error (ERROR MicroBatchExecution: Query [id = daae4c34-9c8a-4c28-9e2e-88e5fcf3d614, runId = ca57d90c-d584-41d3-a8de-6f9534ead0a0] terminated with error java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z). How can i solve this issue. I work on windows 10 machine.

App Class:

I have installed GCS connector of hadoop 3 version and added the below config to core-site.xml as described in Install.md . The intention is to migrate data from hdfs in local cluster to cloud storage.

core-site.xml