hdfs | go bindings for libhdfs

When a particular task fails that causes RDD to be recomputed from lineage (maybe by reading input file again), how does Spark ensure that there is no duplicate processing of data? What if the task that failed had written half of the data to some output like HDFS or Kafka ? Will it re-write that part of the data again? Is this related to exactly once processing?

we have file like the following

more file

My code:

I have 3 remote computers (servers):

• computer 1 has internal IP: 10.1.7.245
• computer 2 has internal IP: 10.1.7.246
• computer 3 has internal IP: 10.1.7.247

(The 3 computers above are in the same network, these 3 computers are all using Ubuntu 18.04.5 LTS Operating System)

(My personal laptop is in another different network, my laptop also uses Ubuntu 18.04.5 LTS Operating System)

I use my personal laptop to connect to the 3 remote computers using SSH protocol and using user root : (Below ABC is a name)

• computer 1: ssh root@ABC.University.edu.vn -p 12001
• computer 2: ssh root@ABC.University.edu.vn -p 12002
• computer 3: ssh root@ABC.University.edu.vn -p 12003

I have successfully set up a Hadoop Cluster which contains 3 above computer:

• computer 1 is the Hadoop Master
• computer 2 is the Hadoop Slave 1
• computer 3 is the Hadoop Slave 2

======================================================

I starts HDFS of the Hadoop Cluster by using the below command on Computer 1: start-dfs.sh

Everything is successful:

• computer 1 (the Master) is running the NameNode
• computer 2 (the Slave 1) is running the DataNode
• computer 3 (the Slave 2) is running the DataNode

I know that the the Web Interface for the NameNode is running on Computer 1, on IP 0.0.0.0 and on port 9870 . Therefore, if I open the web browser on computer 1 (or on computer 2, or on computer 3), I will enter the 10.1.7.245:9870 on the URL bar (address bar) of the web browser to see the Web Interface of the NameNode.

======================================================

Now, I am using the web browser of my personal laptop.

How could I access to the Web Interface of the NameNode ?

I've always heard that Spark is 100x faster than classic Map Reduce frameworks like Hadoop. But recently I'm reading that this is only true if RDDs are cached, which I thought was always done but instead requires the explicit cache () method.

I would like to understand how all produced RDDs are stored throughout the work. Suppose we have this workflow:

1. I read a file -> I get the RDD_ONE
2. I use the map on the RDD_ONE -> I get the RDD_TWO
3. I use any other transformation on the RDD_TWO

QUESTIONS:

if I don't use cache () or persist () is every RDD stored in memory, in cache or on disk (local file system or HDFS)?

if RDD_THREE depends on RDD_TWO and this in turn depends on RDD_ONE (lineage) if I didn't use the cache () method on RDD_THREE Spark should recalculate RDD_ONE (reread it from disk) and then RDD_TWO to get RDD_THREE?

Thanks in advance.

I am trying to drop a table names "union" but I keep getting an error. I am not sure who and how created that table, but nothing works on it, including describe or select. Using "hdfs dfs -ls" outside of hive, I can see that table exists and there is data in it, but cannot drop the table. I am assuming there may be a problem because the table is called "union" and the error I get is

"cannot recognize input near 'union'".

How can I drop the table?

I have a hive external partitioned table with following data structure:

I have some twice-partitioned files in HDFS with the following structure:

I have two tables in impala and I want to move the data from one to another. Both tables have hdfs path like