HadoopInternals | Diagrams describing Apache Hadoop internals

To understand where exactly the mapper is stuck at, jstack can be used [to get the thread dump].

Jstack ships with the jdk and you can use it on the stuck mapper process as follows.

Step0: Find the hostname in which your map task were running and make a note of the task_id

step1: login to the node and run

ps aux | grep task_id

identify the process id and the username of the process which is starting with /usr/java/jdk/bin/java 

step2: su to the process owner username

step3: export java home and bin path [example: export JAVA_HOME=/usr/java/jdk1.7.0_67 && export PATH=$JAVA_HOME/bin:$PATH]

step4:replace pid with pid you obtained in step1:

export PID=PID for i in $(seq 1 10); do echo "Jstack Iteration $i"; jstack $PID > /tmp/hungtask-hostname-${PID}.jstack.$i; sleep 5s; done tar zcvf hungtask.tar.gz /tmp/hungtask-hostname-${PID}.jstack.*

the hungtask.tar.gz will contain the thread dump of the process taken at intervals of 5 seconds for ten times. You may need to run the script at the point when the task goes into hung state.

After this if you can upload hungtask.tar.gz to this thread, I can see and share my observation.

Also to understand if the process is undergoing frequent GC you can try the below command

jstat -gc -t PID STEP

PID is the process ID of the java process to monitor  STEP is the sample timestep 

You can paste the contents to the website http://nix-on.blogspot.in/2015/01/java-jstat-how-to-visualize-garbage.html to understand if it is undergoing excessive GC