Hadoop-2.2.0源码编译，搭建与配置

Sep302014

作者：Fisher 发布：2014-09-30 14:03 分类：云计算, 分布式抢沙发

【摘要】之前一直使用Hadoop1.x的版本，计划升级到2.x，找了3台测试vm机器，搭建了一下，本以为非常简单，但由于Hadoop2.x版本与Hadoop1.x版本变化较大，还是费了一些周折，下面是一些详细的步骤。本例的步骤应该可以跑起来，Hadoop参数还需要一些具体的优化。

一、搭建：
下载需要的包：JDK,svn,cmake,ncurses,openssl,gcc,maven,protobuf

mkdir software
cd software
wget http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
wget http://mirrors.cnnic.cn/apache/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz
wget https://github.com/twitter/hadoop-lzo/archive/master.zip -O lzo.tgz
wget wget http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.4-bin.tar.gz
wget "http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gz?AuthParam=1411887643_cf59aa6f30309ae6b7447b4621e645a1"
wget --no-check-certificate "https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz"

mkdir software

cd software

wget http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz

wget http://mirrors.cnnic.cn/apache/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz

wget https://github.com/twitter/hadoop-lzo/archive/master.zip -O lzo.tgz

wget wget http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.4-bin.tar.gz

wget "http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gz?AuthParam=1411887643_cf59aa6f30309ae6b7447b4621e645a1"

wget --no-check-certificate "https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz"

操作系统为：CentOS6.5-64bit
1. 安装需要的包：

yum install svn ncurses-devel autoconf automake libtool cmake openssl-devel gcc* telnet screen wget curl -y

1	yum install svn ncurses-devel autoconf automake libtool cmake openssl-devel gcc* telnet screen wget curl -y

2.安装maven,下载并解压maven:

wget http://mirrors.cnnic.cn/apache/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz
tar zxvf apache-maven-3.2.3-bin.tar.gz
mv apache-maven-3.2.3 /usr/local/

wget http://mirrors.cnnic.cn/apache/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz

tar zxvf apache-maven-3.2.3-bin.tar.gz

mv apache-maven-3.2.3 /usr/local/

3. 安装ant,下载并解压ant:

wget http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.4-bin.tar.gz
tar zvxf apache-ant-1.9.4-bin.tar.gz
mv apache-ant-1.9.4 /usr/local/

wget http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.4-bin.tar.gz

tar zvxf apache-ant-1.9.4-bin.tar.gz

mv apache-ant-1.9.4 /usr/local/

4.安装protobuf

tar zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
.configure
make && make install
$protoc --version
  libprotoc 2.5.0

tar zxvf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

.configure

make && make install

$protoc --version

libprotoc 2.5.0

5.配置环境变量：

#cat /etc/profile
PATH=/usr/java/jdk/bin:/home/hadoop/hadoop/sbin:/home/hadoop/hadoop/bin:/usr/local/maven/bin:/usr/local/ant/bin:$PATH

JAVA_HOME=/usr/java/jdk
HADOOP_INSTALL=/home/hadoop/hadoop
HADOOP_CONF_DIR=/home/hadoop/hadoop/etc/hadoop
CLASSPATH=:/usr/java/jdk/lib/:/usr/java/jdk/jre/lib     

export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native" 

export JAVA_HOME CLASSPATH HADOOP_INSTALL HADOOP_CONF_DIR PATH

#cat /etc/profile

PATH=/usr/java/jdk/bin:/home/hadoop/hadoop/sbin:/home/hadoop/hadoop/bin:/usr/local/maven/bin:/usr/local/ant/bin:$PATH

JAVA_HOME=/usr/java/jdk

HADOOP_INSTALL=/home/hadoop/hadoop

HADOOP_CONF_DIR=/home/hadoop/hadoop/etc/hadoop

CLASSPATH=:/usr/java/jdk/lib/:/usr/java/jdk/jre/lib

export HADOOP_HOME=/home/hadoop/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native"

export JAVA_HOME CLASSPATH HADOOP_INSTALL HADOOP_CONF_DIR PATH

6. 源码编译Hadoop
官方提供的二进制下载文件是32位的，因为虚拟机是64位的，所以需要下载源码进行手动编译：

tar zxvf hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src/
mvn package -Pdist,native -DskipTests -Dtar
mvn编译时间很长，虚拟机里面编译了11分钟。

最后提示整个编译情况：
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.057 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:02 min
[INFO] Finished at: 2014-09-29T18:51:40+08:00
[INFO] Final Memory: 137M/335M
[INFO] ------------------------------------------------------------------------

tar zxvf hadoop-2.2.0-src.tar.gz

cd hadoop-2.2.0-src/

mvn package -Pdist,native -DskipTests -Dtar

mvn编译时间很长，虚拟机里面编译了11分钟。

最后提示整个编译情况：

[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.057 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 11:02 min

[INFO] Finished at: 2014-09-29T18:51:40+08:00

[INFO] Final Memory: 137M/335M

[INFO] ------------------------------------------------------------------------

编译后的包：

cd hadoop-2.2.0-src/hadoop-dist/target
ls -tlr
drwxr-xr-x. 2 root root      4096 9月  29 18:51 antrun
drwxr-xr-x. 2 root root      4096 9月  29 18:51 test-dir
-rw-r--r--. 1 root root      1627 9月  29 18:51 dist-layout-stitching.sh
drwxr-xr-x. 9 root root      4096 9月  29 18:51 hadoop-2.2.0
drwxr-xr-x. 2 root root      4096 9月  29 18:51 maven-archiver
-rw-r--r--. 1 root root      2743 9月  29 18:51 hadoop-dist-2.2.0.jar
-rw-r--r--. 1 root root       644 9月  29 18:51 dist-tar-stitching.sh
-rw-r--r--. 1 root root  96166470 9月  29 18:51 hadoop-2.2.0.tar.gz
drwxr-xr-x. 2 root root      4096 9月  29 18:51 javadoc-bundle-options
-rw-r--r--. 1 root root 192884820 9月  29 18:51 hadoop-dist-2.2.0-javadoc.jar

cd hadoop-2.2.0-src/hadoop-dist/target

ls -tlr

drwxr-xr-x. 2 root root 4096 9月 29 18:51 antrun

drwxr-xr-x. 2 root root 4096 9月 29 18:51 test-dir

-rw-r--r--. 1 root root 1627 9月 29 18:51 dist-layout-stitching.sh

drwxr-xr-x. 9 root root 4096 9月 29 18:51 hadoop-2.2.0

drwxr-xr-x. 2 root root 4096 9月 29 18:51 maven-archiver

-rw-r--r--. 1 root root 2743 9月 29 18:51 hadoop-dist-2.2.0.jar

-rw-r--r--. 1 root root 644 9月 29 18:51 dist-tar-stitching.sh

-rw-r--r--. 1 root root 96166470 9月 29 18:51 hadoop-2.2.0.tar.gz

drwxr-xr-x. 2 root root 4096 9月 29 18:51 javadoc-bundle-options

-rw-r--r--. 1 root root 192884820 9月 29 18:51 hadoop-dist-2.2.0-javadoc.jar

需要的是hadoop-2.2.0.tar.gz，64bit的哦。

二、Hadoop配置
三个节点：

/etc/hosts
172.16.1.32     kvm27-v02.sudops.com
172.16.1.35     kvm26-v02.sudops.com
172.16.1.31     kvm28-v01.sudops.com

/etc/hosts

172.16.1.32 kvm27-v02.sudops.com

172.16.1.35 kvm26-v02.sudops.com

172.16.1.31 kvm28-v01.sudops.com

$HADOOP_HOME=/home/hadoop/hadoop

配置文件不在conf中了，新版本的目录是在： $HADOOP_HOME/etc/hadoop
hdfs-site.xml
mapred-site.xml
yarn-site.xml
core-site.xml
yarn-env.sh
hadoop-env.sh

cat core-site.xml

<configuration>
<!-- fs config -->
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://172.16.1.35:9000</value>
</property>
<property>
  <name>hadoop.http.staticuser.user</name>
  <value>hadoop</value>
</property>

</configuration>

<name>fs.defaultFS</name>

</property>

<name>hadoop.http.staticuser.user</name>

<value>hadoop</value>

</property>

</configuration>

cat hdfs-site.xml

<configuration>
<property>
     <name>dfs.namenode.name.dir</name>
     <value>/home/hadoop/hdfs/data/dfs.name.dir</value>
</property>
<property>
     <name>dfs.datanode.data.dir</name>
     <value>/home/hadoop/hdfs/data/dfs.data.dir</value>
</property>
<property>
     <name>dfs.permissions</name>
     <value>false</value>
</property>
<property>
     <name>dfs.replication</name>
     <value>3</value>
</property>
</configuration>

<name>dfs.namenode.name.dir</name>

<value>/home/hadoop/hdfs/data/dfs.name.dir</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/home/hadoop/hdfs/data/dfs.data.dir</value>

</property>

<name>dfs.permissions</name>

<value>false</value>

</property>

<name>dfs.replication</name>

</property>

</configuration>

cat mapred-site.xml

<configuration>
<property>
  <name>mapreduce.framework.name</name> 
  <value>Yarn</value> 
</property>
</configuration>

<name>mapreduce.framework.name</name>

</property>

</configuration>

cat yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property> 
  <name>yarn.nodemanager.aux-services</name> 
  <value>mapreduce_shuffle</value> 
</property> 

<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
　　<name>yarn.resourcemanager.address</name>
　　<value>172.16.1.35:8032</value>
</property>
<property>
　　<name>yarn.resourcemanager.scheduler.address</name>
　　<value>172.16.1.35:8030</value>
</property>
<property>
　　<name>yarn.resourcemanager.resource-tracker.address</name>
　　<value>172.16.1.35:8031</value>
</property>   
<property>
　　<name>yarn.resourcemanager.admin.address</name>
　　<value>172.16.1.35:8033</value>
</property>   
<property>
　　<name>yarn.resourcemanager.webapp.address</name>
　　<value>172.16.1.35:8088</value>
</property>

</configuration>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

　　<name>yarn.resourcemanager.address</name>

</property>

　　<name>yarn.resourcemanager.scheduler.address</name>

</property>

　　<name>yarn.resourcemanager.resource-tracker.address</name>

</property>

　　<name>yarn.resourcemanager.admin.address</name>

</property>

　　<name>yarn.resourcemanager.webapp.address</name>

</property>

</configuration>

三、Hadoop启动
hdfs namenode -format

启动hdfs

hadoop-daemon.sh start namenode
hadoop-daemon.sh start datanode
hadoop-daemon.sh start secondarynamenode
或者：start-dfs.sh

启动yarn
yarn-daemon.sh start resourcemanager
yarn-daemon.sh start nodemanager
或者：start-yarn.sh

 $ start-dfs.sh
Starting namenodes on [kvm26-v02.sudops.com]
kvm26-v02.sudops.com: starting namenode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-namenode-kvm26-v02.sudops.com.out
172.16.1.32: starting datanode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-datanode-kvm27-v02.sudops.com.out
172.16.1.31: starting datanode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-datanode-kvm28-v01.sudops.com.out
172.16.1.35: starting datanode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-datanode-kvm26-v02.sudops.com.out
Starting secondary namenodes [kvm27-v02.sudops.com]
kvm27-v02.sudops.com: starting secondarynamenode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-secondarynamenode-kvm27-v02.sudops.com.out

 $ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop-2.2.0/logs/yarn-hadoop-resourcemanager-kvm26-v02.sudops.com.out
172.16.1.32: starting nodemanager, logging to /home/hadoop/hadoop-2.2.0/logs/yarn-hadoop-nodemanager-kvm27-v02.sudops.com.out
172.16.1.31: starting nodemanager, logging to /home/hadoop/hadoop-2.2.0/logs/yarn-hadoop-nodemanager-kvm28-v01.sudops.com.out
172.16.1.35: starting nodemanager, logging to /home/hadoop/hadoop-2.2.0/logs/yarn-hadoop-nodemanager-kvm26-v02.sudops.com.out

$ start-dfs.sh

Starting namenodes on [kvm26-v02.sudops.com]

kvm26-v02.sudops.com: starting namenode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-namenode-kvm26-v02.sudops.com.out

172.16.1.32: starting datanode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-datanode-kvm27-v02.sudops.com.out

172.16.1.31: starting datanode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-datanode-kvm28-v01.sudops.com.out

172.16.1.35: starting datanode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-datanode-kvm26-v02.sudops.com.out

Starting secondary namenodes [kvm27-v02.sudops.com]

kvm27-v02.sudops.com: starting secondarynamenode, logging to /home/hadoop/hadoop-2.2.0/logs/hadoop-hadoop-secondarynamenode-kvm27-v02.sudops.com.out

$ start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /home/hadoop/hadoop-2.2.0/logs/yarn-hadoop-resourcemanager-kvm26-v02.sudops.com.out

172.16.1.32: starting nodemanager, logging to /home/hadoop/hadoop-2.2.0/logs/yarn-hadoop-nodemanager-kvm27-v02.sudops.com.out

172.16.1.31: starting nodemanager, logging to /home/hadoop/hadoop-2.2.0/logs/yarn-hadoop-nodemanager-kvm28-v01.sudops.com.out

172.16.1.35: starting nodemanager, logging to /home/hadoop/hadoop-2.2.0/logs/yarn-hadoop-nodemanager-kvm26-v02.sudops.com.out

$ hdfs dfsadmin -report
Configured Capacity: 1032277524480 (961.38 GB)
Present Capacity: 978602246144 (911.39 GB)
DFS Remaining: 978602172416 (911.39 GB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 3 (3 total, 0 dead)

Live datanodes:
Name: 172.16.1.32:50010 (kvm27-v02.sudops.com)
Hostname: kvm27-v02.sudops.com
Decommission Status : Normal
Configured Capacity: 344092508160 (320.46 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 17817788416 (16.59 GB)
DFS Remaining: 326274695168 (303.87 GB)
DFS Used%: 0.00%
DFS Remaining%: 94.82%
Last contact: Tue Sep 30 10:19:35 CST 2014


Name: 172.16.1.31:50010 (kvm28-v01.sudops.com)
Hostname: kvm28-v01.sudops.com
Decommission Status : Normal
Configured Capacity: 344092508160 (320.46 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 17817526272 (16.59 GB)
DFS Remaining: 326274957312 (303.87 GB)
DFS Used%: 0.00%
DFS Remaining%: 94.82%
Last contact: Tue Sep 30 10:19:35 CST 2014


Name: 172.16.1.35:50010 (kvm26-v02.sudops.com)
Hostname: kvm26-v02.sudops.com
Decommission Status : Normal
Configured Capacity: 344092508160 (320.46 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 18039963648 (16.80 GB)
DFS Remaining: 326052519936 (303.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 94.76%
Last contact: Tue Sep 30 10:19:35 CST 2014

$ hdfs dfsadmin -report

Configured Capacity: 1032277524480 (961.38 GB)

Present Capacity: 978602246144 (911.39 GB)

DFS Remaining: 978602172416 (911.39 GB)

DFS Used: 73728 (72 KB)

DFS Used%: 0.00%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

-------------------------------------------------

Datanodes available: 3 (3 total, 0 dead)

Live datanodes:

Name: 172.16.1.32:50010 (kvm27-v02.sudops.com)

Hostname: kvm27-v02.sudops.com

Decommission Status : Normal

Configured Capacity: 344092508160 (320.46 GB)

DFS Used: 24576 (24 KB)

Non DFS Used: 17817788416 (16.59 GB)

DFS Remaining: 326274695168 (303.87 GB)

DFS Used%: 0.00%

DFS Remaining%: 94.82%

Last contact: Tue Sep 30 10:19:35 CST 2014

Name: 172.16.1.31:50010 (kvm28-v01.sudops.com)

Hostname: kvm28-v01.sudops.com

Decommission Status : Normal

Configured Capacity: 344092508160 (320.46 GB)

DFS Used: 24576 (24 KB)

Non DFS Used: 17817526272 (16.59 GB)

DFS Remaining: 326274957312 (303.87 GB)

DFS Used%: 0.00%

DFS Remaining%: 94.82%

Last contact: Tue Sep 30 10:19:35 CST 2014

Name: 172.16.1.35:50010 (kvm26-v02.sudops.com)

Hostname: kvm26-v02.sudops.com

Decommission Status : Normal

Configured Capacity: 344092508160 (320.46 GB)

DFS Used: 24576 (24 KB)

Non DFS Used: 18039963648 (16.80 GB)

DFS Remaining: 326052519936 (303.66 GB)

DFS Used%: 0.00%

DFS Remaining%: 94.76%

Last contact: Tue Sep 30 10:19:35 CST 2014

参考文档
https://hadoop.apache.org/docs/r2.2.0/

测试

cd $HADOOP_HOME
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 20 10
Number of Maps  = 20
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Wrote input for Map #12
Wrote input for Map #13
Wrote input for Map #14
Wrote input for Map #15
Wrote input for Map #16
Wrote input for Map #17
Wrote input for Map #18
Wrote input for Map #19
Starting Job
14/09/30 11:45:40 INFO client.RMProxy: Connecting to ResourceManager at /172.16.1.35:8032
14/09/30 11:45:41 INFO input.FileInputFormat: Total input paths to process : 20
14/09/30 11:45:41 INFO mapreduce.JobSubmitter: number of splits:20
14/09/30 11:45:41 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/09/30 11:45:41 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/09/30 11:45:41 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/09/30 11:45:41 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/09/30 11:45:41 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/09/30 11:45:41 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/09/30 11:45:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1412048707137_0001
14/09/30 11:45:41 INFO impl.YarnClientImpl: Submitted application application_1412048707137_0001 to ResourceManager at /172.16.1.35:8032
14/09/30 11:45:41 INFO mapreduce.Job: The url to track the job: http://kvm26-v02.sudops.com:8088/proxy/application_1412048707137_0001/
14/09/30 11:45:41 INFO mapreduce.Job: Running job: job_1412048707137_0001
14/09/30 11:45:48 INFO mapreduce.Job: Job job_1412048707137_0001 running in uber mode : false
14/09/30 11:45:48 INFO mapreduce.Job:  map 0% reduce 0%
14/09/30 11:46:07 INFO mapreduce.Job:  map 30% reduce 0%
14/09/30 11:46:09 INFO mapreduce.Job:  map 60% reduce 0%
14/09/30 11:46:15 INFO mapreduce.Job:  map 100% reduce 0%
14/09/30 11:46:16 INFO mapreduce.Job:  map 100% reduce 100%
14/09/30 11:46:16 INFO mapreduce.Job: Job job_1412048707137_0001 completed successfully
14/09/30 11:46:16 INFO mapreduce.Job: Counters: 43
     File System Counters
          FILE: Number of bytes read=446
          FILE: Number of bytes written=1691681
          FILE: Number of read operations=0
          FILE: Number of large read operations=0
          FILE: Number of write operations=0
          HDFS: Number of bytes read=5370
          HDFS: Number of bytes written=215
          HDFS: Number of read operations=83
          HDFS: Number of large read operations=0
          HDFS: Number of write operations=3
     Job Counters 
          Launched map tasks=20
          Launched reduce tasks=1
          Data-local map tasks=20
          Total time spent by all maps in occupied slots (ms)=411605
          Total time spent by all reduces in occupied slots (ms)=7356
     Map-Reduce Framework
          Map input records=20
          Map output records=40
          Map output bytes=360
          Map output materialized bytes=560
          Input split bytes=3010
          Combine input records=0
          Combine output records=0
          Reduce input groups=2
          Reduce shuffle bytes=560
          Reduce input records=40
          Reduce output records=0
          Spilled Records=80
          Shuffled Maps =20
          Failed Shuffles=0
          Merged Map outputs=20
          GC time elapsed (ms)=3893
          CPU time spent (ms)=5160
          Physical memory (bytes) snapshot=4045692928
          Virtual memory (bytes) snapshot=17784217600
          Total committed heap usage (bytes)=2726051840
     Shuffle Errors
          BAD_ID=0
          CONNECTION=0
          IO_ERROR=0
          WRONG_LENGTH=0
          WRONG_MAP=0
          WRONG_REDUCE=0
     File Input Format Counters 
          Bytes Read=2360
     File Output Format Counters 
          Bytes Written=97
Job Finished in 35.516 seconds
Estimated value of Pi is 3.12000000000000000000