Hadoop Speculative Exetution 推測執行

Speculative Execution 推測執行, 在hadoop的各個節點中，如果因為負載不均衡的情況，造成某些繁忙或是因硬體故障發生效能問題的data node又接收到新的map reduce 需求，導致此需求執行變慢，然而hadoop不會自動診斷或是修復這些慢任務，而是透過Hadoop speculative execution的功能，會把緩慢的map作業(sub-task)丟到不繁忙的節點執行，且在執行完畢後，自動kill那些緩慢的map作業(sub-task)

Ref:

https://zhuanlan.zhihu.com/p/22348162

https://pdfs.semanticscholar.org/ae65/8fbe6f2defcbb449b3922207177466e75b98.pdf

Apache Hadoop namenode 手動恢復步驟

類比namenode崩潰，例如將name目錄的內容全部刪除，然後通過secondary namenode恢復namenode，抓圖實驗過程

Metadata Disk Failure

If the NameNode machine fails, manual intervention is necessary. Currently, automatic restart and failover of the NameNode software to another machine is not supported

Namenode主要有底下兩個檔案組成, fsimage 為存放checkpoint的namespace , edits為異動的紀錄，可用來恢復namenode, 說明如下:

NameNode persists its namespace using two files:

fsimage, which is the latest checkpoint of the namespace and

edits, a journal (log) of changes to the namespace since the checkpoint.

如果NameNode節點掛了，可以按照如下步驟來從Secondary NameNode來恢復：

觀察:

fs.checkpoint.dir :在secondary namenode, 在namenode皆有檔案

dfs.data.dir , 在namenode上面, secondary namenode會是空的

dfs.name.dir : 存放在namenode, 在secondary namenode上面會是空的

dfs.name.edits.dir: 存放在namenode, 在secondary namenode上面會是空的

結果發現以下兩個參數屬於namenode (nn) :

dfs.name.dir

dfs.name.edits.dir

兩個參數屬於secondary namenode (snn) :

fs.checkpoint.edits.dir

fs.checkpoint.dir

實際測試步驟如下:

hadoop-daemon.sh stop namenode hadoop-daemon.sh stop secondarynamenode 模擬數據全毀 rm -rf /opt/HDP/meta/* 建立資料夾回namenode mkdir -p /home/hadoop/tmp /opt/HDP/meta /opt/HDP/meta/edits /opt/HDP/ckpoint/image/edits ssh dn1 chmod 755 /opt/HDP/data ssh dn2 chmod 755 /opt/HDP/data @把snn的checkpoint dir 複製回nn scp -pr nn2:/opt/HDP/ckpoint/* /opt/HDP/ckpoint/. mkdir –p /opt/HDP/ckpoint/image/edits 於$conf/core-site.xml暫時加入    fs.checkpoint.dir    /opt/HDP/ckpoint        Determines where on the local filesystem theDFS secondary name node should store the temporary images to merge. If this isa comma-delimited list of directories then the image is replicated in all of thedirectories for redundancy.

   fs.checkpoint.edits.dir

   /opt/HDP/ckpoint/image/edits

   

   

hadoop namenode -importCheckpoint

13/10/29 22:55:15 INFO ipc.Server: IPC Server listener on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 1 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 2 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 4 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 5 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 6 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 7 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 8 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 3 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 0 on 8020: starting

13/10/29 22:55:15 INFO ipc.Server: IPC Server handler 9 on 8020: starting

13/10/29 23:21:52 INFO logs: Aliases are enabled

13/10/29 23:22:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled. Name node is in safe mode.

The reported blocks is only 0 but the threshold is 0.9500 and the total blocks 3. Safe mode will be turned off automatically.

13/10/29 23:22:10 INFO ipc.Server: IPC Server handler 2 on 8020, call rollEditLog() from 192.168.35.131:45427: error: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled. Name node is in safe mode.

The reported blocks is only 0 but the threshold is 0.9500 and the total blocks 3. Safe mode will be turned off automatically.

[Ctrl + C]  è  

hadoop-daemon.sh start namenode

@another session, browse to http://nn1:50070

[hadoop@nn1 meta]$ hadoop-daemon.sh start namenode

[hadoop@nn2 meta]$ hadoop-daemon.sh start secondarynamenode

[hadoop@nn1 meta]$ jps

10104 Jps

9786 NameNode

9964 JobTracker

[hadoop@nn1 meta]$ hadoop dfs -ls testjay

Found 1 items

-rw-r--r--   2 hadoop supergroup         16 2013-10-29 22:14 /user/hadoop/testjay/test128m

[hadoop@nn1 meta]$ hadoop dfs -cat testjay/test128m

12312dsofjosdjf

[hadoop@nn1 meta]$

[hadoop@nn1 meta]$ hadoop dfs -cat testjay/test128m

12312dsofjosdjf

[hadoop@nn1 meta]$ cd

[hadoop@nn1 ~]$ dd if=/dev/zero of=testfile bs=10240k count=1

1+0 records in

1+0 records out

10485760 bytes (10 MB) copied, 0.011087 seconds, 946 MB/s

[hadoop@nn1 ~]$ hadoop fs -put testfile testjay/testfile

tail -f hadoop-hadoop-secondarynamenode-nn2.log

eNode: Posted URL nn1:50070putimage=1&port=50090&machine=nn2&token=-41:62854137:0:1383058879000:1383058699786&newChecksum=319867456f25e3b20d3553248b09788b 2013-10-29 23:01:20,205 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://nn1:50070/getimage?putimage=1&port=50090&machine=nn2&token=-41:62854137:0:1383058879000:1383058699786&newChecksum=319867456f25e3b20d3553248b09788b 2013-10-29 23:01:20,262 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done. New Image Size: 989

[hadoop@nn1 ~]$ hadoop fs -rm testjay/testfile

Moved to trash: hdfs://nn1:8020/user/hadoop/testjay/testfile

tail -f hadoop-hadoop-secondarynamenode-nn2.log

2013-10-29 23:04:20,298 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 0 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 2013-10-29 23:04:20,299 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: closing edit log: position=4, editlog=/opt/HDP/ckpoint/image/edits/current/edits 2013-10-29 23:04:20,299 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: close success: truncate to 4, editlog=/opt/HDP/ckpoint/image/edits/current/edits 2013-10-29 23:04:20,330 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://nn1:50070/getimage?getimage=1 2013-10-29 23:04:20,339 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Downloaded file fsimage size 989 bytes. 2013-10-29 23:04:20,339 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://nn1:50070/getimage?getedit=1 2013-10-29 23:04:20,344 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Downloaded file edits size 976 bytes. 2013-10-29 23:04:20,344 INFO org.apache.hadoop.hdfs.util.GSet: Computing capacity for map BlocksMap 2013-10-29 23:04:20,344 INFO org.apache.hadoop.hdfs.util.GSet: VM type       = 64-bit 2013-10-29 23:04:20,345 INFO org.apache.hadoop.hdfs.util.GSet: 2.0% max memory = 1013645312 2013-10-29 23:04:20,345 INFO org.apache.hadoop.hdfs.util.GSet: capacity      = 2^21 = 2097152 entries 2013-10-29 23:04:20,345 INFO org.apache.hadoop.hdfs.util.GSet: recommended=2097152, actual=2097152 2013-10-29 23:04:20,575 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop 2013-10-29 23:04:20,576 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup 2013-10-29 23:04:20,576 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true 2013-10-29 23:04:20,577 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100 2013-10-29 23:04:20,577 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 2013-10-29 23:04:20,577 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: dfs.namenode.edits.toleration.length = -1 2013-10-29 23:04:20,578 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times 2013-10-29 23:04:20,581 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 11 2013-10-29 23:04:20,594 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0 2013-10-29 23:04:20,594 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Start loading edits file /opt/HDP/ckpoint/image/edits/current/edits 2013-10-29 23:04:20,606 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: EOF of /opt/HDP/ckpoint/image/edits/current/edits, reached end of edit log Number of transactions found: 10.  Bytes read: 976 2013-10-29 23:04:20,606 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Edits file /opt/HDP/ckpoint/image/edits/current/edits of size 976 edits # 10 loaded in 0 seconds. 2013-10-29 23:04:20,607 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 0 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 2013-10-29 23:04:20,612 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: closing edit log: position=976, editlog=/opt/HDP/ckpoint/image/edits/current/edits 2013-10-29 23:04:20,612 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: close success: truncate to 976, editlog=/opt/HDP/ckpoint/image/edits/current/edits 2013-10-29 23:04:20,616 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file /opt/HDP/ckpoint/current/fsimage of size 1625 bytes saved in 0 seconds. 2013-10-29 23:04:20,712 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: closing edit log: position=4, editlog=/opt/HDP/ckpoint/image/edits/current/edits 2013-10-29 23:04:20,713 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: close success: truncate to 4, editlog=/opt/HDP/ckpoint/image/edits/current/edits 2013-10-29 23:04:20,728 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL nn1:50070putimage=1&port=50090&machine=nn2&token=-41:62854137:0:1383059060000:1383058880239&newChecksum=25aedb3e4690896fc583ba7bab176cd7 2013-10-29 23:04:20,728 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://nn1:50070/getimage?putimage=1&port=50090&machine=nn2&token=-41:62854137:0:1383059060000:1383058880239&newChecksum=25aedb3e4690896fc583ba7bab176cd7 2013-10-29 23:04:20,782 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done. New Image Size: 1625

以上恢復namenode心得, datanode不須重新啟動, 僅須停止nn, snn即可。

估計直接由snn恢復也是類似做法, 但須修改以下參數(原有的nn需改成snn的ip位置) :

fs.default.name

dfs.http.address

dfs.https.address

mapred.job.tracker

mapred.job.tracker.http.address

且調整以下參數到新的snn

dfs.secondary.http.address

$conf/masters 的內容也要改成新的snn

將Hadoop 1.2.1 安裝到單台standalone的方法 (Pseudo-Distributed Mode)

前言

一般而言Hadoop會需要5個node, 其中有

一個namenode,

一個secondary namenode,

三個data node

為了lab節省資源, 把namenode, datanode放到同一台, replica設定為1

VM環境為: Redhat 5.6, x64bit, RAM : 1GB

1.Download jdk 1.6 (jdk-6u45-linux-x64.bin)  (from http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html )

  chmod +x jdk-6u45-linux-x64.bin

  ./jdk-6u45-linux-x64.bin

  將解開的資料夾jdk1.6.0_45放到/usr底下

2. Download Hadoop 1.2.1版本 :  hadoop-1.2.1-bin.tar.gz  (from http://www.apache.org/dyn/closer.cgi/hadoop/common/ )

  #把檔案hadoop-1.2.1-bin.tar.gz解壓縮, 放到/opt底下

  tar -xzvf hadoop-1.2.1-bin.tar.gz

  mv hadoop-1.2.1 /opt/.

3.修改必要檔案

  cd /opt/hadoop-1.2.1/conf

vi hadoop-env.sh

# The java implementation to use.  Required.

# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

export JAVA_HOME=/usr/jdk1.6.0_45

vi core-site.xml

=========================

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

=========================

vi hdfs-site.xml

=========================

<configuration>

<property>

<name>dfs.data.dir</name>

<value>/usr/hadoop_1.2.1/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

=========================

vi mapred-site.xml

=========================

<configuration>

<property>

<name>mapred.job.tracker </name>

<value>localhost:9001</value>

</property>

</configuration>

=========================

[root@h1 bin]# ./start-all.sh

starting namenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-namenode-h1.out

localhost: starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-h1.out

localhost: starting secondarynamenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-h1.out

starting jobtracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-jobtracker-h1.out

localhost: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-h1.out

#啟動後發現無法列出HDFS資料

[root@h1 bin]# ./hadoop dfs -ls

13/09/26 13:30:46 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)\

#遵照網路上方法, 需先將namenode format

[root@h1 bin]# ./hadoop namenode -format

13/09/26 13:32:15 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = h1/192.168.35.129

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.2.1

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG:   java = 1.6.0_45

************************************************************/

13/09/26 13:32:15 INFO util.GSet: Computing capacity for map BlocksMap

13/09/26 13:32:15 INFO util.GSet: VM type       = 64-bit

13/09/26 13:32:15 INFO util.GSet: 2.0% max memory = 1013645312

13/09/26 13:32:15 INFO util.GSet: capacity      = 2^21 = 2097152 entries

13/09/26 13:32:15 INFO util.GSet: recommended=2097152, actual=2097152

13/09/26 13:32:16 INFO namenode.FSNamesystem: fsOwner=root

13/09/26 13:32:17 INFO namenode.FSNamesystem: supergroup=supergroup

13/09/26 13:32:17 INFO namenode.FSNamesystem: isPermissionEnabled=true

13/09/26 13:32:17 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

13/09/26 13:32:17 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

13/09/26 13:32:17 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0

13/09/26 13:32:17 INFO namenode.NameNode: Caching file names occuring more than 10 times

13/09/26 13:32:17 INFO common.Storage: Image file /tmp/hadoop-root/dfs/name/current/fsimage of size 110 bytes saved in 0 seconds.

13/09/26 13:32:18 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/tmp/hadoop-root/dfs/name/current/edits

13/09/26 13:32:18 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/tmp/hadoop-root/dfs/name/current/edits

13/09/26 13:32:18 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.

13/09/26 13:32:18 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at h1/192.168.35.129

************************************************************/

[root@h1 bin]#

[root@h1 bin]# ./stop-all.sh

stopping jobtracker

localhost: stopping tasktracker

no namenode to stop

localhost: stopping datanode

localhost: stopping secondarynamenode

[root@h1 bin]#

[root@h1 bin]# ./start-all.sh

#列出所有Java process

[root@h1 tmp]# /usr/jdk1.6.0_45/bin/jps

4960 SecondaryNameNode

4840 DataNode

5039 JobTracker

6334 Jps

5174 TaskTracker

4717 NameNode

[root@h1 tmp]#

#於HDFS建立資料夾

[root@h1 tmp]# hadoop fs -mkdir testjay

[root@h1 tmp]# hadoop dfs -ls

Found 1 items

drwxr-xr-x   - root supergroup          0 2013-09-26 13:46 /user/root/testjay

[root@h1 tmp]#

#於Linux建出測試資料

[root@h1 ~]# cd ~

[root@h1 ~]# mkdir input

[root@h1 ~]# cd input

[root@h1 ~]# echo "hello world" > test1.txt

[root@h1 ~]# echo "hello hadoop" > test2.txt

#把linux資料放到HDFS

[root@h1 ~]# hadoop dfs -put ./input in

[root@h1 ~]# hadoop dfs -ls ./in/*

-rw-r--r--   1 root supergroup         12 2013-09-26 13:58 /user/root/in/test1.txt

-rw-r--r--   1 root supergroup         13 2013-09-26 13:58 /user/root/in/test2.txt

#列出資料, 成功

[root@h1 ~]# hadoop dfs -cat ./in/test1.txt

hello world

[root@h1 ~]# hadoop dfs -cat ./in/test2.txt

hello hadoop

[root@h1 ~]#

#檢查jobtracker 狀態

#Block資料都寫到了哪?
[root@h1 current]# ls -ltrh
total 32K
-rw-r--r-- 1 root root 158 Sep 26 13:34 VERSION
-rw-r--r-- 1 root root  11 Sep 26 13:34 blk_-6668900420856905772_1022.meta
-rw-r--r-- 1 root root   4 Sep 26 13:34 blk_-6668900420856905772
-rw-r--r-- 1 root root  11 Sep 26 13:58 blk_8720904599102954439_1024.meta
-rw-r--r-- 1 root root  13 Sep 26 13:58 blk_8720904599102954439
-rw-r--r-- 1 root root  11 Sep 26 13:58 blk_2440773453556412055_1023.meta
-rw-r--r-- 1 root root  12 Sep 26 13:58 blk_2440773453556412055
-rw-r--r-- 1 root root 289 Sep 26 14:00 dncp_block_verification.log.curr
[root@h1 current]# pwd
/usr/hadoop_1.2.1/data/current

我沒有做後續的測試, 估計其他元件包括HBase, Hive等等的都要另外安裝才對.

Hadoop存放的block size 建議/ 檔案大小建議

Ref:

http://stackoverflow.com/questions/10764493/namenode-file-quantity-limit

http://f.dataguru.cn/thread-174725-1-1.html

https://www.google.com.tw/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC4QFjAA&url=http%3A%2F%2Fforum.hadoop.tw%2Fviewtopic.php%3Ff%3D4%26t%3D36007&ei=O_8nUtGJBcnHlAXE34GwBQ&usg=AFQjCNE6Nh_SXYxYE7twoV3UkEji4Gn6_w&sig2=oyQVOlyBhfVttq6xWS7FtA

Q1:

Hadoop存放的block size為何建議不能太小, 至少64MB或128MB?

Block size如果太小, 容易有一個問題, 當data node重開機 或毀損時, 太多細小檔案, 會增加copy replica的時間, 就好比在linux裡面copy 大量小檔案時, 速度快不起來, 是一樣的道理。

Q2:

Hadoop存放的各個物件檔案, 不建議太小?

因為namenode在記憶體中存儲hdfs中的檔資訊。每個檔案、目錄或區塊（block）需要大約150Byte

如果HDFS 有一百萬個檔案或資料夾, 則共需要150 Byte * 1000000 / 1024/ 1024 = 143MB的記憶體空間, 建議粗估300MB(Name node : heap size)

而真正的速度影響是, 在執行map-reduce工作時, 需要不斷的去reference參照各個檔案, 造成Java MR process的時候, Name node與data node交換的檔案資訊變多, 大大的拖慢了parsing的速度。

另外一個考量, 

存放一百萬個檔案，每個都是 64 MB，那空間總共為 64 TB。如果平均每個是 64 KB，那只能儲存 64 GB。

[轉載] Bigtop 專案- 快速架設Apache Hadoop環境

Ref:

http://ithelp.ithome.com.tw/question/10111071?tag=ithome.article

Apache Bigtop 有做 Hodoop 的 RPM，不過只支援 64 位元。如果你想在幾分鐘內就裝好 Hodoop 開始試玩，可以直接用 Bigtop 的 RPM 安裝，10 步驟之內完成，含測試。步驟如下：
1. 安裝 Bigtop 的 Repository
wget -O /etc/yum.repos.d/bigtop.repo http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-repository/label=centos6/lastSuccessfulBuild/artifact/repo/bigtop.repo
2. 安裝 Hadoop 相關套件
yum install hadoop\* flume\* mahout\* oozie\* whirr\* hbase\* --nogpgcheck -y
3. 下載 jdk
wget http://download.oracle.com/otn-pub/java/jdk/6u29-b11/jdk-6u29-linux-x64-rpm.bin
4. 安裝 jdk
chmod 755 jdk-6u29-linux-x64-rpm.bin; ./jdk-6u29-linux-x64-rpm.bin
5. 設定環境參數
export JAVA_HOME=/usr/java/jdk1.6.0_29
6. 用 hdfs 身份初始化 hdfs
sudo -u hdfs hadoop namenode -format
7. 啟動相關服務
for i in hadoop-namenode hadoop-datanode hadoop-jobtracker hadoop-tasktracker ; do sudo service $i start ; done
8. 用 hdfs 身份測試建立兩個目錄
sudo -u hdfs hadoop fs -mkdir /user/$USER
sudo -u hdfs hadoop fs -chown $USER /user/$USER
9. 測試 hadoop 列出檔案系統
hadoop fs -lsr /
10. 測試 hadoop 做 map reduce
hadoop jar /usr/lib/hadoop/hadoop-examples.jar pi 10 1000