hadoop的三种方式-白红宇

hadoop的三种方式

阅读量：5228 次

发布时间：2019-06-14

本文共 22736 字，大约阅读时间需要 75 分钟。

时间同步：date

火墙：iptables

解析：hosts

主机：192.168.2.149

节点：192.168.2.150

192.168.2.125

192.168.2.126

rsync和ssh

网站：http://hadoop.apache.org/

有三种方式：单个节点

尾部式（用于测试）

完全分布式

～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～

1.基本安装及单个节点

(1)下载 lftp i

get hadoop-1.2.1.tar.gz jdk-6u32-linux-x64.bin

(2)java：*sh jdk-6u32-linux-x64.bin

结果：Java(TM) SE Development Kit 6 successfully installed.

Product Registration is FREE and includes many benefits:

* Notification of new versions, patches, and updates

* Special offers on Oracle products, services and training

* Access to early releases and documentation

Product and system data will be collected. If your configuration

supports a browser, the JDK Product Registration form will

be presented. If you do not register, none of this information

will be saved. You may also register your JDK later by

opening the register.html file (located in the JDK installation

directory) in a browser.

For more information on what data Registration collects and

how it is managed and used, see:

http://java.sun.com/javase/registration/JDKRegistrationPrivacy.html

Press Enter to continue.....

Done.

查看：ls

结果：hadoop-1.2.1.tar.gz jdk1.6.0_32 （多了此目录） jdk-6u32-linux-x64.bin

安装：tar zfx hadoop-1.2.1.tar.gz

移动：mv jdk1.6.0_32/ hadoop-1.2.1/jdk （为了方便一次性端走服务）

链接：ln -s hadoop-1.2.1 hadoop （为了方便以后的更新）

（3）配置 vim hadoop/conf/hadoop-env.sh

内容：export JAVA_HOME=/root/hadoop/jdk（ 9行 java所在的目录）

（4）测试

1.输入 *mkdir hadoop/input

*cp /root/hadoop/conf/*.xml /root/hadoop/input/

*ls /root/hadoop/input/

结果：capacity-scheduler.xml fair-scheduler.xml hdfs-site.xml mapred-site.xml

core-site.xml hadoop-policy.xml mapred-queue-acls.xml

2.输出 *cd /root/hadoop

*bin/hadoop jar hadoop-examples-1.2.1.jar （查看所支持的功能）

结果：An example program must be given as the first argument.

Valid program names are:

aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.

aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.

dbcount: An example job that count the pageview counts from a database.

grep: A map/reduce program that counts the matches of a regex in the input. （过滤）

join: A job that effects a join over sorted, equally partitioned datasets

multifilewc: A job that counts words from several files.

pentomino: A map/reduce tile laying program to find solutions to pentomino problems.

pi: A map/reduce program that estimates Pi using monte-carlo method.

randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.

randomwriter: A map/reduce program that writes 10GB of random data per node.

secondarysort: An example defining a secondary sort to the reduce.

sleep: A job that sleeps at each map and reduce task.

sort: A map/reduce program that sorts the data written by the random writer.

sudoku: A sudoku solver.

teragen: Generate data for the terasort

terasort: Run the terasort

teravalidate: Checking results of terasort

wordcount: A map/reduce program that counts the words in the input files.

*bin/hadoop jar hadoop-examples-1.2.1.jar grep input/ output 'dfs[a-z.]+' （把input的有dfs开头的过滤到output里，自动建立output目录）

结果：14/08/05 09:50:28 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0

14/08/05 09:50:28 INFO mapred.JobClient: Map output records=1

*ls /root/hadoop/output/

结果：part-00000 _SUCCESS

～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～

2.尾分布式文件系统

/root/hadoop/bin/stop-all.sh

(1)ssh （实现无密钥验证）

*ssh-keygen （获取密钥，空格即可）

结果：Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

07:f2:66:76:2b:59:76:29:c9:d9:b1:50:0f:5a:e9:2d root@server149.example.com

The key's randomart image is:

+--[ RSA 2048]----+

| +. |

| +.o |

| . . o.... |

| o o =E+. |

| S X =. |

| + * + |

| o . |

| . |

| |

+-----------------+

*ssh-copy-id server149.example.com

结果：The authenticity of host 'server149.example.com (192.168.2.149)' can't be established.

RSA key fingerprint is 66:9e:7c:f4:48:10:e3:20:59:1e:e9:44:35:32:42:14.

Are you sure you want to continue connecting (yes/no)? yes ***

Warning: Permanently added 'server149.example.com,192.168.2.149' (RSA) to the list of known hosts.

root@server149.example.com's password: （输入密码）

Now try logging into the machine, with "ssh 'server149.example.com'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

*ssh-copy-id localhost

结果：The authenticity of host 'localhost (::1)' can't be established.

RSA key fingerprint is 66:9e:7c:f4:48:10:e3:20:59:1e:e9:44:35:32:42:14.

Are you sure you want to continue connecting (yes/no)? yes ***

Warning: Permanently added 'localhost' (RSA) to the list of known hosts.

Now try logging into the machine, with "ssh 'localhost'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

测试：ssh server149.example.com

结果：Last login: Tue Aug 5 08:44:19 2014 from 192.168.2.1

离开：logout

结果：Connection to server149.example.com closed.

网站：http://hadoop.apache.org/docs/r1.2.1/single_node_setup.html

（2）配置文件 *vim /root/hadoop/conf/core-site.xml

内容：<configuration>

<name>fs.default.name</name>

<value>hdfs://server149.example.com:9000</value>

</property>

</configuration>

*vim /root/hadoop/conf/hdfs-site.xml

内容：<configuration>

<name>dfs.replication</name>

</property>

</configuration>

*vim /root/hadoop/conf/mapred-site.xml

内容：<configuration>

<name>mapred.job.tracker</name>

<value>server149.example.com:9001</value>

</property>

</configuration>

（3）解析 vim /etc/hosts

内容：192.168.2.149 server149.example.com

（4）格式化并开启服务

格式化： */root/hadoop/bin/hadoop namenode -format

结果：14/08/05 10:22:08 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = server149.example.com/192.168.2.149

STARTUP_MSG: args = [-format]

STARTUP_MSG: version = 1.2.1

STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG: java = 1.6.0_32

************************************************************/

14/08/05 10:22:08 INFO util.GSet: Computing capacity for map BlocksMap

14/08/05 10:22:08 INFO util.GSet: VM type = 64-bit

14/08/05 10:22:08 INFO util.GSet: 2.0% max memory = 1013645312

14/08/05 10:22:08 INFO util.GSet: capacity = 2^21 = 2097152 entries

14/08/05 10:22:08 INFO util.GSet: recommended=2097152, actual=2097152

14/08/05 10:22:08 INFO namenode.FSNamesystem: fsOwner=root

14/08/05 10:22:08 INFO namenode.FSNamesystem: supergroup=supergroup

14/08/05 10:22:08 INFO namenode.FSNamesystem: isPermissionEnabled=true

14/08/05 10:22:08 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

14/08/05 10:22:08 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

14/08/05 10:22:08 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0

14/08/05 10:22:08 INFO namenode.NameNode: Caching file names occuring more than 10 times

14/08/05 10:22:09 INFO common.Storage: Image file /tmp/hadoop-root/dfs/name/current/fsimage of size 110 bytes saved in 0 seconds.

14/08/05 10:22:09 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/tmp/hadoop-root/dfs/name/current/edits

14/08/05 10:22:09 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/tmp/hadoop-root/dfs/name/current/edits

14/08/05 10:22:09 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.

14/08/05 10:22:09 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at server149.example.com/192.168.2.149

************************************************************

*/root/hadoop/bin/start-all.sh

结果：namenode running as process 2504. Stop it first.

localhost: starting datanode, logging to /root/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-server149.example.com.out

localhost: starting secondarynamenode, logging to /root/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-server149.example.com.out

jobtracker running as process 2674. Stop it first.

localhost: starting tasktracker, logging to /root/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-server149.example.com.out

（5）查看 /root/hadoop/jdk/bin/jps

结果：3183 TaskTracker

2674 JobTracker

3035 SecondaryNameNode

2932 DataNode

2504 NameNode

3276 Jps

3.测试

（1）网页监控 http://192.168.2.149:50070/dfshealth.jsp

http://192.168.2.149:50030/jobtracker.jsp

（2）测试 rm -fr /root/hadoop/output/

*建目录：test

/root/hadoop/bin/hadoop fs -mkdir test

查看：/root/hadoop/bin/hadoop fs -ls

结果：Found 1 items

drwxr-xr-x - root supergroup 0 2014-08-05 11:13 /user/root/test

*上传：传到test目录里

/root/hadoop/bin/hadoop fs -put /root/hadoop/conf/*.xml test

查看：/root/hadoop/bin/hadoop fs -ls test

结果：Found 7 items

-rw-r--r-- 1 root supergroup 7457 2014-08-05 11:15 /user/root/test/capacity-scheduler.xml

-rw-r--r-- 1 root supergroup 348 2014-08-05 11:15 /user/root/test/core-site.xml

-rw-r--r-- 1 root supergroup 327 2014-08-05 11:15 /user/root/test/fair-scheduler.xml

-rw-r--r-- 1 root supergroup 4644 2014-08-05 11:15 /user/root/test/hadoop-policy.xml

-rw-r--r-- 1 root supergroup 316 2014-08-05 11:15 /user/root/test/hdfs-site.xml

-rw-r--r-- 1 root supergroup 2033 2014-08-05 11:15 /user/root/test/mapred-queue-acls.xml

-rw-r--r-- 1 root supergroup 344 2014-08-05 11:15 /user/root/test/mapred-site.xml

*输出：cd /root/hadoop

bin/hadoop jar hadoop-examples-1.2.1.jar grep test output 'dfs[a-z.]+'

结果：14/08/05 11:16:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library

14/08/05 11:16:57 WARN snappy.LoadSnappy: Snappy native library not loaded

14/08/05 11:16:57 INFO mapred.FileInputFormat: Total input paths to process : 7

14/08/05 11:16:58 INFO mapred.JobClient: Running job: job_201408051022_0001

14/08/05 11:16:59 INFO mapred.JobClient: map 0% reduce 0% ......

14/08/05 11:23:34 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1445437440

14/08/05 11:23:34 INFO mapred.JobClient: Map output records=2

*查看：/root/hadoop/bin/hadoop fs -ls

结果：Found 2 items

drwxr-xr-x - root supergroup 0 2014-08-05 11:23 /user/root/output

drwxr-xr-x - root supergroup 0 2014-08-05 11:15 /user/root/test

/root/hadoop/bin/hadoop fs -cat output/*

结果：1 dfs.replication

1 dfsadmin

cat: File does not exist: /user/root/output/_logs

*下载：/root/hadoop/bin/hadoop fs -get output test （下载到本地）

查看：ll -d /root/hadoop/test/

结果：drwxr-xr-x 3 root root 4096 Aug 5 11:27 /root/hadoop/test/

cat /root/hadoop/test/*

结果：cat: /root/hadoop/test/_logs: Is a directory

1 dfs.replication

1 dfsadmin

*删除：rm -fr /root/hadoop/test/

/root/hadoop/bin/hadoop fs -rmr output

结果：Deleted hdfs://server149.example.com:9000/user/root/output

查看：/root/hadoop/bin/hadoop fs -ls

结果：Found 1 items

drwxr-xr-x - root supergroup 0 2014-08-05 11:15 /user/root/test

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

master slives

HDFS namenode datanode

mp jobtracker tasktracter

mfs：数据存储

HDFS：存储与计算。

～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～～

3.完全式（服务分开，添加节点）

1.主机（ 149 ）

（1）停服务 */root/hadoop/bin/stop-all.sh

结果：stopping jobtracker

localhost: stopping tasktracker

stopping namenode

localhost: stopping datanode

localhost: stopping secondarynamenode

*/root/hadoop/jdk/bin/jps

结果：8675 Jps

（2）解析 vim /etc/hosts （都需要）

内容：192.168.2.125 server125.example.com

192.168.2.150 server150.example.com

192.168.2.149 server149.example.com

（3）ssh ssh-copy-id server150.example.com

ssh-copy-id server125.example.com

（4）配置文件vim /root/hadoop/conf/hdfs-site.xml

内容：<value>2</value> （ 9 ）

vim /root/hadoop/conf/masters

内容：server149.example.com

vim /root/hadoop/conf/slaves

内容：server150.example.com

server125.example.com

（5）复制给添加的节点（添加的节点主机做链接）

scp -r /root/hadoop-1.2.1 server150.example.com:

scp -r /root/hadoop-1.2.1 server125.example.com:

（6）格式化 /root/hadoop/bin/hadoop namenode -format