Hadoop Multi Node – Configurations

1> Link for installation

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

2> Prerequisites

1-Java 1.6 and higher version installed
2-Hadoop single node cluster

3> Setting up network

Change /etc/hosts on both master and slave machine to
ip of master
ip of slave

4> SSH access

Adding the Master’s SSH key into the Slaves authorized_keys
hadoop@master:~$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub hduser@slave_ip

5> Connecting Master to Master and Master to Slave

from master machine
ssh master_username@
ssh slave_username@

6> Configuration

-for MASTER only

edit conf/masters file and add
master

edit conf/slaves file and add
master
slave

-for both machines

fs.default.name
hdfs://<>:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri’s scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri’s authority is used to
determine the host, port, etc. for a filesystem.

mapred.job.tracker
<>:54311
The host and port that the MapReduce job tracker runs
at. If “local”, then jobs are run in-process as a single map
and reduce task.

dfs.replication
2
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.

7> Formatting the namenode

$HADOOP>bin/hadoop namenode -format

8> STarting Multi node clusters

on MASTER only

1-start HDFS
$HADOOP>bin/start-dfs.sh

check jps on both master and slave, and you will get

on master
14799 NameNode
15314 Jps
14880 DataNode
14977 SecondaryNameNode

on slave
15183 DataNode
15616 Jps

2-start mapreduce
$HADOOP>bin/start-mapred.sh

check jps on master and slave

on master
16017 Jps
14799 NameNode
15686 TaskTracker
14880 DataNode
15596 JobTracker
14977 SecondaryNameNode

on slave
15183 DataNode
15897 TaskTracker
16284 Jps

9> Stoping the cluster

on MASTER only

stop mapreduce first
$HADOOP> bin/stop-mapred.sh
then hdfs
$HADOOP> bin/stop-dfs.sh