Hadoop Multi Node – Configurations

1> Link for installation

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

2> Prerequisites

1-Java 1.6 and higher version installed
2-Hadoop single node cluster

3> Setting up network

Change /etc/hosts on both master and slave machine to
ip of master
ip of slave

4> SSH access

Adding the Master’s SSH key into the Slaves authorized_keys
hadoop@master:~$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub hduser@slave_ip

5> Connecting Master to Master and Master to Slave

from master machine
ssh master_username@
ssh slave_username@

6> Configuration

-for MASTER only

edit conf/masters file and add
master

edit conf/slaves file and add
master
slave

-for both machines

fs.default.name
hdfs://<>:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri’s scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri’s authority is used to
determine the host, port, etc. for a filesystem.

mapred.job.tracker
<>:54311
The host and port that the MapReduce job tracker runs
at. If “local”, then jobs are run in-process as a single map
and reduce task.

dfs.replication
2
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.

7> Formatting the namenode

$HADOOP>bin/hadoop namenode -format

8> STarting Multi node clusters

on MASTER only

1-start HDFS
$HADOOP>bin/start-dfs.sh

check jps on both master and slave, and you will get

on master
14799 NameNode
15314 Jps
14880 DataNode
14977 SecondaryNameNode

on slave
15183 DataNode
15616 Jps

2-start mapreduce
$HADOOP>bin/start-mapred.sh

check jps on master and slave

on master
16017 Jps
14799 NameNode
15686 TaskTracker
14880 DataNode
15596 JobTracker
14977 SecondaryNameNode

on slave
15183 DataNode
15897 TaskTracker
16284 Jps

9> Stoping the cluster

on MASTER only

stop mapreduce first
$HADOOP> bin/stop-mapred.sh
then hdfs
$HADOOP> bin/stop-dfs.sh

Advertisements

About ashokabhat

I am a C,C ,JAVA,Adobe Flex,.NET Programmer Currently working as a Software Developer
This entry was posted in Uncategorized. Bookmark the permalink.

3 Responses to Hadoop Multi Node – Configurations

  1. Every author knows, there may be a mechanical part to writing, or possibly a creative aspect.

  2. click here says:

    Terminal night I was nerve-racking to download the Felt hat put
    in DVD’s to research something… in a virtual simple machine, as I’m not moving out from Sabayon GNU/Linux, and all I olfactory perception, mussy ashes
    and cigarette butts. http://www.moy-p.co.uk/ In that location are brands that include 1
    supernumerary to obtain a prescription medicine for this medicine which moldiness be filled in Canada.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s