Set up Hadoop in pseudo distributed mode

Posted By : Md Qasim Siddiqui | 28-Dec-2014

Following my previous blog on 'Set up Hadoop on single node', This Blog will explain how to set up hadoop in pseudo-distributed mode

 

  • Configure the core-site.xml,mapred-site.xml and hdfs-site.xml  present in /opt/hadoop-2.5.1/etc/hadoop

  • core-site.xml

 

  	fs.default.name
 	 hdfs://localhost:9000
  	The name of the default file system. A URI whose
  	scheme and authority determine the FileSystem implementation.
  	
  • Hadoop’s default configurations use hadoop.tmp.dir as the base temporary directory both for the local file system and HDFS

  • Change hadoop.tmp.dir property to /app/hadoop/tmp in core-site.xml like this 

 

hadoop.tmp.dir
/app/hadoop/tmp
A base for other temporary directories.


  	fs.default.name
 	 hdfs://localhost:9000
  	The name of the default file system. A URI whose
  	scheme and authority determine the FileSystem implementation.
  	


 
  • mapred-site.xml.template 
 

	mapred.job.tracker
	hostname:8021


 
  • hdfs-site.xml
 

  	dfs.replication
  	1
  	Default block replication.
  	The actual number of replications can be specified when the file is created.
  	The default is used if replication is not specified in create time.
  	
  • Create hadoop tmp directory , sudo mkdir -p /app/hadoop/tmp

  • change ownership permission of tmp in /app/hadoop/tmp to user from root , tmp directory user permission should be same as hadoop directory

  • sudo chown user:user /app/hadoop/tmp

  • Now format the namenode , Type hdfs namenode -format

  • Note: If you forget to set the required ownerships and permissions, you will see a java.io.IOException when you try to format the name node

  • Now start the hadoop services , /opt/hadoop-2.5.1/sbin//start-dfs.sh and /opt/hadoop-2.5.1/sbin//start-yarn.sh

  • The hadoop daemon log output is written to the ${HADOOP_LOG_DIR} directory (defaults to ${HADOOP_HOME}/logs).

  • Type jps, to check all hadoop daemons started successfully

     12882 Jps

12320 SecondaryNameNode

12592 NodeManager

12106 DataNode

11978 NameNode

12468 ResourceManager

Congratz Ur Hadoop installed successfully!

 

About Author

Author Image
Md Qasim Siddiqui

Qasim is an experienced web app developer with expertise in groovy and grails,Hadoop , Hive, Mahout, AngularJS and Spring frameworks. He likes to listen music in idle time and plays counter strike.

Request for Proposal

Name is required

Comment is required

Sending message..