Setting up your own Hadoop cluster with Cloudera distro for EC2
Before you begin, figure out what distro you are on (if you don’t already know) by issuing this command from the shell:
For my set up, I used Ubuntu 8.04.3 LTS Hardy Heron.
Add Cloudera repositories for your distro to apt sources. Create a file called
/etc/apt/sources.list.d/cloudera.list and add the following two lines in it:
deb http://archive.cloudera.com/debian [distro]-stable contrib deb-src http://archive.cloudera.com/debian [distro]-stable contrib
Add repository GPG key to your keyring:
curl -s http://archive.cloudera.com/debian/archive.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install hadoop
simplejson for Python (I used
pip but you could use
easy_install as well):
sudo pip install boto simplejson
Setting Up Cloudera Client Scripts
Download the Cloudera client scripts:
The location of the scripts may change and the only link to the scripts can be found on this page:
Once downloaded, extract the archive’s contents to
/opt/cloudera. You could put it anywhere else you like.
Modify your environment variables and add the following to your
export AWS_ACCESS_KEY_ID=[your AWS access key ID] export AWS_SECRET_ACCESS_KEY=[your AWS secret access key] export PATH=$PATH:/opt/cloudera
The Cloudera client scripts need to have access to your AWS credentials from the system environment in order to work properly. Adding the path to where the the scripts reside will allow you to call the
hadoop-ec2 command that comes bundled with the package.
Source your .bash_profile to update your environment with the changes.
.hadoop-ec2 config directory in your home directory:
Create a file called
ec2-clusters.cfg in your
.hadoop-ec2 directory with the following contents:
[my-hadoop-cluster] ami=ami-6159bf08 instance_type=c1.medium key_name=[your EC2 key name] availability_zone=us-east-1b private_key=/path/to/keypair.pem ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
You could have multiple blocks for each of your clusters defined in this file. We just created on block for a cluster named
Test that the Cloudera client scripts are working:
If everything is configured properly up to this point, you should get output that says:
No running clusters
Launch your cluster (in this case with a master and one slave node):
hadoop-ec2 launch-cluster my-hadoop-cluster 1
Set up a proxy to the cluster:
hadoop-ec2 proxy my-hadoop-cluster
Setting Up Pig
The version of Pig that is a part of the Cloudera apt repositories was a little out of date for me so I decided to install a more updated version manually. The version of Hadoop that runs on the cluster is 0.18.3 and the latest version of Pig that is compatible with that Hadoop version is 0.4.0. You can download it from here:
If the link above doesn’t work, then try any of the mirrors for Pig listed on this page:
Extract the contents of the archive to
/opt/pig. Again, you could put it anywhere you desire.
Add the following to your
export HADOOP_CONF_DIR=~/.hadoop-ec2/my-hadoop-cluster export HADOOP_HOME=/usr/lib/hadoop export JAVA_HOME=/usr/lib/jvm/java-6-sun export PIG_CLASSPATH=/opt/pig/pig-0.4.0-core.jar:$HADOOP_CONF_DIR export PIG_HADOOP_VERSION=0.18.3
HADOOP_CONF_DIR environment variable should point to the location of your cluster’s
hadoop-site.xml file. If you are going to be running multiple clusters, you may want to create some shell scripts that set the appropriate environment variables for each cluster before running any commands like Pig, which depend on those values.
Don’t forget to add the path to the Pig
bin directory to your path as well:
Source your .bash_profile to update your environment with the changes and then run the pig command:
It should launch Pig in
mapreduce mode by default. In this mode, the
grunt shell will be connected to your EC2 Hadoop cluster vs. your local Hadoop installation. You should see something like the following:
2009-11-07 15:02:15,930 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://ec2-xx-xxx-xx-xx.compute-1.amazonaws.com:8020/ 2009-11-07 15:02:16,653 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: ec2-xx-xxx-xx-xx.compute-1.amazonaws.com:8021
Congratulations, you now have a fully functioning Hadoop cluster running on EC2 with Pig ready to do your bidding.