Comments on: Notes on setting up hadoop on CentOS 5 x86 http://www.spicylogic.com/allenday/blog/2008/06/20/notes-on-setting-up-hadoop-on-centos-5-x86/ ♥data♥ Wed, 19 Nov 2008 11:47:45 +0000 http://wordpress.org/?v=2.5.1 By: yodafide http://www.spicylogic.com/allenday/blog/2008/06/20/notes-on-setting-up-hadoop-on-centos-5-x86/#comment-48 yodafide Thu, 07 Aug 2008 23:28:32 +0000 http://www.spicylogic.com/allenday/blog/?p=34#comment-48 Thanks allen. Say have you every tried putting your dfs on another drive when you were going through the pseudo-distibuted single-machine Hadoop tutorial? Thanks allen. Say have you every tried putting your dfs on another drive when you were going through the pseudo-distibuted single-machine Hadoop tutorial?

]]> By: allenday http://www.spicylogic.com/allenday/blog/2008/06/20/notes-on-setting-up-hadoop-on-centos-5-x86/#comment-46 allenday Sat, 12 Jul 2008 00:16:57 +0000 http://www.spicylogic.com/allenday/blog/?p=34#comment-46 Hi yodafide, It looks like these are windows-specific problems. My notes are only tested on Linux CentOS 5. Just after a glance though, it looks like you're having line terminator problems (the "\r" errors). The reason the ssh is passwordless is that the hadoop admin scripts ssh to each of the storage/exec nodes, and you would have to type a password for each host if you didn't configure the keys to be authorized. Hi yodafide,

It looks like these are windows-specific problems. My notes are only tested on Linux CentOS 5.

Just after a glance though, it looks like you’re having line terminator problems (the “\r” errors).

The reason the ssh is passwordless is that the hadoop admin scripts ssh to each of the storage/exec nodes, and you would have to type a password for each host if you didn’t configure the keys to be authorized.

]]> By: yodafide http://www.spicylogic.com/allenday/blog/2008/06/20/notes-on-setting-up-hadoop-on-centos-5-x86/#comment-45 yodafide Thu, 10 Jul 2008 21:40:31 +0000 http://www.spicylogic.com/allenday/blog/?p=34#comment-45 $ ssh localhost ssh: connect to host localhost port 22: Connection refused Also, why do we need passwordless ssh access? And what should I do if the alternate solution that the Quick Start tutorial offers does not work? $ ssh localhost
ssh: connect to host localhost port 22: Connection refused

Also, why do we need passwordless ssh access? And what should I do if the alternate solution that the Quick Start tutorial offers does not work?

]]> By: yodafide http://www.spicylogic.com/allenday/blog/2008/06/20/notes-on-setting-up-hadoop-on-centos-5-x86/#comment-44 yodafide Thu, 10 Jul 2008 21:17:23 +0000 http://www.spicylogic.com/allenday/blog/?p=34#comment-44 $ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 2: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 7: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 10: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 13: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 16: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 19: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 22: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 25: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 28: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 31: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 36: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 39: $'\r': command not found /cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co nf/hadoop-env.sh: line 42: $'\r': command not found 08/07/10 14:12:20 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName =JobTracker, sessionId= 08/07/10 14:12:20 INFO mapred.FileInputFormat: Total input paths to process : 2 08/07/10 14:12:21 INFO mapred.JobClient: Running job: job_local_1 08/07/10 14:12:21 INFO mapred.MapTask: numReduceTasks: 1 08/07/10 14:12:21 INFO mapred.LocalJobRunner: file:/c:/Documents and Settings/ys khoo/Desktop/hadoop/hadoop-0.16.4/input/hadoop-default.xml:0+34064 08/07/10 14:12:21 INFO mapred.TaskRunner: Task 'job_local_1_map_0000' done. 08/07/10 14:12:21 INFO mapred.TaskRunner: Saved output of task 'job_local_1_map_ 0000' to file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/gre p-temp-56794453 08/07/10 14:12:21 INFO mapred.MapTask: numReduceTasks: 1 08/07/10 14:12:21 INFO mapred.LocalJobRunner: file:/c:/Documents and Settings/ys khoo/Desktop/hadoop/hadoop-0.16.4/input/hadoop-site.xml:0+184 08/07/10 14:12:21 INFO mapred.TaskRunner: Task 'job_local_1_map_0001' done. 08/07/10 14:12:21 INFO mapred.TaskRunner: Saved output of task 'job_local_1_map_ 0001' to file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/gre p-temp-56794453 08/07/10 14:12:21 INFO mapred.LocalJobRunner: reduce > reduce 08/07/10 14:12:21 INFO mapred.TaskRunner: Task 'reduce_dtdqnd' done. 08/07/10 14:12:21 INFO mapred.TaskRunner: Saved output of task 'reduce_dtdqnd' t o file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/grep-temp- 56794453 08/07/10 14:12:22 INFO mapred.JobClient: Job complete: job_local_1 08/07/10 14:12:22 INFO mapred.JobClient: Counters: 9 08/07/10 14:12:22 INFO mapred.JobClient: Map-Reduce Framework 08/07/10 14:12:22 INFO mapred.JobClient: Map input records=1124 08/07/10 14:12:22 INFO mapred.JobClient: Map output records=39 08/07/10 14:12:22 INFO mapred.JobClient: Map input bytes=34248 08/07/10 14:12:22 INFO mapred.JobClient: Map output bytes=1114 08/07/10 14:12:22 INFO mapred.JobClient: Combine input records=39 08/07/10 14:12:22 INFO mapred.JobClient: Combine output records=38 08/07/10 14:12:22 INFO mapred.JobClient: Reduce input groups=38 08/07/10 14:12:22 INFO mapred.JobClient: Reduce input records=38 08/07/10 14:12:22 INFO mapred.JobClient: Reduce output records=38 08/07/10 14:12:22 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with proces sName=JobTracker, sessionId= - already initialized 08/07/10 14:12:22 INFO mapred.FileInputFormat: Total input paths to process : 1 08/07/10 14:12:22 INFO mapred.JobClient: Running job: job_local_2 08/07/10 14:12:22 INFO mapred.MapTask: numReduceTasks: 1 08/07/10 14:12:22 INFO mapred.LocalJobRunner: file:/c:/Documents and Settings/ys khoo/Desktop/hadoop/hadoop-0.16.4/grep-temp-56794453/part-00000:0+1491 08/07/10 14:12:22 INFO mapred.TaskRunner: Task 'job_local_2_map_0000' done. 08/07/10 14:12:22 INFO mapred.TaskRunner: Saved output of task 'job_local_2_map_ 0000' to file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/out put 08/07/10 14:12:22 INFO mapred.LocalJobRunner: reduce > reduce 08/07/10 14:12:22 INFO mapred.TaskRunner: Task 'reduce_iy0cmt' done. 08/07/10 14:12:22 INFO mapred.TaskRunner: Saved output of task 'reduce_iy0cmt' t o file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/output 08/07/10 14:12:23 INFO mapred.JobClient: Job complete: job_local_2 08/07/10 14:12:23 INFO mapred.JobClient: Counters: 9 08/07/10 14:12:23 INFO mapred.JobClient: Map-Reduce Framework 08/07/10 14:12:23 INFO mapred.JobClient: Map input records=38 08/07/10 14:12:23 INFO mapred.JobClient: Map output records=38 08/07/10 14:12:23 INFO mapred.JobClient: Map input bytes=1405 08/07/10 14:12:23 INFO mapred.JobClient: Map output bytes=1101 08/07/10 14:12:23 INFO mapred.JobClient: Combine input records=0 08/07/10 14:12:23 INFO mapred.JobClient: Combine output records=0 08/07/10 14:12:23 INFO mapred.JobClient: Reduce input groups=2 08/07/10 14:12:23 INFO mapred.JobClient: Reduce input records=38 08/07/10 14:12:23 INFO mapred.JobClient: Reduce output records=38 I don't quite get the errors at the very top. This is for the stand-alone operation of Hadoop. I did not find it as trivial, unfortunately. I'm setting this up on Windows btw. $ bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 2: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 7: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 10: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 13: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 16: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 19: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 22: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 25: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 28: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 31: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 36: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 39: $’\r’: command not found
/cygdrive/c/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/bin/../co
nf/hadoop-env.sh: line 42: $’\r’: command not found
08/07/10 14:12:20 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName
=JobTracker, sessionId=
08/07/10 14:12:20 INFO mapred.FileInputFormat: Total input paths to process : 2
08/07/10 14:12:21 INFO mapred.JobClient: Running job: job_local_1
08/07/10 14:12:21 INFO mapred.MapTask: numReduceTasks: 1
08/07/10 14:12:21 INFO mapred.LocalJobRunner: file:/c:/Documents and Settings/ys
khoo/Desktop/hadoop/hadoop-0.16.4/input/hadoop-default.xml:0+34064
08/07/10 14:12:21 INFO mapred.TaskRunner: Task ‘job_local_1_map_0000′ done.
08/07/10 14:12:21 INFO mapred.TaskRunner: Saved output of task ‘job_local_1_map_
0000′ to file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/gre
p-temp-56794453
08/07/10 14:12:21 INFO mapred.MapTask: numReduceTasks: 1
08/07/10 14:12:21 INFO mapred.LocalJobRunner: file:/c:/Documents and Settings/ys
khoo/Desktop/hadoop/hadoop-0.16.4/input/hadoop-site.xml:0+184
08/07/10 14:12:21 INFO mapred.TaskRunner: Task ‘job_local_1_map_0001′ done.
08/07/10 14:12:21 INFO mapred.TaskRunner: Saved output of task ‘job_local_1_map_
0001′ to file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/gre
p-temp-56794453
08/07/10 14:12:21 INFO mapred.LocalJobRunner: reduce > reduce
08/07/10 14:12:21 INFO mapred.TaskRunner: Task ‘reduce_dtdqnd’ done.
08/07/10 14:12:21 INFO mapred.TaskRunner: Saved output of task ‘reduce_dtdqnd’ t
o file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/grep-temp-
56794453
08/07/10 14:12:22 INFO mapred.JobClient: Job complete: job_local_1
08/07/10 14:12:22 INFO mapred.JobClient: Counters: 9
08/07/10 14:12:22 INFO mapred.JobClient: Map-Reduce Framework
08/07/10 14:12:22 INFO mapred.JobClient: Map input records=1124
08/07/10 14:12:22 INFO mapred.JobClient: Map output records=39
08/07/10 14:12:22 INFO mapred.JobClient: Map input bytes=34248
08/07/10 14:12:22 INFO mapred.JobClient: Map output bytes=1114
08/07/10 14:12:22 INFO mapred.JobClient: Combine input records=39
08/07/10 14:12:22 INFO mapred.JobClient: Combine output records=38
08/07/10 14:12:22 INFO mapred.JobClient: Reduce input groups=38
08/07/10 14:12:22 INFO mapred.JobClient: Reduce input records=38
08/07/10 14:12:22 INFO mapred.JobClient: Reduce output records=38
08/07/10 14:12:22 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with proces
sName=JobTracker, sessionId= - already initialized
08/07/10 14:12:22 INFO mapred.FileInputFormat: Total input paths to process : 1
08/07/10 14:12:22 INFO mapred.JobClient: Running job: job_local_2
08/07/10 14:12:22 INFO mapred.MapTask: numReduceTasks: 1
08/07/10 14:12:22 INFO mapred.LocalJobRunner: file:/c:/Documents and Settings/ys
khoo/Desktop/hadoop/hadoop-0.16.4/grep-temp-56794453/part-00000:0+1491
08/07/10 14:12:22 INFO mapred.TaskRunner: Task ‘job_local_2_map_0000′ done.
08/07/10 14:12:22 INFO mapred.TaskRunner: Saved output of task ‘job_local_2_map_
0000′ to file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/out
put
08/07/10 14:12:22 INFO mapred.LocalJobRunner: reduce > reduce
08/07/10 14:12:22 INFO mapred.TaskRunner: Task ‘reduce_iy0cmt’ done.
08/07/10 14:12:22 INFO mapred.TaskRunner: Saved output of task ‘reduce_iy0cmt’ t
o file:/c:/Documents and Settings/yskhoo/Desktop/hadoop/hadoop-0.16.4/output
08/07/10 14:12:23 INFO mapred.JobClient: Job complete: job_local_2
08/07/10 14:12:23 INFO mapred.JobClient: Counters: 9
08/07/10 14:12:23 INFO mapred.JobClient: Map-Reduce Framework
08/07/10 14:12:23 INFO mapred.JobClient: Map input records=38
08/07/10 14:12:23 INFO mapred.JobClient: Map output records=38
08/07/10 14:12:23 INFO mapred.JobClient: Map input bytes=1405
08/07/10 14:12:23 INFO mapred.JobClient: Map output bytes=1101
08/07/10 14:12:23 INFO mapred.JobClient: Combine input records=0
08/07/10 14:12:23 INFO mapred.JobClient: Combine output records=0
08/07/10 14:12:23 INFO mapred.JobClient: Reduce input groups=2
08/07/10 14:12:23 INFO mapred.JobClient: Reduce input records=38
08/07/10 14:12:23 INFO mapred.JobClient: Reduce output records=38

I don’t quite get the errors at the very top. This is for the stand-alone operation of Hadoop. I did not find it as trivial, unfortunately. I’m setting this up on Windows btw.

]]>