ada
[未知用户] 代码就是上文中您的示例mapreduce(input = '/user/rtest01', map = function(k, v) cbind(v, v^2)),错误提示是sh: -input: command not found
Error in mr(map = map, reduce = reduce, reduceondataframe = reduceondataframe, : hadoop streaming failed with error code 127.
另外,操作系统是centos 6.3,hadoop的版本是Hadoop 2.0.0-cdh4.1.2,以及jdk1.6.0_30。
bsspirit
[未知用户] input = ‘/user/rtest01′,能输出一下吗?
function(k, v) cbind(v, v^2)),是对数字做平方操作,不能处理文本内容。
请再次确认一下。
ada
[未知用户] rtest01的内容就是通过to.dfs(1:10,'/user/rtest01')写入的。函数from.dfs('/user/rtest01')的执行结果:
[[1]]
[[1]]$key
NULL
[[1]]$val
[1] 1
attr(,"rmr.keyval")
[1] TRUE
[[2]]
[[2]]$key
NULL
[[2]]$val
[1] 2
影
不好意思
想請問您一個問題
當我前置作業都已完成後
我想要安裝rhdfs R CMD INSTALL /root/R/rhdfs_1.0.5.tar.gz
但系統會出現下列警告
ERROR: dependency 'rJava' is not available for package 'rhdfs'
請問這是什麼問題呢??
麻煩您幫我解答一下
謝謝
bsspirit
[未知用户] rJava,你还没有安装或者安装失败,rhdfs对rJava有依赖。请按照操作步骤执行。
影
[未知用户] 您好:
我想我應該找到原因了
因為我的R的版本只有2.13.1
導致許多東西裝不上去
想請問一下我該如何更新R的版本呢??
bsspirit
[未知用户] 正文中有写:
“4.1 如果是Ubuntu 12.04,请更新源再下载R2.15.3版本”
其他的系统你要自己解决了。
影
[未知用户] 好的!!
我在嘗試看看
謝謝您的回答
dongguozhong
Job not successful. Error: # of failed Map Tasks exceeded allowed limit
这是什么问题啊 我ulimit -n 已经65536了,谢谢
bsspirit
[未知用户] 你这个错误是hadoop本身的,不是R语言引起的。
你同样写个JAVA程序应该也会有这个错误。
dongguozhong
[未知用户] Caused by: java.io.IOException: Cannot run program "Rscript": java.io.IOException: error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
有提示这个,是不是不能调用R
bsspirit
[未知用户] 你在执行什么操作的时候,出现这个错误呢? rmr2? rhdfs?
1. 先确认Java程序,是不是运行都正常。
2. 如果是rmr2,看本地环境的/usr/bin/R,/usr/bin/Rscript,是不是装好了,thrift是不是装好了?
dongguozhong
[未知用户] 1.JAVA程序运行正常
2.根据您文档前面所有运行都可以,执行wordcount的时候提示
packageJobJar: [/tmp/RtmpSbSXJC/rmr-local-env2c932a90e626, /tmp/RtmpSbSXJC/rmr-global-env2c937efce0c2, /tmp/RtmpSbSXJC/rmr-streaming-map2c9359dcb07d, /tmp/RtmpSbSXJC/rmr-streaming-reduce2c93f412d85, /tmp/RtmpSbSXJC/rmr-streaming-combine2c9312b57f78, /home/hadoop_tmp/hadoop-unjar3036903685334201444/] [] /tmp/streamjob2684192384705100214.jar tmpDir=null
13/04/28 10:07:05 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/04/28 10:07:05 WARN snappy.LoadSnappy: Snappy native library not loaded
13/04/28 10:07:05 INFO mapred.FileInputFormat: Total input paths to process : 1
13/04/28 10:07:08 INFO streaming.StreamJob: getLocalDirs(): [/home/hadoop_tmp/mapred/local]
13/04/28 10:07:08 INFO streaming.StreamJob: Running job: job_201304272135_0004
13/04/28 10:07:08 INFO streaming.StreamJob: To kill this job, run:
13/04/28 10:07:08 INFO streaming.StreamJob: /hadoop/conan/hadoop-1.0.4/libexec/../bin/hadoop job -Dmapred.job.tracker=172.16.19.241:9001 -kill job_201304272135_0004
13/04/28 10:07:08 INFO streaming.StreamJob: Tracking URL: http://namenode.local:50030/jobdetails.jsp?jobid=job_201304272135_0004
13/04/28 10:07:09 INFO streaming.StreamJob: map 0% reduce 0%
13/04/28 10:08:42 INFO streaming.StreamJob: map 100% reduce 100%
13/04/28 10:08:42 INFO streaming.StreamJob: To kill this job, run:
13/04/28 10:08:42 INFO streaming.StreamJob: /hadoop/conan/hadoop-1.0.4/libexec/../bin/hadoop job -Dmapred.job.tracker=172.16.19.241:9001 -kill job_201304272135_0004
13/04/28 10:08:42 INFO streaming.StreamJob: Tracking URL: http://namenode.local:50030/jobdetails.jsp?jobid=job_201304272135_0004
13/04/28 10:08:42 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201304272135_0004_m_000000
13/04/28 10:08:42 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, :
hadoop streaming failed with error code 1
Deleted hdfs://172.16.19.241:9000/tmp/RtmpSbSXJC/file2c9328dd6066
我的/usr/bin/R,/usr/bin/Rscript在namenode上都安装了,但是thrift根据你的文档里没有写,就没有安装,怎么安装
dongguozhong
还有本篇文章的实例需要安装第六步中的rhbase么
bsspirit
[未知用户] 本例不需要rhbase,rhbase部分请参考
http://cos.name/2013/04/rhadoop4-rhbase/
bsspirit
[未知用户] 这步操作thrift不用装。
你能把tracking日志都打印出来吗?找到准确的异常信息!
http://namenode.local:50030/jobdetails.jsp?jobid=job_201304272135_0004
这个异常只是告诉你map重试次数过多,看不出来问题。
ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1
dongguozhong
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: configuration exception
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
... 22 more
Caused by: java.io.IOException: Cannot run program "Rscript": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
... 23 more
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
这种情况是不是我datanode上的R环境没有安装好
dongguozhong
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: configuration exception
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
... 22 more
Caused by: java.io.IOException: Cannot run program "Rscript": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
... 23 more
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
这种情况 是我datanode上的R环境没有安装好么,datanode上所有的配置需要和namenode一致,是么
dongguozhong
[未知用户] 问题解决了,是datanode的问题,谢谢您耐心回答
vetri
hi ,
i am using rmr2 and rhdfs ,R-1.15.2, and cdh4
also set hadoop ,hadoop_cmd and hadoop_streaming environment variables.
please guide me urgents......please
Streaming command failed!
Error in rmr(map=map,reduce=reduce,combine=combine.....:
hadoop streaming error failed with error code 1