我使用单节点。
我在 sudo R 中安装了 rmr2 和 hdfs
我在 rstudio-server 中写了一些代码。
但是，它发生了错误。
我不知道怎么了。
谢谢阅读。如果有人帮助我，我会很感激你。
> library("rmr2", lib.loc="/usr/lib64/R/library")
Please review your hadoop settings. See help(hadoop.settings)
> library("rhdfs", lib.loc="/usr/lib64/R/library")
Loading required package: rJava
HADOOP_CMD=/home/knu/hadoop/hadoop-2.7.3/bin/hadoop
Be sure to run hdfs.init()
> hdfs.init()
17/04/12 21:39:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> detach("package:rmr2", unload=TRUE)
> library("rmr2", lib.loc="/usr/lib64/R/library")
Please review your hadoop settings. See help(hadoop.settings)
> small.ints <- to.dfs(1:10)
17/04/12 21:41:07 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
17/04/12 21:41:07 INFO compress.CodecPool: Got brand-new compressor [.deflate]
> from.dfs(small.ints)
17/04/12 21:41:25 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
17/04/12 21:41:25 INFO compress.CodecPool: Got brand-new decompressor [.deflate]
$key
NULL
$val
[1] 1 2 3 4 5 6 7 8 9 10
> result <- mapreduce(input = small.ints,
map = function(k,v) cbind(v,v^2) )
执行日志：
packageJobJar: [/tmp/hadoop-unjar5646463829062726981/] [] /tmp/streamjob3604004992263150530.jar tmpDir=null
17/04/12 21:41:38 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
17/04/12 21:41:38 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
17/04/12 21:41:39 INFO mapred.FileInputFormat: Total input paths to process : 1
17/04/12 21:41:39 INFO mapreduce.JobSubmitter: number of splits:2
17/04/12 21:41:39 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
17/04/12 21:41:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1491995822063_0001
17/04/12 21:41:40 INFO impl.YarnClientImpl: Submitted application application_1491995822063_0001
17/04/12 21:41:40 INFO mapreduce.Job: The url to track the job: http://0.0.0.0:8089/proxy/application_1491995822063_0001/
17/04/12 21:41:40 INFO mapreduce.Job: Running job: job_1491995822063_0001
17/04/12 21:41:54 INFO mapreduce.Job: Job job_1491995822063_0001 running in uber mode : false
17/04/12 21:41:54 INFO mapreduce.Job: map 0% reduce 0%
17/04/12 21:42:01 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000001_0, Status : FAILED
Container [pid=12055,containerID=container_1491995822063_0001_01_000003] is running beyond virtual memory limits. Current usage: 109.6 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1491995822063_0001_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 12073 12055 12055 12055 (java) 193 8 2153918464 27756 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_0 3
|- 12055 12053 12055 12055 (bash) 0 0 115847168 303 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_0 3 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000003/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000003/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
17/04/12 21:42:01 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000000_0, Status : FAILED
Container [pid=12054,containerID=container_1491995822063_0001_01_000002] is running beyond virtual memory limits. Current usage: 107.3 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1491995822063_0001_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 12072 12054 12054 12054 (java) 204 9 2154971136 27169 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_0 2
|- 12054 12052 12054 12054 (bash) 0 0 115847168 302 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_0 2 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000002/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000002/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
17/04/12 21:42:07 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000001_1, Status : FAILED
Container [pid=12143,containerID=container_1491995822063_0001_01_000004] is running beyond virtual memory limits. Current usage: 105.0 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1491995822063_0001_01_000004 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 12151 12143 12143 12143 (java) 177 5 2153918464 26571 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_1 4
|- 12143 12142 12143 12143 (bash) 0 0 115847168 303 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_1 4 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000004/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000004/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
17/04/12 21:42:10 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000000_1, Status : FAILED
Container [pid=12164,containerID=container_1491995822063_0001_01_000005] is running beyond virtual memory limits. Current usage: 130.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1491995822063_0001_01_000005 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 12172 12164 12164 12164 (java) 293 9 2175307776 32535 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_1 5
|- 12164 12163 12164 12164 (bash) 0 0 115847168 302 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_1 5 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000005/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000005/stderr
|- 12224 12172 12164 12164 (R) 0 0 116293632 469 /bin/sh /usr/lib64/R/bin/R --slave --no-restore --vanilla --file=./rmr-streaming-map2d5212712621
|- 12228 12224 12164 12164 (R) 0 0 116293632 158 /bin/sh /usr/lib64/R/bin/R --slave --no-restore --vanilla --file=./rmr-streaming-map2d5212712621
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
17/04/12 21:42:16 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000001_2, Status : FAILED
Container [pid=12254,containerID=container_1491995822063_0001_01_000007] is running beyond virtual memory limits. Current usage: 160.6 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1491995822063_0001_01_000007 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 12319 12262 12254 12254 (R) 10 2 268447744 8256 /usr/lib64/R/bin/exec/R --slave --no-restore --vanilla --file=./rmr-streaming-map2d5212712621
|- 12262 12254 12254 12254 (java) 295 9 2175307776 32549 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000007/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000007 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_2 7
|- 12254 12253 12254 12254 (bash) 0 0 115847168 302 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000007/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000007 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_2 7 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000007/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000007/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
17/04/12 21:42:16 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000000_2, Status : FAILED
Container [pid=12280,containerID=container_1491995822063_0001_01_000008] is running beyond virtual memory limits. Current usage: 92.1 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1491995822063_0001_01_000008 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 12280 12279 12280 12280 (bash) 0 0 115847168 303 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000008/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000008 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_2 8 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000008/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000008/stderr
|- 12289 12280 12280 12280 (java) 137 5 2142965760 23280 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000008/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000008 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_2 8
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
17/04/12 21:42:23 INFO mapreduce.Job: map 100% reduce 0%
17/04/12 21:42:23 INFO mapreduce.Job: Job job_1491995822063_0001 failed with state FAILED due to: Task failed task_1491995822063_0001_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0
17/04/12 21:42:23 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=7
Killed map tasks=1
Launched map tasks=8
Other local map tasks=6
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=39157
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=39157
Total vcore-milliseconds taken by all map tasks=39157
Total megabyte-milliseconds taken by all map tasks=40096768
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
17/04/12 21:42:23 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, :
hadoop streaming failed with error code 1