这是在此处的支持论坛中发布的一个未解决问题,但由于我没有得到任何回复,我想我应该尝试在这里提问。
我有一个使用 MongoDB 作为数据层的现有应用程序。目前我正在使用 Mongo 的 Map reduce 机制,但是我面临一些性能问题。所以我想到了使用 Hadoop 来实现这个逻辑。
我已经成功地运行了财政部收益示例,并考虑创建一个简单的项目来了解 mongo-hadoop 驱动程序。所以我创建了一个项目,在构建路径中插入了适当的 jar 文件并运行它。
这是我的java代码:
final Configuration conf = new Configuration();
MongoConfigUtil.setInputURI( conf, "mongodb://
username:passw...@192.168.1.198/locations" );
MongoConfigUtil.setOutputURI( conf, "mongodb://localhost/
test.out" );
System.out.println( "Conf: " + conf );
final Job job = new Job( conf, "word count" );
job.setJarByClass( WordCount.class );
job.setMapperClass( TokenizerMapper.class );
job.setCombinerClass( IntSumReducer.class );
job.setReducerClass( IntSumReducer.class );
job.setOutputKeyClass( Text.class );
job.setOutputValueClass( IntWritable.class );
job.setInputFormatClass( MongoInputFormat.class );
job.setOutputFormatClass( MongoOutputFormat.class );
System.exit( job.waitForCompletion( true ) ? 0 : 1 );"
但我收到此错误:
Conf: Configuration: core-default.xml, core-site.xml
12/05/20 14:12:03 WARN util.NativeCodeLoader: Unable to load native-
hadoop library for your platform... using builtin-java classes where
applicable
12/05/20 14:12:03 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
12/05/20 14:12:03 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or
JobConf#setJar(String).
12/05/20 14:12:03 INFO mapred.JobClient: Cleaning up the staging area
file:/tmp/hadoop-maximos/mapred/staging/maximos1261801897/.staging/
job_local_0001
Exception in thread "main" java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:
796)
at com.mongodb.DBApiLayer.doGetCollection(DBApiLayer.java:116)
at com.mongodb.DBApiLayer.doGetCollection(DBApiLayer.java:43)
at com.mongodb.DB.getCollection(DB.java:81)
at
com.mongodb.hadoop.util.MongoSplitter.calculateSplits(MongoSplitter.java:
51)
at
com.mongodb.hadoop.MongoInputFormat.getSplits(MongoInputFormat.java:
51)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:
962)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:
1093)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:
850)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at
com.mongodb.hadoop.examples.wordcount.WordCount.main(WordCount.java:
100)
我做错了什么?这是 Mongo、Hadoop 还是 Mongo-Hadoop 问题?