2

我一直在尝试各种事情,但无济于事。我对 Nutch/Solr 的配置基于此:

http://ubuntuforums.org/showthread.php?t=1532230

现在我已经启动并运行了 Nutch 和 Solr,我想使用 Solr 来索引爬网数据。Nutch 成功爬取了我指定的域,但在我运行命令将该数据传送到 Solr 时失败。这是命令:

bin/nutch solrindex http://solr:8181/solr/ crawl/crawldb crawl/linkdb crawl/segments/*

这是输出:

Indexer: starting at 2013-09-12 10:34:43
Indexer: deleting gone documents: false
Indexer: URL filtering: false
Indexer: URL normalizing: false
Active IndexWriters :
SOLRIndexWriter
solr.server.url : URL of the SOLR instance (mandatory)
solr.commit.size : buffer size when sending to SOLR (default 1000)
solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
solr.auth : use authentication (default false)
solr.auth.username : use authentication (default false)
solr.auth : username for authentication
solr.auth.password : password for authentication


Indexer: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:             file:/usr/share/apache-nutch-1.7/crawl/linkdb/crawl_fetch
Input path does not exist: file:/usr/share/apache-nutch-1.7/crawl/linkdb/crawl_parse
Input path does not exist: file:/usr/share/apache-nutch-1.7/crawl/linkdb/parse_data
Input path does not exist: file:/usr/share/apache-nutch-1.7/crawl/linkdb/parse_text
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:40)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1081)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1073)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:123)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:185)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:195)

经过大量谷歌搜索后,我还尝试了另一个命令:

bin/nutch solrindex http://solr:8181/solr/ crawl/crawldb -linkdb crawl/linkdb crawl/segments/*

有了这个输出:

Indexer: starting at 2013-09-12 10:45:51
Indexer: deleting gone documents: false
Indexer: URL filtering: false
Indexer: URL normalizing: false
Active IndexWriters :
SOLRIndexWriter
solr.server.url : URL of the SOLR instance (mandatory)
solr.commit.size : buffer size when sending to SOLR (default 1000)
solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
solr.auth : use authentication (default false)
solr.auth.username : use authentication (default false)
solr.auth : username for authentication
solr.auth.password : password for authentication


Indexer: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:123)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:185)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:195)

有没有人对如何克服这些错误有任何想法?

4

2 回答 2

1

期待在新的 Solr 5.2.1 和 Nutch 1.10 上出现同样的错误:

2015-07-30 20:56:23,015 警告 mapred.LocalJobRunner - job_local_0001 org.apache.solr.common.SolrException:未找到

未找到

请求: http: //127.0.0.1 :8983/solr/update?wt=javabin&version=2

所以我创建了一个集合(或核心,我不是 SOLR 专家):

bin/solr 创建 -c 演示

并更改了 Nutch 索引脚本中的 URL:

bin/nutch solrindex http://127.0.0.1:8983/solr/demo crawl/crawldb -linkdb crawl/linkdb crawl/segments/*

我知道这个问题很老了,但也许我会帮助别人......

于 2015-07-30T18:28:15.903 回答
0

你有没有看到 solr 中显示错误原因的日志。我在 nutch 中遇到过同样的问题,solr 的日志显示一条​​消息“未知字段‘主机’”。在我为 solr 修改 schema.xml 后,问题就消失了。

于 2014-01-09T01:26:19.310 回答