1

我在部署模式下使用 Nutch2.0 抓取 URL：

  1. 我用 Ant 构建 Nutch
  2. 我将 Nutch-2.0.job 与 Hadoop 一起使用。

我运行了以下命令：

  1. sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.crawl.InjectorJob /user/bluesky/nutch/urls

  2. sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.crawl.GeneratorJob

  3. sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.fetcher.FetcherJob (batch -id)

但是当我运行最后一个命令时，它在日志中给了我错误：

2012-10-09 15:35:04,292 ERROR org.apache.nutch.fetcher.FetcherJob: Unexpected error for http://www.imdb.com/title/tt0327056/
java.lang.AbstractMethodError: org.apache.gora.persistency.ListGenericArray.size()I
        at org.apache.avro.generic.GenericDatumWriter.getArraySize(GenericDatumWriter.java:142)
        at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:128)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
        at org.apache.gora.avro.PersistentDatumWriter.writeRecord(PersistentDatumWriter.java:74)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.gora.util.IOUtils.serialize(IOUtils.java:170)
        at org.apache.gora.sql.store.SqlStore.setField(SqlStore.java:718)
        at org.apache.gora.sql.store.SqlStore.setObject(SqlStore.java:669)
        at org.apache.gora.sql.statement.MySqlInsertUpdateStatement.toStatement(MySqlInsertUpdateStatement.java:96)
        at org.apache.gora.sql.store.SqlStore.put(SqlStore.java:616)
        at org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:60)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:514)
        at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
        at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.output(FetcherReducer.java:668)
        at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:540)
2012-10-09 15:35:04,293 ERROR org.apache.nutch.fetcher.FetcherJob: fetcher throwable caught
java.lang.AbstractMethodError: org.apache.gora.persistency.ListGenericArray.size()I
        at org.apache.avro.generic.GenericDatumWriter.getArraySize(GenericDatumWriter.java:142)
        at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:128)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
        at org.apache.gora.avro.PersistentDatumWriter.writeRecord(PersistentDatumWriter.java:74)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.gora.util.IOUtils.serialize(IOUtils.java:170)
        at org.apache.gora.sql.store.SqlStore.setField(SqlStore.java:718)
        at org.apache.gora.sql.store.SqlStore.setObject(SqlStore.java:669)
        at org.apache.gora.sql.statement.MySqlInsertUpdateStatement.toStatement(MySqlInsertUpdateStatement.java:96)
        at org.apache.gora.sql.store.SqlStore.put(SqlStore.java:616)
        at org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:60)
4

0 回答 0