当我使用 HSQLDB 运行 apache Nutch 时，我得到 ClassNotFoundException : org/apache/avro/ipc/ByteBufferOutputStream 尽管我在 lib 下拥有所有与 avro 相关的 jar 文件
avro-1.7.6.jar
avro-compiler-1.7.6.jar
avro-ipc-1.7.6.jar
avro-mapred-1.7.6.jar
这就是我所做的：
启动并运行 HSQLDB
root@elephant hsqldb# sudo java -cp /home/hsqldb/hsqldb-2.3.3/hsqldb/lib/hsqldb.jar org.hsqldb.server.Server --props /home/hsqldb/hsqldb-2.3.3/hsqldb/conf/server.properties [Server@372f7a8d]: [Thread[main,5,main]]: checkRunning(false) entered [Server@372f7a8d]: [Thread[main,5,main]]: checkRunning(false) exited [Server@372f7a8d]: Startup sequence initiated from main() method [Server@372f7a8d]: Loaded properties from [/home/hsqldb/hsqldb-2.3.3/hsqldb/conf/server.properties] [Server@372f7a8d]: Initiating startup sequence... [Server@372f7a8d]: Server socket opened successfully in 28 ms. [Server@372f7a8d]: Database [index=0, id=0, db=file:/home/hsqldb/hsqldb-2.3.3/hsqldb/data/nutch, alias=nutchdb] opened sucessfully in 1406 ms. [Server@372f7a8d]: Startup sequence completed in 1438 ms. [Server@372f7a8d]: 2015-12-26 18:30:13.841 HSQLDB server 2.3.3 is online on port 9001 [Server@372f7a8d]: To close normally, connect and execute SHUTDOWN SQL [Server@372f7a8d]: From command line, use [Ctrl]+[C] to abort abruptly配置常春藤/ivy.xml
ivy.xml 中的以下行未注释
<dependency org="org.apache.gora" name="gora-core" rev="0.5" conf="*->default"/>
和
<dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating"
conf="*->default" />
取消注释以下行 conf/gora.properites
###############################
# Default SqlStore properties #
###############################
gora.sqlstore.jdbc.driver=org.hsqldb.jdbc.JDBCDriver
gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchdb
gora.sqlstore.jdbc.user=sa
gora.sqlstore.jdbc.password=
跑蚂蚁构建
ant runtime为 nutch-site.xml 添加了配置
root@elephant conf# cat nutch-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>storage.data.store.class</name> <value>org.apache.gora.sql.store.SqlStore</value> </property> <property> <name>http.agent.name</name> <value>NutchCrawler</value> </property> <property> <name>http.robots.agents</name> <value>NutchCrawler,*</value> </property> </configuration>在 urls 文件夹下创建了 seed.txt
通过注入 url 执行 nutch
[root@elephant local]# bin/nutch inject urls/ InjectorJob: starting at 2015-12-26 19:11:24 InjectorJob: Injecting urlDir: urls Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/ipc/ByteBufferOutputStream at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:259) at org.apache.nutch.storage.StorageUtils.getDataStoreClass(StorageUtils.java:93) at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:77) at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:218) at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252) at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284) Caused by: java.lang.ClassNotFoundException: org.apache.avro.ipc.ByteBufferOutputStream at java.net.URLClassLoader$1.run(URLClassLoader.java:372) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:360) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 9 more