1

我在 Hbase 表中有一些爬网内容（通过 Nutch）。我已经写过处理一个表并通过 mapreduce 作业将其统计信息输出到一个新表中。以下是 MR 作业的代码片段。

NutchJob job = NutchJob.getInstance(getConf(), "customJob");

// === Map ===
DataStore<String, WebPage> pageStore = StorageUtils.createWebStore(
    job.getConfiguration(), String.class, WebPage.class);
Query<String, WebPage> query = pageStore.newQuery();
query.setFields(StorageUtils.toStringArray(FIELDS)); // Note: pages without
                                                     // these fields are
                                                     // skipped
LOG.info( "Table before mapper: " + job.getConfiguration().get(Nutch.CRAWL_ID_KEY ) );

GoraMapper.initMapperJob(job, pageStore, Text.class, WebPage.class,
        TableCopy.Mapper2.class, true);

job.setNumReduceTasks(1);


job.getConfiguration().set(Nutch.CRAWL_ID_KEY, "txt" );
LOG.info( "Table before reducer: " + job.getConfiguration().get(Nutch.CRAWL_ID_KEY ) );

DataStore<String, WebPage> hostStore = StorageUtils.createWebStore(
        job.getConfiguration(), String.class, WebPage.class);


GoraReducer.initReducerJob(job, hostStore, MarkerUpdateReducer2.class);

job.waitForCompletion(true);

在这种情况下，有两个表，一个在公共行给出，第二个是硬编码的（“txt”）。我的目的是用一些新的表名创建 reducer 数据存储，以便我可以在那里存储数据。但是发生的情况是在映射器中处理了“txt”表，并且由于该表中没有数据，因此请注意。以下是日志片段

019-10-15 15:38:05,007 WARN  util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-10-15 15:38:07,028 WARN  store.HBaseStore - Mismatching schema's names. Mappingfile schema: 'webpage'. PersistentClass schema's name: 'a_webpage'Assuming they are the same.
2019-10-15 15:38:07,647 INFO  marker.TableCopy - Table before mapper: a
2019-10-15 15:38:07,738 INFO  marker.TableCopy - Table before reducer: txt
2019-10-15 15:38:07,775 WARN  store.HBaseStore - Mismatching schema's names. Mappingfile schema: 'webpage'. PersistentClass schema's name: 'txt_webpage'Assuming they are the same.
2019-10-15 15:38:08,316 WARN  store.HBaseStore - Mismatching schema's names. Mappingfile schema: 'webpage'. PersistentClass schema's name: 'txt_webpage'Assuming they are the same.
2019-10-15 15:38:09,401 WARN  store.HBaseStore - Mismatching schema's names. Mappingfile schema: 'webpage'. PersistentClass schema's name: 'txt_webpage'Assuming they are the same.
2019-10-15 15:38:09,453 WARN  store.HBaseStore - Mismatching schema's names. Mappingfile schema: 'webpage'. PersistentClass schema's name: 'txt_webpage'Assuming they are the same.
2019-10-15 15:38:09,491 WARN  store.HBaseStore - Mismatching schema's names. Mappingfile schema: 'webpage'. PersistentClass schema's name: 'txt_webpage'Assuming they are the same.
2019-10-15 15:38:09,604 INFO  marker.TableCopy - map table: txt
2019-10-15 15:38:09,869 WARN  store.HBaseStore - Mismatching schema's names. Mappingfile schema: 'webpage'. PersistentClass schema's name: 'txt_webpage'Assuming they are the same.

我在设置方法中打印了表名。它显示了上面日志“map table: txt”中给出的文本。实际表格“a”

4

0 回答 0