0

我正在尝试从文件系统中索引大量数据，数据大小只有几 TB。我将 FileListEntityProcessor 与 TikaEntityProcessor 一起使用。

数据配置.xml

<dataConfig>
    <dataSource name="bin" type="BinFileDataSource" />
    <document>
        <entity name="f" dataSource="null" rootEntity="true"
            processor="FileListEntityProcessor" transformer="TemplateTransformer"
            baseDir="//mathworks/devel/bat/A/logs/66048/"
            fileName=".*\.*" onError="skip" recursive="true">

            <field column="fileAbsolutePath" name="path" />
            <field column="fileSize" name="size"/>
            <field column="fileLastModified" name="lastmodified" />

            <entity name="file" dataSource="bin" processor="TikaEntityProcessor" url="${f.fileAbsolutePath}" format="text" onError="skip"
           rootEntity="true">
                <field column="text" name="text"/>   
            </entity>
        </entity>
    </document>
</dataConfig>

我用 -Xms1024m -Xmx10240m 声明码头服务器，尽管我的机器非常好，但我仍然不断收到 java 堆异常。我在索引数据库时遇到了同样的问题，但我的问题通过“batchSize=-1”解决了。我可以在这里做类似的事情吗？或者是否有任何东西可以从文件系统中索引 TB 的数据。

编辑：我正在尝试索引所有文本文件的日志文件。我没有看到任何导致该过程失败的特定文档。一个半小时后，它只显示异常。

INFO: [tika] webapp=null path=null params={event=newSearcher&q=static+newSearcher+warming+query+from+solrconfig.xml} hits=872 status=0 QTime=1
Oct 8, 2013 6:04:27 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Oct 8, 2013 6:04:27 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [tika] Registered new searcher Searcher@15cab4c0 main
Oct 8, 2013 6:04:27 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@67071180 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=5,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=10,evictions=0,size=10,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Oct 8, 2013 6:04:33 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {deleteByQuery=*:*,add=[(null), (null), (null), (null), (null), (null), (null), (null), ... (20157 adds)]} 0 4
Oct 8, 2013 6:04:33 PM org.apache.solr.common.SolrException log
SEVERE: Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:264)
        at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:375)
        at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:445)
        at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:426)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:621)
        at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:327)
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:225)
        ... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:759)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:619)
        ... 5 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
        at java.lang.AbstractStringBuilder.append(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.common.SolrInputField.toString(SolrInputField.java:200)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at java.util.AbstractMap.toString(Unknown Source)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.common.SolrInputDocument.toString(SolrInputDocument.java:182)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:68)
        at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:293)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:723)
        ... 6 more

Oct 8, 2013 6:04:33 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback
Oct 8, 2013 6:04:33 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback

预先感谢。普雷纳

4

0 回答 0