1

我需要你的帮助来解决一个大问题。我必须用 solr 索引来自 EMC xDB 源的一些数据。我将 xDB 配置为像 RDBMS 一样被请求。然后我将 Squirrel 配置为请求 xDB。它有效，例如我尝试：

 Select * from books

我使用与 squirrel 相同的驱动程序和相同的连接字符串配置了 solr。

这是我的 data-config.xml：

<dataConfig>
<dataSource type="JdbcDataSource" driver="com.emc.ia.xdbjdbc.Driver" url="jdbc:xdbeas://localhost:1235/MaDatabase" user="*****" password="*****"/>
<document name="books">
    <entity name="book" query="select * from books">
        <field column="author" name="author" />
        <field column="price" name="price" />
        <field column="name" name="name" />
        <field column="description" name="description" />
        <field column="title" name="title" />
        <field column="publish_date" name="publish_date" />         
    </entity>
</document>
</dataConfig>

solrconfig.xml

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
    <lst name="defaults">
        <str name="config">C:\solr\solr-5.2.1\server\solr\jcg\conf\data-config.xml</str>
    </lst>
</requestHandler>

我的 xDB BOOKS-MetaData.xml：

<?xml version="1.0" encoding="UTF-16"?>
<TABLEMETADATA TABLENAME="books">
  <RECORD-COUNT>2</RECORD-COUNT>
  <ROOT-ELEMENT>books</ROOT-ELEMENT>
  <ROW-ELEMENT>book</ROW-ELEMENT>
  <DATATYPES>
     <BOOKS-TYPE>
      <NAME>author</NAME>
      <COLNO>1</COLNO>
      <DATATYPE>VARCHAR2(75)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>title</NAME>
      <COLNO>2</COLNO>
      <DATATYPE>VARCHAR2(75)</DATATYPE>
    </BOOKS-TYPE>
    <BOOKS-TYPE>
      <NAME>genre</NAME>
      <COLNO>3</COLNO>
      <DATATYPE>VARCHAR2(25)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>price</NAME>
      <COLNO>4</COLNO>
      <DATATYPE>VARCHAR2(25)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>publish_date</NAME>
      <COLNO>5</COLNO>
      <DATATYPE>DATETIME</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>description</NAME>
      <COLNO>6</COLNO>
      <DATATYPE>VARCHAR2(350)</DATATYPE>
    </BOOKS-TYPE>
  </DATATYPES>
</TABLEMETADATA>

solr 响应的一部分：

"verbose-output": [
"entity:book",
[
  "document#1",
  [
    "query",
    "select * from books",
    "time-taken",
    "0:0:0.114",
    "EXCEPTION",
    "org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1

当我检查 solr 日志时，我有 3 个异常。1号：

getNext() failed for query 'select * from books':org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

2号：

Exception while processing: book document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

最后一个：

Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
... 29 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
... 31 more
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

提前感谢您的帮助，因为我真的不明白为什么它适用于松鼠而不适用于 solr。

问题编辑以回复@Cheffe

由于 getObject() 方法，这确实是一个 xDB 驱动程序问题。我查看了 squirrel 代码，它之所以有效，是因为它使用列的索引而不是它的名称。

int idx = _columnIndices != null ? _columnIndices[i] : i + 1;
        try
        {
            final int columnType = _rsmd.getColumnType(idx);
            //final String columnClassName = _rsmd.getColumnClassName(idx);
            switch (columnType)
            {
                case Types.NULL:
                    row[i] = null;
                    break;

                // TODO: When JDK1.4 is the earliest JDK supported
                // by Squirrel then remove the hardcoding of the
                // boolean data type.
                case Types.BIT:
                case 16:

// case Types.BOOLEAN: row[i] = _rs.getObject(idx); . . .

无论如何它适用于

convertType="true"
4

1 回答 1

1

检查 Solr 源代码后：该数据库的驱动程序似乎不适合使用 Solr 的 DIH 运行。不幸的是，该数据库是专有的，因此我无法尝试。

异常是从供应商的驱动程序中引发的。所有三个根本原因都存在于那里。当您向下滚动到堆栈跟踪的最底部时，您可以看到这一点。所有这些堆栈跟踪都告诉你

方法 xxx.getObject-columnLabel 尚未实现。

这意味着：您的数据库供应商 EMC 没有在他们的 JDBC 驱动程序中实现此方法。如果您不想弄脏自己的手，您应该向他们提交您需要实现此功能的错误报告或功能请求。

您可能有一个镜头是堆栈跟踪中记录 Solr 来源的最后一行导致内部getARow方法

private Map<String, Object> getARow() {
  if (resultSet == null)
    return null;

  Map<String, Object> result = new HashMap<>();
  for (String colName : colNames) {
    try {
      if (!convertType) {
      Object value = resultSet.getObject(colName); // <-- here the exception takes of
      if (value instanceof BigDecimal || value instanceof BigInteger) {
        result.put(colName, value.toString());
      } else {
        result.put(colName, value);
      }
      continue;
    }
// more code there, but left out

resultSet.getObject您可以通过设置 dataimporthandler 的配置参数来绕过调用。即这是

convertType（默认值：false）——将数据库返回的字段类型的附加转换应用到 schema.xml 中定义的字段类型。默认值似乎更安全，因为它不会导致额外的、神奇的转换。但是，在特殊情况下（例如 BLOB 字段），这种转换是解决问题的方法之一。[参考]

您需要在data-config.xml中执行此操作

<dataConfig>
<dataSource convertType="true" type="JdbcDataSource" driver="com.emc.ia.xdbjdbc.Driver" url="jdbc:xdbeas://localhost:1235/MaDatabase" user="*****" password="*****"/>

但是，我无法告诉您这是否成功地使用了您的数据库及其驱动程序。可能这只会导致另一个例外。在这种情况下，您将需要回退到解决方案

  1. 请您的供应商这样做
  2. 编写自定义数据导入例程
于 2015-08-17T06:51:20.153 回答