0

我使用以下脚本设置了我的河流：

curl -XPUT 'localhost:9200/_river/foo/_meta' -d '{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://...:3306/....",
        "user" : "...",
        "password" : "...",
        "sql" : "SELECT v.id as _id,v.name,v.entrydate, v.link, v.html,v.created AS _created,vc.name AS company, vp.name AS position FROM foo v LEFT JOIN foocompany vc ON vc.id=v.company LEFT JOIN fooposition vp ON vp.id=v.position ",
        "fetchsize" : 100,
        "bulk_size" : 100,
        "max_bulk_requests" : 2,
        "bulk_flush_interval" : "30s",
        "strategy": "simple",
        "poll": "30s",
        "autocommit": true
    }
}'

在这条河流运行一段时间后，我得到一个异常，这可能是由于 MySQL 服务器本身的配置：

[2014-11-27 16:54:02,301][ERROR][org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow] com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:231)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.move(SimpleRiverFlow.java:129)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.run(SimpleRiverFlow.java:88)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:928)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2053)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:406)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:385)
        at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:163)
        at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:7472)
        at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:919)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.close(SimpleRiverSource.java:613)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.execute(SimpleRiverSource.java:263)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:227)
        ... 3 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3166)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3620)
        ... 15 more

这里的问题是在我的设置中重新配置 MySQL 不是一个选项。唉，我不得不在别处寻找选择。

  1. ElasticSearch 是否可以在一段时间后重新建立连接然后恢复索引？
  2. 有没有其他方法可以在不使用 JDBC River 的情况下将 ElasticSearch 与 MySQL 连接起来？
4

1 回答 1

2

Elastic Rivers 让我非常头疼。不仅仅是 JDBC，还有自定义编写的河流、网络爬虫河流等。

一个重要的注意事项是河流很快就会被弃用。（将批量数据索引到 ElasticSearch 的首选方法？）

我见过的一个问题是，当 Elastic 重新启动时，河流并不总是可靠地启动。有时河流根本不开始，有时它们会。非常令人沮丧。

Elastic 的官方建议是将流程移到 Elastic 之外并将数据输入。

我已经用在 Linux 上运行的小型 C# 应用程序替换了我们所有的 JDBC 河流，作为同一 Elastic 服务器上的 cron 作业。效果很好，它更可靠，更容易启动/重启。在 Elastic 中重新创建河流对我来说一直很痛苦。

于 2014-12-01T17:59:56.917 回答