1

我们在 CentO 上有两个 DatastaxEnterprise Solr 集群（4.5 版）的数据中心（欧洲 DC1，北美 DC2）：

DC1: 2 nodes with rf set to 2
DC2: 1 nodes with rf set to 1

每个节点都有 2 个内核和 4GB 的 RAM。我们只创建了一个 keyspace，DC1 的 2 个节点各有 400MB 的数据，而 DC2 的节点是空的。

如果我在 DC2 中的节点上启动 nodetool 修复，该命令可以正常运行大约 20/30 分钟，然后停止工作，仍然卡住。

在 DC2 中节点的日志中，我可以读到：

WARN [NonPeriodicTasks:1] 2014-10-01 05:57:44,188 WorkPool.java (line 398) Timeout while waiting for workers when flushing pool {}. IndexCurrent timeout is Failure to flush may cause excessive growth of Cassandra commit log.
 millis, consider increasing it, or reducing load on the node.
ERROR [NonPeriodicTasks:1] 2014-10-01 05:57:44,190 CassandraDaemon.java (line 199) Exception in thread Thread[NonPeriodicTasks:1,5,main]
org.apache.solr.common.SolrException: java.lang.RuntimeException: Timeout while waiting for workers when flushing pool {}. IndexCurrent timeout is Failure to flush may cause excessive growth of Cassandra commit log.
 millis, consider increasing it, or reducing load on the node.
    at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.commit(CassandraDirectUpdateHandler.java:351)
    at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doCommit(AbstractSolrSecondaryIndex.java:994)
    at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.forceBlockingFlush(AbstractSolrSecondaryIndex.java:139)
    at org.apache.cassandra.db.index.SecondaryIndexManager.flushIndexesBlocking(SecondaryIndexManager.java:338)
    at org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:144)
    at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:113)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Timeout while waiting for workers when flushing pool {}. IndexCurrent timeout is Failure to flush may cause excessive growth of Cassandra commit log.
 millis, consider increasing it, or reducing load on the node.
    at com.datastax.bdp.concurrent.WorkPool.doFlush(WorkPool.java:399)
    at com.datastax.bdp.concurrent.WorkPool.flush(WorkPool.java:339)
    at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.flushIndexUpdates(AbstractSolrSecondaryIndex.java:484)
    at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.commit(CassandraDirectUpdateHandler.java:278)
    ... 12 more
 WARN [commitScheduler-3-thread-1] 2014-10-01 05:58:47,351 WorkPool.java (line 398) Timeout while waiting for workers when flushing pool {}. IndexCurrent timeout is Failure to flush may cause excessive growth of Cassandra commit log.
 millis, consider increasing it, or reducing load on the node.
ERROR [commitScheduler-3-thread-1] 2014-10-01 05:58:47,352 SolrException.java (line 136) auto commit error...:org.apache.solr.common.SolrException: java.lang.RuntimeException: Timeout while waiting for workers when flushing pool {}. IndexCurrent timeout is Failure to flush may cause excessive growth of Cassandra commit log.
 millis, consider increasing it, or reducing load on the node.
    at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.commit(CassandraDirectUpdateHandler.java:351)
    at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Timeout while waiting for workers when flushing pool {}. IndexCurrent timeout is Failure to flush may cause excessive growth of Cassandra commit log.
 millis, consider increasing it, or reducing load on the node.
    at com.datastax.bdp.concurrent.WorkPool.doFlush(WorkPool.java:399)
    at com.datastax.bdp.concurrent.WorkPool.flush(WorkPool.java:339)
    at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.flushIndexUpdates(AbstractSolrSecondaryIndex.java:484)
    at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.commit(CassandraDirectUpdateHandler.java:278)
    ... 8 more

我尝试在 cassandra.yaml 文件中增加一些超时，但没有运气。谢谢

4

3 回答 3

1

对于 DSE solr 安装，您的节点完全没有指定。

我通常会推荐至少 8 个内核和至少 64 Gb 的内存。将堆分配到 12-14 Gb。

以下故障排除指南非常好：

https://support.datastax.com/entries/38367716-Solr-Configuration-Best-Practices-and-Troubleshooting-Tips

您当前的数据负载很小，因此您可能不需要完整的内存 - 我猜这里的瓶颈是 CPU。

如果您没有运行 4.0.4 或 4.5.2，我会使用其中一个版本。

于 2014-10-09T14:02:28.897 回答
1

可能有帮助的两个项目：

  1. 您在日志中看到的RuntimeException是沿着将索引更改提交到磁盘的 Lucene 代码路径，所以我肯定会确定写入磁盘是否是您的瓶颈。（您是否为数据和提交日志使用不同的物理磁盘？）

  2. 同时您可能想要调整的参数是控制WorkPool刷新超时的参数，dse.yaml称为flush_max_time_per_core.

于 2014-10-14T01:22:02.720 回答
0

减少 solr 索引争用的一种方法是增加 solrconfig.xml 中的 autoSoftCommit maxTime

<autoSoftCommit>
   <maxTime>1000000</maxTime>
</autoSoftCommit>
于 2014-11-05T19:18:01.340 回答