1

我们的 3 节点集群中的一个节点已关闭，在检查日志文件时，它显示以下消息

INFO  [keyspace.core Index WorkPool work thread-2] 2016-09-14 14:05:32,891  AbstractMetrics.java:114 - Cannot record QUEUE latency of 11 minutes because higher than 10 minutes.
INFO  [keyspace.core Index WorkPool work thread-2] 2016-09-14 14:05:33,233  AbstractMetrics.java:114 - Cannot record QUEUE latency of 10 minutes because higher than 10 minutes.
WARN  [keyspace.core Index WorkPool work thread-2] 2016-09-14 14:05:33,398  Worker.java:99 - Interrupt/timeout detected.
java.util.concurrent.BrokenBarrierException: null
at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:200) ~[na:1.7.0_79]
at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355) ~[na:1.7.0_79]
at com.datastax.bdp.concurrent.FlushTask.bulkSync(FlushTask.java:76) ~[dse-core-4.8.3.jar:4.8.3]
at com.datastax.bdp.concurrent.Worker.run(Worker.java:94) ~[dse-core-4.8.3.jar:4.8.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
WARN  [keyspace.core Index WorkPool work thread-2] 2016-09-14 14:05:33,398  Worker.java:99 - Interrupt/timeout detected.
java.util.concurrent.BrokenBarrierException: null
at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:200) ~[na:1.7.0_79]
at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355) ~[na:1.7.0_79]
at com.datastax.bdp.concurrent.FlushTask.bulkSync(FlushTask.java:76) ~[dse-core-4.8.3.jar:4.8.3]
at com.datastax.bdp.concurrent.Worker.run(Worker.java:94) ~[dse-core-4.8.3.jar:4.8.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
INFO  [keyspace.core Index WorkPool work thread-4] 2016-09-14 14:05:33,720  AbstractMetrics.java:114 - Cannot record QUEUE latency of 13 minutes because higher than 10 minutes.
INFO  [keyspace.core Index WorkPool work thread-4] 2016-09-14 14:05:33,721  AbstractMetrics.java:114 - Cannot record QUEUE latency of 13 minutes because higher than 10 minutes.

节点配置为 8 CPU、32 GB RAM、500 GB 磁盘空间。只有一个特定节点出现故障的原因可能是什么？

4

1 回答 1

0

因此，我将在这里回答一些一般信息，您的情况可能更复杂。对于 Solr 节点，32GB RAM 可能不够大；在 Java 1.8 上使用 G1 收集器已证明对于堆大小超过 26GB 的 Solr 更好。

我也不确定堆大小、JVM 设置以及您在这里拥有多少个 solr 内核。但是，当节点忙于索引并试图跟上时，我已经看到了与此类似的错误。根据我的经验，在 Solr 节点上看到的最常见问题max_solr_concurrency_per_core之一是dse.yaml. 这通常会将索引线程的数量分配给 CPU 内核的数量，并且为了进一步复杂化问题，您可能会看到 8 个内核，但如果您有 HT，那么它实际上可能是 4 个物理内核。

检查您的dse.yaml并确保您num physcal cpu cores / num of solr cores至少将其设置为 2。这可能会变慢索引，但您应该消除节点的压力。

我会推荐这个有用的博客作为调整 DSE Solr 的良好开端：

http://www.datastax.com/dev/blog/tuning-dse-search

还有关于这个主题的文档：

https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/srch/srchTune.html

于 2016-10-11T17:11:55.757 回答