2

环境

我有以下环境：

  • 卡桑德拉 2.1.0
  • 一个 DC 5 个节点，第二个 DC 4 个节点
  • 每秒 2500 次写入
  • 最少读取（通常没有，有时很少）

问题

一个节点运行时间长后，GC开始耗时越来越长，直到nodetool报告这个节点宕机。

我可视化 gc.log 并有以下屏幕截图： 在此处输入图像描述

JVM 设置

评论后，工作 Cassandra 的完整命令行是：

java -ea -javaagent:/usr/lib/cassandra/bin/../lib/jamm-0.2.6.jar
 -XX:+UseThreadPriorities
 -XX:ThreadPriorityPolicy=42
 -Xms8192M
 -Xmx8192M
 -Xmn2048M
 -XX:+HeapDumpOnOutOfMemoryError
 -Xss192k
 -XX:+UseParNewGC
 -XX:+UseConcMarkSweepGC
 -XX:+CMSParallelRemarkEnabled
 -XX:SurvivorRatio=8
 -XX:MaxTenuringThreshold=1
 -XX:CMSInitiatingOccupancyFraction=75
 -XX:+UseCMSInitiatingOccupancyOnly
 -XX:+PrintGCDetails
 -XX:+PrintGCTimeStamps
 -XX:+PrintClassHistogram
 -XX:+PrintTenuringDistribution
 -XX:+PrintGCApplicationStoppedTime
 -Xloggc:/var/log/cassandra/gc.log
 -Djava.net.preferIPv4Stack=true
 -Dcom.sun.management.jmxremote.port=8080
 -Dcom.sun.management.jmxremote.ssl=false
 -Dcom.sun.management.jmxremote.authenticate=false
 -Dlogback.configurationFile=logback.xml
 -Dcassandra.logdir=/usr/lib/cassandra/bin/../logs
 -Dcassandra.storagedir=/usr/lib/cassandra/bin/../data
 -Dcassandra-pidfile=/var/run/cassandra.pid
 -cp /usr/lib/cassandra/bin/../conf:/usr/lib/cassandra/bin/../build/classes/main:/usr/lib/cassandra/bin/../build/classes/thrift:/usr/lib/cassandra/bin/../lib/airline-0.6.jar:/usr/lib/cassandra/bin/../lib/antlr-3.2.jar:/usr/lib/cassandra/bin/../lib/apache-cassandra-2.1.0.jar:/usr/lib/cassandra/bin/../lib/apache-cassandra-clientutil-2.1.0.jar:/usr/lib/cassandra/bin/../lib/apache-cassandra-thrift-2.1.0.jar:/usr/lib/cassandra/bin/../lib/commons-cli-1.1.jar:/usr/lib/cassandra/bin/../lib/commons-codec-1.2.jar:/usr/lib/cassandra/bin/../lib/commons-lang3-3.1.jar:/usr/lib/cassandra/bin/../lib/commons-math3-3.2.jar:/usr/lib/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/usr/lib/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/lib/cassandra/bin/../lib/disruptor-3.0.1.jar:/usr/lib/cassandra/bin/../lib/guava-16.0.jar:/usr/lib/cassandra/bin/../lib/high-scale-lib-1.0.6.jar:/usr/lib/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/lib/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/lib/cassandra/bin/../lib/jamm-0.2.6.jar:/usr/lib/cassandra/bin/../lib/javax.inject.jar:/usr/lib/cassandra/bin/../lib/jbcrypt-0.3m.jar:/usr/lib/cassandra/bin/../lib/jline-1.0.jar:/usr/lib/cassandra/bin/../lib/jna-4.0.0.jar:/usr/lib/cassandra/bin/../lib/jna.jar:/usr/lib/cassandra/bin/../lib/json-simple-1.1.jar:/usr/lib/cassandra/bin/../lib/libthrift-0.9.1.jar:/usr/lib/cassandra/bin/../lib/logback-classic-1.1.2.jar:/usr/lib/cassandra/bin/../lib/logback-core-1.1.2.jar:/usr/lib/cassandra/bin/../lib/lz4-1.2.0.jar:/usr/lib/cassandra/bin/../lib/metrics-core-2.2.0.jar:/usr/lib/cassandra/bin/../lib/netty-all-4.0.20.Final.jar:/usr/lib/cassandra/bin/../lib/reporter-config-2.1.0.jar:/usr/lib/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/usr/lib/cassandra/bin/../lib/snakeyaml-1.11.jar:/usr/lib/cassandra/bin/../lib/snappy-java-1.0.5.2.jar:/usr/lib/cassandra/bin/../lib/stream-2.5.2.jar:/usr/lib/cassandra/bin/../lib/super-csv-2.1.0.jar:/usr/lib/cassandra/bin/../lib/thrift-server-0.3.5.jar org.apache.cassandra.service.CassandraDaemon

问题

我应该检查什么以找出问题所在？此外，我最近

com.google.common.util.concurrent.UncheckedExecutionException：java.lang.RuntimeException：org.apache.cassandra.exceptions.ReadTimeoutException：操作超时 - 仅收到 2 个响应

4

1 回答 1

0

如果您的写入工作量很大，那么您最终会遇到大量带有 2gb 新一代的小型次要 GC。您可能希望将其拨回 800mb。

于 2015-01-10T06:02:58.307 回答