0

一旦 cassandra 在其中一个节点上开始压缩，我们就会看到我们的应用程序无法连接到该节点。我注意到的是，当在其中一个节点中开始压缩时，该节点的 CPU 负载变为 100%，并且随着压缩的进行和 cpu 负载变低，我们的应用程序能够连接回 cassandra。

我们使用 m3.xlarge Amazon EC2 实例在同一个节点上运行我们的 java 应用程序和 cassandra。我们的应用程序和 cassandra 都有以下 jvm 设置 -Xms2G -Xmx2G -Xmn512M。Cassandra 几乎没有 300MB 的数据，但我们仍然看到如此多的压缩发生。这是我们的 cassandra 列族设置之一的样子，我们有大约 20 个具有几乎相同设置的列族。每个列族都有一个二级索引。

bloom_filter_fp_chance=0.010000 AND
  caching='ROWS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'min_threshold': '6', 'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

我们正在运行 cassandra 社区版本 2.0.5 和数据映射库https://github.com/valchkou/cassandra-driver-mapping。

我环顾四周，没有看到任何人面临这种​​问题。任何帮助将不胜感激。

错误堆栈跟踪

2015-07-29 18:22:50.341 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 234 : [] [] [] Defuncting connection to /172.31.22.125:9042
com.datastax.driver.core.TransportException: [/172.31.22.125:9042] Channel has been closed
    at com.datastax.driver.core.Connection$Dispatcher.channelClosed(Connection.java:636) [cassandra-driver-core-2.1.0.jar!/:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:375) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.0.Final.jar!/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
2015-07-29 18:22:50.341 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Cluster 1370 : [] [] [] Host /172.31.22.125:9042 is Suspected
2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-171] closing connection
2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-172] closing connection
2015-07-29 18:22:50.342 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Connection 103 : [] [] [] [/172.31.22.125:9042-173] Error connecting to /172.31.22.125:9042 (Connection refused: /172.31.22.125:9042)
2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #6] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-173] closing connection
2015-07-29 18:22:50.343 DEBUG 29056 --- [New I/O boss #9] com.datastax.driver.core.Connection 608 : [] [] [] [/172.31.22.125:9042-173] connection error
java.net.ConnectException: Connection refused: /172.31.22.125:9042
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_75]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.0.Final.jar!/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Connection 234 : [] [] [] Defuncting connection to /172.31.22.125:9042
com.datastax.driver.core.TransportException: [/172.31.22.125:9042] Cannot connect
    at com.datastax.driver.core.Connection.<init>(Connection.java:104) [cassandra-driver-core-2.1.0.jar!/:na]
    at com.datastax.driver.core.Connection$Factory.open(Connection.java:445) [cassandra-driver-core-2.1.0.jar!/:na]
    at com.datastax.driver.core.Cluster$Manager$4.runMayThrow(Cluster.java:1405) [cassandra-driver-core-2.1.0.jar!/:na]
    at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) [cassandra-driver-core-2.1.0.jar!/:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.net.ConnectException: Connection refused: /172.31.22.125:9042
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_75]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.0.Final.jar!/:na]
    ... 3 common frames omitted
2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Cluster 1428 : [] [] [] Host /172.31.22.125:9042 is DOWN
2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] c.d.driver.core.ControlConnection 592 : [] [] [] [Control connection] /172.31.22.125:9042 is down, currently connected to /172.31.27.168:9042
2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-37] com.datastax.driver.core.Cluster 1428 : [] [] [] Host /172.31.22.125:9042 is DOWN
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Cluster 1473 : [] [] [] /172.31.22.125:9042 is down, scheduling connection retries
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] c.d.driver.core.ControlConnection 592 : [] [] [] [Control connection] /172.31.22.125:9042 is down, currently connected to /172.31.27.168:9042
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-36] c.d.d.core.AbstractReconnectionHandler 60 : [] [] [] First reconnection scheduled in 1000ms
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] com.datastax.driver.core.Cluster 1473 : [] [] [] /172.31.22.125:9042 is down, scheduling connection retries
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] c.d.d.core.AbstractReconnectionHandler 60 : [] [] [] First reconnection scheduled in 1000ms
2015-07-29 18:22:51.345 DEBUG 29056 --- [Reconnection-0] com.datastax.driver.core.Connection 103 : [] [] [] [/172.31.22.125:9042-174] Error connecting to /172.31.22.125:9042 (Connection refused: /172.31.22.125:9042)
2015-07-29 18:22:51.345 DEBUG 29056 --- [New I/O worker #7] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-174] closing connection
2015-07-29 18:22:51.346 DEBUG 29056 --- [New I/O boss #9] com.datastax.driver.core.Connection 608 : [] [] [] [/172.31.22.125:9042-174] connection error
java.net.ConnectException: Connection refused: /172.31.22.125:9042
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]
4

1 回答 1

0
  1. 首先确保您的系统防火墙已打开或关闭。
  2. Cassandra 服务是 up 或 down 。
  3. 放置与 cassandra 和 cassandra-mapping jar 相同的 jar 版本。
于 2015-07-30T05:45:28.423 回答