使用默认配置安装 Cassandra。只是一个单节点服务器,48GB 内存,2 TB 硬盘。当它显着减慢时,插入了大约 8000 万行。新连接因超时错误而被拒绝。
Opssenter 也会引发超时。
htop 显示 1 个 cassandra 进程,它为 100% 加载 CPU
iotop 显示定期读取\写入,但强度非常低 - 所以 HDD 不是瓶颈
大量 RAM 仍然可用,并且没有交换任何内容
nodetool tpstats - 没有工作,被“java.net.SocketTimeoutException:读取超时”压垮
nodetool status - 显示服务器正常工作(!):UN,负载为 122GB,Owns 100%,tokens 256
tail /var/log/cassandra/system.log - 对我来说没有任何信息,最后一行是
INFO [ScheduledTasks:1] 2014-02-16 04:36:21,824 StatusLogger.java (line 121) system.local
怎么了?如何找到当前执行的操作列表?如何找到该行为的原因?以及如何让它恢复正常?
谢谢!
PS最后它抛出了一个异常:
错误 [ReadStage:1550] 2014-02-16 05:22:26,476 CassandraDaemon.java(第 192 行)线程异常 [ReadStage:1550,5,main] java.lang.OutOfMemoryError:org.apache 处的 Java 堆空间。 cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:69) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.(CompressedRandomAccessReader.java:76) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open( CompressedRandomAccessReader.java:43) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48) at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39) at org .apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195) 在 org.apache.cassandra.db。columniterator.IndexedSliceReader.setToRowStart(IndexedSliceReader.java:108) at org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:84) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java :65) org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42) org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167) org.apache.cassandra .db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) 在 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:273) 在 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java :53) 在 org.apache.cassandra.db.ColumnFamilyStore。getTopLevelColumns(ColumnFamilyStore.java:1560) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379) at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) at org.apache .cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1396) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run (StorageProxy.java:1931) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread。运行(Thread.java:744)getColumnFamily(ColumnFamilyStore.java:1379) at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) at org.apache .cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1396) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1931) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor .java:1142) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:744)getColumnFamily(ColumnFamilyStore.java:1379) at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) at org.apache .cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1396) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1931) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor .java:1142) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:744)cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1396) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1931) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1142) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:744)cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1396) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1931) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1142) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:744)