需要从 CentOS 6.3 版虚拟机上运行的 cassandra 1.1.6 数据库读取约 2.5Gb 的记录。当查询具有默认开箱即用配置的守护程序时,我收到错误:
INFO [Thread-2] 2012-10-30 20:05:13,345 CassandraDaemon.java (line 212) Listening for thrift clients...
WARN [ScheduledTasks:1] 2012-10-30 20:06:27,076 GCInspector.java (line 145) Heap is 0.8434091049049706 full. You may need to reduce memtable and/or
WARN [ScheduledTasks:1] 2012-10-30 20:06:27,077 StorageService.java (line 2855) Flushing CFS(Keyspace='system', ColumnFamily='Versions') to relieve m
INFO [ScheduledTasks:1] 2012-10-30 20:06:27,077 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Versions@1970754472(83/103 serialized/l
INFO [FlushWriter:2] 2012-10-30 20:06:27,078 Memtable.java (line 264) Writing Memtable-Versions@1970754472(83/103 serialized/live bytes, 3 ops)
INFO [FlushWriter:2] 2012-10-30 20:06:27,096 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/Versions/system-Versions-hf-1
WARN [ScheduledTasks:1] 2012-10-30 20:06:28,793 GCInspector.java (line 139) Heap is 0.9390217139392345 full. You may need to reduce memtable and/or
WARN [ScheduledTasks:1] 2012-10-30 20:06:28,794 AutoSavingCache.java (line 156) Reducing KeyCache capacity from 2075306 to 12 to reduce memory pressu
WARN [ScheduledTasks:1] 2012-10-30 20:06:28,794 GCInspector.java (line 145) Heap is 0.9390217139392345 full. You may need to reduce memtable and/or
INFO [ScheduledTasks:1] 2012-10-30 20:06:28,795 StorageService.java (line 2851) Unable to reduce heap usage since there are no dirty column families
WARN [ScheduledTasks:1] 2012-10-30 20:06:30,181 GCInspector.java (line 145) Heap is 0.9984246325381808 full. You may need to reduce memtable and/or
INFO [ScheduledTasks:1] 2012-10-30 20:06:30,182 StorageService.java (line 2851) Unable to reduce heap usage since there are no dirty column families
WARN [ScheduledTasks:1] 2012-10-30 20:06:34,740 GCInspector.java (line 145) Heap is 0.9983338780063149 full. You may need to reduce memtable and/or
INFO [ScheduledTasks:1] 2012-10-30 20:06:34,741 StorageService.java (line 2851) Unable to reduce heap usage since there are no dirty column families
ERROR [ReadStage:33] 2012-10-30 20:06:34,843 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[ReadStage:33,5,main]
java.lang.OutOfMemoryError: Java heap space
<------>at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:323)
<------>at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
<------>at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363)
<------>at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120)
<------>at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:255)
<------>at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:275)
<------>at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:232)
<------>at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
<------>at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:311)
<------>at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
<------>at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
<------>at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
<------>at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:399)
<------>at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:382)
<------>at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:377)
<------>at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:339)
<------>at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
<------>at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
<------>at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
<------>at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
<------>at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
<------>at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147)
没有太多时间来计算堆/内存表等的大小,补充说
JVM_OPTS="-Xms4g -Xmx4g"
到守护进程配置(测试服务器有 8 gigs 的 ram)。查询再次失败
ERROR [ReadStage:1] 2012-10-30 20:46:22,417 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[ReadStage:1,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:71)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException
at org.apache.cassandra.io.util.FastByteArrayOutputStream.<init>(FastByteArrayOutputStream.java:78)
at org.apache.cassandra.io.util.DataOutputBuffer.<init>(DataOutputBuffer.java:40)
at org.apache.cassandra.db.RangeSliceReply.getReply(RangeSliceReply.java:48)
at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:64)
... 4 more
我无法进一步调试。有谁知道我如何调整 Cassandra 以便我可以运行该查询?数据库有约 500 个超列,每个约 7 mb。我需要全部阅读它们,并且在某些时候在内存中(客户端机器有 40 gigs 的 ram,所以 100% 不是缺少资源的问题)以供进一步处理。查询结果根本没有返回到 api 中。