0

需要从 CentOS 6.3 版虚拟机上运行的 cassandra 1.1.6 数据库读取约 2.5Gb 的记录。当查询具有默认开箱即用配置的守护程序时,我收到错误:

 INFO [Thread-2] 2012-10-30 20:05:13,345 CassandraDaemon.java (line 212) Listening for thrift clients...
 WARN [ScheduledTasks:1] 2012-10-30 20:06:27,076 GCInspector.java (line 145) Heap is 0.8434091049049706 full.  You may need to reduce memtable and/or
 WARN [ScheduledTasks:1] 2012-10-30 20:06:27,077 StorageService.java (line 2855) Flushing CFS(Keyspace='system', ColumnFamily='Versions') to relieve m
 INFO [ScheduledTasks:1] 2012-10-30 20:06:27,077 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Versions@1970754472(83/103 serialized/l
 INFO [FlushWriter:2] 2012-10-30 20:06:27,078 Memtable.java (line 264) Writing Memtable-Versions@1970754472(83/103 serialized/live bytes, 3 ops)
 INFO [FlushWriter:2] 2012-10-30 20:06:27,096 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/Versions/system-Versions-hf-1
 WARN [ScheduledTasks:1] 2012-10-30 20:06:28,793 GCInspector.java (line 139) Heap is 0.9390217139392345 full.  You may need to reduce memtable and/or
 WARN [ScheduledTasks:1] 2012-10-30 20:06:28,794 AutoSavingCache.java (line 156) Reducing KeyCache capacity from 2075306 to 12 to reduce memory pressu
 WARN [ScheduledTasks:1] 2012-10-30 20:06:28,794 GCInspector.java (line 145) Heap is 0.9390217139392345 full.  You may need to reduce memtable and/or
 INFO [ScheduledTasks:1] 2012-10-30 20:06:28,795 StorageService.java (line 2851) Unable to reduce heap usage since there are no dirty column families
 WARN [ScheduledTasks:1] 2012-10-30 20:06:30,181 GCInspector.java (line 145) Heap is 0.9984246325381808 full.  You may need to reduce memtable and/or
 INFO [ScheduledTasks:1] 2012-10-30 20:06:30,182 StorageService.java (line 2851) Unable to reduce heap usage since there are no dirty column families
 WARN [ScheduledTasks:1] 2012-10-30 20:06:34,740 GCInspector.java (line 145) Heap is 0.9983338780063149 full.  You may need to reduce memtable and/or
 INFO [ScheduledTasks:1] 2012-10-30 20:06:34,741 StorageService.java (line 2851) Unable to reduce heap usage since there are no dirty column families
ERROR [ReadStage:33] 2012-10-30 20:06:34,843 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[ReadStage:33,5,main]
java.lang.OutOfMemoryError: Java heap space
<------>at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:323)
<------>at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
<------>at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363)
<------>at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120)
<------>at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:255)
<------>at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:275)
<------>at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:232)
<------>at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
<------>at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:311)
<------>at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
<------>at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
<------>at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
<------>at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:399)
<------>at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:382)
<------>at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:377)
<------>at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:339)
<------>at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
<------>at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
<------>at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
<------>at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
<------>at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
<------>at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147)

没有太多时间来计算堆/内存表等的大小,补充说

JVM_OPTS="-Xms4g -Xmx4g"

到守护进程配置(测试服务器有 8 gigs 的 ram)。查询再次失败

ERROR [ReadStage:1] 2012-10-30 20:46:22,417 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[ReadStage:1,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
        at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:71)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException
        at org.apache.cassandra.io.util.FastByteArrayOutputStream.<init>(FastByteArrayOutputStream.java:78)
        at org.apache.cassandra.io.util.DataOutputBuffer.<init>(DataOutputBuffer.java:40)
        at org.apache.cassandra.db.RangeSliceReply.getReply(RangeSliceReply.java:48)
        at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:64)
        ... 4 more

我无法进一步调试。有谁知道我如何调整 Cassandra 以便我可以运行该查询?数据库有约 500 个超列,每个约 7 mb。我需要全部阅读它们,并且在某些时候在内存中(客户端机器有 40 gigs 的 ram,所以 100% 不是缺少资源的问题)以供进一步处理。查询结果根本没有返回到 api 中。

4

1 回答 1

0

根据建议,您通常应该为 cassandra 使用 8G RAM 甚至 16G ram。

你在运行什么查询?我知道 PlayOrm 使用游标,因此它可以为您提供碎片,因此它不会耗尽内存。我们已经使用 PlayOrm 和 cassandra 轻松地将 100Gigs 的东西检索到我们的客户端,尽管我们在它流回时将其丢弃并且我们没有遇到任何问题。

后来,迪恩

于 2012-10-30T21:11:03.193 回答