1

我目前正在使用 SSTableLoader 将多个 CSV 文件中的数据加载到 cassandra 中。以前当我在 Cassandra 2.1.9 上运行 SSTableLoader 时，它运行成功，没有任何错误。

但是，自从我升级到 cassandra 2.2.3 后，我一直遇到来自 SSTableLoader 的以下错误：

org.apache.cassandra.io.FSReadError: java.io.IOException: An existing connection was forcibly closed by the remote host
    at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[na:1.8.0_45]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51) ~[na:1.8.0_45]
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_45]
    at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[na:1.8.0_45]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_45]
    at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:483) ~[na:1.8.0_45]
    at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:579) ~[na:1.8.0_45]
    at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.3.jar:2.2.3]
    ... 11 common frames omitted


After examining the system.log file in "/var/log/cassandra", I see that the error was caused by "Too many open files".

Here is the error message in the system.log file:

ERROR [STREAM-IN-/169.254.122.80] 2015-12-02 11:37:48,235 StreamSession.java:524 - [Stream #d33ca5f0-98e8-11e5-b1fd-732c31b2403a] Streaming error occurred
java.nio.file.FileSystemException: /var/lib/cassandra/data/pricing/listings-088f7050945411e59780d168f1e8851b/tmp-la-968-big-Data.db: Too many open files
    at sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[na:1.8.0_60]
    at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[na:1.8.0_60]
    at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[na:1.8.0_60]
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(Unknown Source) ~[na:1.8.0_60]
    at java.nio.channels.FileChannel.open(Unknown Source) ~[na:1.8.0_60]
    at java.nio.channels.FileChannel.open(Unknown Source) ~[na:1.8.0_60]
    at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:131) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:66) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:75) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:84) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:106) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:95) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:129) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:78) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]

在研究过有关此主题的先前问题后，我尝试过的一件事是使用 ulimit 命令将打开文件的数量设置为 1000000。但是在重新启动 cassandra 服务后，我仍然收到错误消息。

我还需要做些什么才能使这些更改生效吗？我必须在 cassandra.yaml 文件中设置一个设置来防止这种情况发生吗？非常感谢。

4

3 回答 3

3

这个问题是因为打开文件描述符总数的限制。使用命令检查此限制

ulimit -a  
or  
ulimit -n

使用命令更改它的简单方法 -

ulimit -n 100000

（我以 root 用户身份运行此命令，可能可以使用 sudo）

于 2017-06-19T01:50:29.927 回答
1

使用 ulimit 可能没有按照您预期的方式应用它。可以在您的 limits.conf 中检查 cat /proc/<pid>/limits Configure it，并在那里更新其他内容，例如 nproc 限制。请参阅http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html

于 2015-12-03T16:31:44.277 回答
1

谢谢你，我设法解决了这个问题。

对于其他人的信息，我做了以下事情：

  1. 编辑 /etc/security 中的 limits.conf 文件

  2. 添加以下条目：

    • 软文件 100000
    • 硬文件 100000

  3. 保存更改

  4. 使用“rebo​​ot”命令重启机器
  5. 重启机器后，我运行 'ulimit -a' 命令看到打开的文件数是 100000。
  6. 批量加载过程成功。
于 2016-01-21T10:32:56.103 回答