macos - Os x terminal, ssh , and too many open files

Question

I have some code that runs several rest queries over a connection that is ssh forwarded to an AWS machine (fyi: these queries are hitting a Solr server running on this machine), and the queries run against my localhost (which is forwarded to the AWS instance).

The code initially runs great getting data as necessary, but after running for a while, the code stalls (in eclipse).

At this exact moment, the terminal (i.e. where I have started my ssh tunnel) goes completely frozen, filling up with the String :

"accept : too many open files"

Because this infinite print is not associated with a bash terminal (i.e. I can't tell wether the ssh connection is still alive or not, and there is no text indicating which shell im in... just unpridled, relentless print statements) I cant tell wether it's coming from amazon, or from my client terminal.

I want to find the cause of this behavior and pinpoint the machine which is causing my terminal to explode

To test which of the two machines was causing the infinite print outs of the error, I ran the ulimit command on the server... and found that the max number of open files allowed (on the aws server) was well above the amount of open files (also determined using ulimit) at any given time while the client program (running from my ide) is executing.

I did the same test on my client , and found no significant increase in the number of open files.

Some side details : I'm running several hundreds of queries into a SOLR server that has over 100GB of data in a short time period.

Any hints on how to determine why my sshd mac os x terminal is dying and infinitely printing this message would be potentially very useful to me. Of course, wether or not they were specific to solr. That said, any insights into why this would happen when using a solr service may also help to solve this problem.

Answer 1

您可以尝试查看ulimit（通过终端类型）：

ulimit -a

具体来说，检查 的值open files。在我的机器（OS X）上，它报告 256。您可能想尝试将其增加到 512：

ulimit -n 512

Answer 2

这似乎对我有用：

echo 'kern.maxfiles=20480' | sudo tee -a /etc/sysctl.conf
echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
echo 'ulimit -n 4096' | sudo tee -a /etc/profile

然后重新启动 OS X。

https://superuser.com/questions/302754/increase-the-maximum-number-of-open-file-descriptors-in-snow-leopard

Answer 3

这里没有足够的信息可以确定，但听起来像是ssh在尝试在本地接受到转发套接字的连接时达到其每个进程的文件描述符限制，这反过来表明已经打开的连接没有及时关闭时尚。您可以运行sshwith-d查看连接和断开连接的详细信息；您可能想要捕获它的标准错误并使用脚本来跟踪套接字操作，因为它们将隐藏在许多其他调试信息中。

请注意，ssh它本身可能会限制其打开的文件描述符超出默认值，以避免对其转发端口的拒绝服务攻击；我在文档中没有看到任何迹象，但这是我所期望的。

Answer 4

sudo launchctl limit maxfiles 1000000 unlimited

或者

sudo sysctl -w kern.maxfilesperproc=1000000
sudo sysctl -w kern.maxfilesperproc=18000

要使更改永久生效，请使用 sudo 将您的设置放入 /etc/sysctl.conf（您可能必须创建），如下所示：

kern.maxfiles=20480 
kern.maxfilesperproc=18000

注意 - 选择号码需要您自担风险

Answer 5

以下命令帮助了我，

launchctl limit maxfiles