我在 Unicorn 和 Nginx(多个 Web 节点)上运行一个非常高流量的 Rails 3.2 应用程序,但每隔一段时间我就会看到 Unicorn 工作人员开始超时并在所有节点上被 Unicorn master sigkilled。当然,当 Unicorn 工人被 Unicorn 主人 sigkill 时,一个新工人会被分叉,但它也会挂起 60 秒然后超时并被杀死。这基本上会反复发生,直到我硬杀死所有独角兽大师和工人。
Unicorn log:
E, [2013-04-18T12:57:50.007623 #14002] ERROR -- : worker=8 PID:14968 timeout (62s > 60s), killing
E, [2013-04-18T12:57:50.108364 #14002] ERROR -- : reaped #<Process::Status: pid 14968 SIGKILL (signal 9)> worker=8
I, [2013-04-18T12:57:50.489505 #15726] INFO -- : worker=8 ready
E, [2013-04-18T12:57:52.175842 #14002] ERROR -- : worker=5 PID:15033 timeout (61s > 60s), killing
E, [2013-04-18T12:57:52.276586 #14002] ERROR -- : reaped #<Process::Status: pid 15033 SIGKILL (signal 9)> worker=5
I, [2013-04-18T12:57:52.653069 #15782] INFO -- : worker=5 ready
E, [2013-04-18T12:57:56.340290 #14002] ERROR -- : worker=3 PID:15074 timeout (61s > 60s), killing
E, [2013-04-18T12:57:56.440993 #14002] ERROR -- : reaped #<Process::Status: pid 15074 SIGKILL (signal 9)> worker=3
I, [2013-04-18T12:57:56.809730 #15832] INFO -- : worker=3 ready
E, [2013-04-18T12:57:57.504142 #14002] ERROR -- : worker=7 PID:15087 timeout (61s > 60s), killing
E, [2013-04-18T12:57:57.604886 #14002] ERROR -- : reaped #<Process::Status: pid 15087 SIGKILL (signal 9)> worker=7
I, [2013-04-18T12:57:57.983581 #15845] INFO -- : worker=7 ready
E, [2013-04-18T12:57:59.669664 #14002] ERROR -- : worker=4 PID:15108 timeout (61s > 60s), killing
E, [2013-04-18T12:57:59.770427 #14002] ERROR -- : reaped #<Process::Status: pid 15108 SIGKILL (signal 9)> worker=4
I, [2013-04-18T12:58:00.155461 #15879] INFO -- : worker=4 ready
E, [2013-04-18T12:58:06.839906 #14002] ERROR -- : worker=9 PID:15192 timeout (61s > 60s), killing
E, [2013-04-18T12:58:06.940829 #14002] ERROR -- : reaped #<Process::Status: pid 15192 SIGKILL (signal 9)> worker=9
I, [2013-04-18T12:58:07.302766 #15956] INFO -- : worker=9 ready
E, [2013-04-18T12:58:08.003330 #14002] ERROR -- : worker=6 PID:15213 timeout (61s > 60s), killing
E, [2013-04-18T12:58:08.104006 #14002] ERROR -- : reaped #<Process::Status: pid 15213 SIGKILL (signal 9)> worker=6
I, [2013-04-18T12:58:08.466790 #15973] INFO -- : worker=6 ready
监控系统显示外部服务(Postgres 数据库、Memcached、Redis)都在正常响应并且没有延迟问题。
以下是一些可能有价值的输出:
在这些中断期间,我注意到与 Unicorn 套接字的连接尝试大量积压。当站点未关闭时,通常以下命令仅返回一两行。
netstat | grep unic
....
unix 2 [ ] STREAM CONNECTING 0 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTING 0 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTING 0 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTING 0 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTING 0 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTED 7768134 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTED 7767311 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTED 7766999 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTED 7767309 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTED 7766941 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTED 7767287 /tmp/unicorn.sock
unix 2 [ ] STREAM CONNECTED 7766225 /tmp/unicorn.sock
任何人都知道可能导致这种情况的原因是什么?这种情况同时发生在多台服务器上。