在过去的几天里,Debian(谷歌计算引擎)服务器每天都面临套接字不可用的问题,需要重新启动 tomcat。
Dec 30, 2015 1:16:53 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:220)
at java.lang.Thread.run(Thread.java:745)
在使用 netstat -ap
找到的 GOOGLE :( 时检查的情况下,正在淹没服务器。
tcp 1 0 backend-tomcat:53976 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42251 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45929 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42159 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:55348 wo-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:44973 wa-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42148 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45729 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45721 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42146 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:36557 wq-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45723 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45915 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45295 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:53819 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 0 0 backend-tomcat:45968 wn-in-f95.1e100.n:https ESTABLISHED 5081/java
tcp 1 0 backend-tomcat:41737 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:55132 wm-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:53969 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:40883 wj-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45734 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:40889 wj-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:41834 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
检查了这个过程,lsof -p 5081
下面有大量的列表
java 5081 tomcat7 1181u IPv4 1959257 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48570->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1182u IPv4 1959420 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48575->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1183u IPv4 1959526 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48577->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1184u IPv4 1959676 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48583->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1185u IPv4 1960099 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48591->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1186u IPv4 1959857 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48589->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1187u IPv4 1960209 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48596->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1188u IPv4 1960352 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49524->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1189u IPv4 1960555 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49526->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1190u IPv4 1960721 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49531->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1191u IPv4 1960962 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49537->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1192u IPv4 1961151 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49539->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1193u IPv4 1961318 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49544->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1194u IPv4 1961557 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49550->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1195u IPv4 1961828 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46380->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1196u IPv4 1962155 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46386->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1197u IPv4 1962356 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46392->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1198u IPv4 1962461 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46393->wl-in-f95.1e100.net:https (CLOSE_WAIT)
从 ulimit 发现打开文件限制设置为最大值
open files (-n) 65536
是因为 GCE 的 Cloud Debugger 功能吗?有什么方法可以要求谷歌不要让服务器每天流血?