我们在 CentOS 6.3（完全打补丁）上运行 Tomcat 7.0.34，一旦服务器启动，我们所有的应用服务器的 CPU 都会飙升。没有与 Tomcat 建立连接。似乎有几个 CPU 内核完全被我们的 web 应用程序中运行的东西所消耗。
我一生都无法弄清楚可能导致此问题的原因。有没有人见过这个？
我应该提一下，这只发生在 CentOS 机器上。我的 Windows Tomcat 服务器在运行相同的应用程序时没有表现出这种行为。
相关技术
我们正在使用 Tomcat 的会话复制、Replicated EHCache 和 HornetQ for JMS。
测试的 Java 版本
JDK 1.7.0 更新 10 和 JDK 1.6.0 更新 38
杀死 -3 个可运行线程
"http-apr-8080-Acceptor-1" daemon prio=10 tid=0x00007f2598183000 nid=0x46b9 runnable [0x00007f2500685000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1013)
at java.lang.Thread.run(Thread.java:722)
"http-apr-8080-Acceptor-0" daemon prio=10 tid=0x00007f2598181800 nid=0x46b8 runnable [0x00007f2500786000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1013)
at java.lang.Thread.run(Thread.java:722)
"New I/O boss #26" daemon prio=10 tid=0x00007f253d4fb000 nid=0x462f runnable [0x00007f250c49c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006ab002568> (a sun.nio.ch.Util$2)
- locked <0x00000006ab002558> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006ab0024d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"New I/O worker #25" daemon prio=10 tid=0x00007f253d4e4000 nid=0x462e runnable [0x00007f250c59d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006aefbcab8> (a sun.nio.ch.Util$2)
- locked <0x00000006aefbcaa8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006aefbca60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"New I/O worker #24" daemon prio=10 tid=0x00007f253d4b9000 nid=0x462d runnable [0x00007f250c69e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006ab002c70> (a sun.nio.ch.Util$2)
- locked <0x00000006ab002c60> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006ab002bd8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
... ABOUT 20 MORE NIO WORKERS IN RUNNABLE STATE...
"New I/O worker #2" prio=10 tid=0x00007f253d054000 nid=0x4616 runnable [0x00007f250ddb5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006adaa7070> (a sun.nio.ch.Util$2)
- locked <0x00000006adaa7060> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006adaa6fd8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"New I/O worker #1" prio=10 tid=0x00007f253d053000 nid=0x4615 runnable [0x00007f250deb6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006adaa7bc0> (a sun.nio.ch.Util$2)
- locked <0x00000006adaa7bb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006adaa7b28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"NamingBootstrap Pool(1)-1" daemon prio=10 tid=0x00007f253c4f8800 nid=0x45fc runnable [0x00007f25842a3000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-1098" daemon prio=10 tid=0x00007f253c4f4800 nid=0x45fb runnable [0x00007f25843a4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-0" daemon prio=10 tid=0x00007f253cc2d800 nid=0x45e6 runnable [0x00007f25859b9000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-40001" daemon prio=10 tid=0x00007f253cc16000 nid=0x45da runnable [0x00007f25865c5000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"Multicast Heartbeat Receiver Thread" daemon prio=10 tid=0x00007f253cc18000 nid=0x45d8 runnable [0x00007f25867c7000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
- locked <0x00000006ab145b28> (a java.net.PlainDatagramSocketImpl)
at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
- locked <0x00000006ab145b28> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:786)
- locked <0x000000067ffd9240> (a java.net.DatagramPacket)
- locked <0x00000006ab1459f0> (a java.net.MulticastSocket)
at net.sf.ehcache.distribution.MulticastKeepaliveHeartbeatReceiver$MulticastReceiverThread.run(MulticastKeepaliveHeartbeatReceiver.java:124)
"Tribes-MembershipReceiver" daemon prio=10 tid=0x00007f259895c800 nid=0x45d0 runnable [0x00007f258733a000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
- locked <0x00000006b8163208> (a java.net.PlainDatagramSocketImpl)
at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
- locked <0x00000006b8163208> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:786)
- locked <0x00000006b8158768> (a java.net.DatagramPacket)
- locked <0x00000006b7b4fc30> (a java.net.MulticastSocket)
at org.apache.catalina.tribes.membership.McastServiceImpl.receive(McastServiceImpl.java:340)
at org.apache.catalina.tribes.membership.McastServiceImpl$ReceiverThread.run(McastServiceImpl.java:534)
"NioReceiver" daemon prio=10 tid=0x00007f259892f000 nid=0x45cf runnable [0x00007f258743b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006b8155008> (a sun.nio.ch.Util$2)
- locked <0x00000006b8155020> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006b78650a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:281)
at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:420)
at java.lang.Thread.run(Thread.java:722)
"NioReceiver" daemon prio=10 tid=0x00007f259892f000 nid=0x45cf runnable [0x00007f258743b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006b8155008> (a sun.nio.ch.Util$2)
- locked <0x00000006b8155020> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006b78650a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:281)
at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:420)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-0" daemon prio=10 tid=0x00007f25982e0800 nid=0x45cc runnable [0x00007f258c7a6000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-9004" daemon prio=10 tid=0x00007f25982cb000 nid=0x45cb runnable [0x00007f258c8a7000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-0" daemon prio=10 tid=0x00007f25982a7000 nid=0x45ca runnable [0x00007f258c9a8000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"Service Thread" daemon prio=10 tid=0x00007f259817a800 nid=0x45c9 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f2598178000 nid=0x45c8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f2598175800 nid=0x45c7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f2598173800 nid=0x45c6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f2598171800 nid=0x45c5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=10 tid=0x00007f259811c000 nid=0x45c2 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f2598012000 nid=0x45bd runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f2598014000 nid=0x45be runnable
"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f2598015800 nid=0x45bf runnable
"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f2598017800 nid=0x45c0 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f25980a6800 nid=0x45c1 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f25982e3000 nid=0x45cd waiting on condition
All of this seems VERY busy for a server that has just started up, has no replication peers, and has not yet accepted any requests.
Ideas?