我遇到了一个间歇性且难以复制的快乐问题。
在运行几分钟或几天后,我的测试服务器上的 grails 应用程序失败。我无法在 Dev 或我的本地复制。
这里的堆栈跟踪模式示例从中间剪掉了 1000 行(这将继续像这样转储,直到我重新启动 Tomcat):
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
Caused by: javax.servlet.ServletException: javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
at net.sf.ehcache.constructs.web.filter.Filter.logThrowable(Filter.java:143)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:91)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
......
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
Caused by: javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
at grails.plugin.cache.web.filter.AbstractFilter.logThrowable(AbstractFilter.java:116)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:70)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
..................
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
Caused by: org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
... 132 more
Caused by: java.lang.OutOfMemoryError: Java heap space
这看起来可能发生了应用程序错误,调用了 errorHandler 并不断调用自身...kaboom!
环境:
- 圣杯:2.0.4
- Tomcat:apache-tomcat-7.0.28
- 操作系统:红帽企业 Linux 服务器版本 5.8 (Tikanga)
爪哇:
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.8) (rhel-1.27.1.10.8.el5_8-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
在 Dev Java 中(这可能是我试图让 SA 在测试系统中更改的主要问题)
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
该服务器目前只有 0-5 名用户登录,但这会在 24 小时内起起落落。
分配的堆空间为 5g,通常使用 <20%。
在测试中发生了另外两件奇怪的事情,而不是在开发中:
最初几乎没有日志记录但由于问题而导致的日志已在 50 日打开到 DEBUG 和 RollingFile 100Meg 文件。日志很好,记录应用程序特定的东西,直到堆栈爆炸,然后它们中唯一的东西就是堆栈,tomcat 日志记录也是如此。即使我在第 50 个日志滚出之前关闭服务器也是如此。所以我不知道是否有一些应用程序特定的不祥之物会触发这种情况。
JavaMelody 插件:图表上的标签不可读,就像我 3 岁的孩子写的那样。我相信所有操作系统字体都已加载,但可能指向 Java 版本。
调度程序: 是的,有 2 个正在运行,运行它们没有问题
数据库配置: 遗留数据库有 8 个数据源,每个当前可能设置得有点饿,例如:
def connectionPropertiesMedium = [
maxActive: 100,
maxIdle: 30,
minIdle: 5,
initialSize: 30,
testOnBorrow: true,
testWhileIdle: false,
testOnReturn: false,
validationQuery: "SELECT 1",
minEvictableIdleTimeMillis: 600000,
timeBetweenEvictionRunsMillis: 600000,
numTestsPerEvictionRun: 3,
maxWait: 10000,
defaultTransactionIsolation: java.sql.Connection.TRANSACTION_READ_UNCOMMITTED
]
HeapDump:
Running with -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=heapdump.hprof
Reading with VisualVM 这几乎没有显示(无论如何对我来说) 70%+ 是 char[] 充满了上面的堆栈跟踪。Eclipse MAT 不会加载它。
JVM参数:
CATALINA_OPTS="-server -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=heapdump.hprof -XX:MaxPermSize=1024m -XX:MaxNewSize=256m -XX:NewSize=256m -Xms768m -Xmx1024m -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -XX:+UseTLAB -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSIncrementalMode -XX:-UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent
压力测试: 应用程序可以运行几个激烈的查询,这些查询被多达 200 名模拟 JMeter 用户多次点击,没有问题......实际上数据库确实有点慢;)但内存问题是没有在许多运行中复制。
因此,如果有人读到这里,我有 2 条线索(向更多人开放):
- Java 版本 - 旨在摆脱 OpenJDK 并进入标准版本
- 发生错误时,我的 Grails 设置中的某些内容会导致无限/递归处理。
直到我找到能摧毁这些机器中的幽灵的质子包,这个应用程序才会上线。
有任何想法吗?
干杯...
更新#1
安装了 Java 1.7 并删除了 OpenJDK,这肯定解决了 JavaMelody 显示问题。现在它正在等待游戏,看看这是否解决了主要问题。有趣的是,该网站似乎明显更快。