2

嗨,我在服务器上运行 BAM 进行测试,该服务器具有以下文件系统分布:

root@serv:/# df -h
Filesystem                             Size  Used Avail Use% Mounted on
/dev/mapper/VG-LV_root_                3.8G  2.3G  1.4G  62% /
udev                                   3.0G  4.0K  3.0G   1% /dev
tmpfs                                  1.2G  224K  1.2G   1% /run
none                                   5.0M     0  5.0M   0% /run/lock
none                                   3.0G     0  3.0G   0% /run/shm
/dev/mapper/VG-LV_var                  7.3G  4.5G  2.5G  65% /var

我将 BAM 放入具有 7.3GB 空间的 var 中。如您所见,我有足够的空间来运行 WSO2 产品,但 BAM :-D

运行 BAm 3 天后,我面临空间问题:

在 tmp 我有 1GB 的 hadoop

root@serv:/# du -sh /tmp/*
1023M   /tmp/hadoop
4.0K    /tmp/hadoop-root
4.0K    /tmp/hsperfdata_root
4.0K    /tmp/mc-root
117M    /tmp/root

在 BAM 内部,它在 tmp 中使用了 2.9GB。

root@serv:/# du -sh /var/BAM/wso2bam-2.0.1/*
160K    /var/BAM/wso2bam-2.0.1/bin
236K    /var/BAM/wso2bam-2.0.1/dbscripts
8.0K    /var/BAM/wso2bam-2.0.1/INSTALL.txt
5.0M    /var/BAM/wso2bam-2.0.1/lib
52K     /var/BAM/wso2bam-2.0.1/LICENSE.txt
12K     /var/BAM/wso2bam-2.0.1/README.txt
8.0K    /var/BAM/wso2bam-2.0.1/release-notes.html
540M    /var/BAM/wso2bam-2.0.1/repository
80K     /var/BAM/wso2bam-2.0.1/resources
14M     /var/BAM/wso2bam-2.0.1/samples
2.9G    /var/BAM/wso2bam-2.0.1/tmp
88K     /var/BAM/wso2bam-2.0.1/webapp-mode
4.0K    /var/BAM/wso2bam-2.0.1/wso2carbon.pid

我的问题与以下事实有关,即我仍然有磁盘空间,并且在短短 3 天内 BAM 使用了 4GB,所以:长时间部署的最佳 HDD 大小是多少,或者我可以做些什么来避免这个错误(在生产环境中我可以使用更多的磁盘空间,但这是我的客户关心的问题):注意到在这 3 天里,我只监控 2 台服务器,AS 和 ESB,负载最小。

TID: [0] [BAM] [2012-11-06 12:00:00,026]  INFO {org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask} -  Running script executor task for script esb_stats_0. [Tue Nov 06 12:00:00 CST 2012] {org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask}
TID: [0] [BAM] [2012-11-06 12:00:00,026]  INFO {org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask} -  Running script executor task for script service_stats_848. [Tue Nov 06 12:00:00 CST 2012] {org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask}
TID: [0] [BAM] [2012-11-06 12:00:01,213] ERROR {org.apache.hadoop.hive.ql.exec.ExecDriver} -  Exception: /var/BAM/wso2bam-2.0.1/tmp/root/hive_2012-11-06_12-00-00_487_706674934402805629/-local-10000/plan.xml (No space left on device) {org.apache.hadoop.hive.ql.exec.ExecDriver}
TID: [0] [BAM] [2012-11-06 12:00:01,213] ERROR {org.apache.hadoop.hive.ql.exec.ExecDriver} -  Exception: /var/BAM/wso2bam-2.0.1/tmp/root/hive_2012-11-06_12-00-00_487_706674934402805629/-local-10000/plan.xml (No space left on device) {org.apache.hadoop.hive.ql.exec.ExecDriver}
TID: [0] [BAM] [2012-11-06 12:00:01,216] ERROR {org.apache.hadoop.hive.ql.Driver} -  FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask {org.apache.hadoop.hive.ql.Driver}
TID: [0] [BAM] [2012-11-06 12:00:01,216] ERROR {org.apache.hadoop.hive.ql.Driver} -  FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask {org.apache.hadoop.hive.ql.Driver}
TID: [0] [BAM] [2012-11-06 12:00:01,219] ERROR {org.apache.hadoop.hive.ql.exec.ExecDriver} -  Exception: /var/BAM/wso2bam-2.0.1/tmp/root/hive_2012-11-06_12-00-00_500_8714455975079609718/-local-10000/plan.xml (No space left on device) {org.apache.hadoop.hive.ql.exec.ExecDriver}
TID: [0] [BAM] [2012-11-06 12:00:01,219] ERROR {org.apache.hadoop.hive.ql.exec.ExecDriver} -  Exception: /var/BAM/wso2bam-2.0.1/tmp/root/hive_2012-11-06_12-00-00_500_8714455975079609718/-local-10000/plan.xml (No space left on device) {org.apache.hadoop.hive.ql.exec.ExecDriver}
TID: [0] [BAM] [2012-11-06 12:00:01,221] ERROR {org.apache.hadoop.hive.ql.Driver} -  FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask {org.apache.hadoop.hive.ql.Driver}
TID: [0] [BAM] [2012-11-06 12:00:01,221] ERROR {org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl} -  Error while executing Hive script.
Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask {org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl}
java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
    at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:189)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:325)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:225)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
TID: [0] [BAM] [2012-11-06 12:00:01,221] ERROR {org.apache.hadoop.hive.ql.Driver} -  FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask {org.apache.hadoop.hive.ql.Driver}
TID: [0] [BAM] [2012-11-06 12:00:01,223] ERROR {org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl} -  Error while executing Hive script.
Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask {org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl}
java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
    at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:189)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:325)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:225)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
TID: [0] [BAM] [2012-11-06 12:00:01,231] ERROR {org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask} -  Error while executing script : esb_stats_0 {org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask}
org.wso2.carbon.analytics.hive.exception.HiveExecutionException: Error while executing Hive script.Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl.execute(HiveExecutorServiceImpl.java:110)
    at org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask.execute(HiveScriptExecutorTask.java:60)
    at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:56)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
TID: [0] [BAM] [2012-11-06 12:00:01,232] ERROR {org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask} -  Error while executing script : service_stats_848 {org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask}
org.wso2.carbon.analytics.hive.exception.HiveExecutionException: Error while executing Hive script.Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl.execute(HiveExecutorServiceImpl.java:110)
    at org.wso2.carbon.analytics.hive.task.HiveScriptExecutorTask.execute(HiveScriptExecutorTask.java:60)
    at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:56)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
TID: [0] [BAM] [2012-11-06 12:00:01,743]  INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} -  'admin@carbon.super [-1234]' logged in at [2012-11-06 12:00:01,742-0500] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [BAM] [2012-11-06 12:00:01,750]  INFO {org.wso2.carbon.dashboard.gadgetrepopopulator.GadgetRepoPopulator} -  Couldn't find a Dashboard at '/var/BAM/wso2bam-2.0.1/repository/resources/gadget-repo/gadget-repo.xml'. Giving up. {org.wso2.carbon.dashboard.gadgetrepopopulator.GadgetRepoPopulator}
TID: [0] [BAM] [2012-11-06 12:00:04,213] ERROR {org.apache.tiles.jsp.context.JspTilesRequestContext} -  JSPException while including path '/hive-explorer/listscripts.jsp'.  {org.apache.tiles.jsp.context.JspTilesRequestContext}
javax.servlet.jsp.JspException: ServletException while including page.
    at org.apache.tiles.jsp.context.JspUtil.doInclude(JspUtil.java:102)
    at org.apache.tiles.jsp.context.JspTilesRequestContext.include(JspTilesRequestContext.java:88)
    at org.apache.tiles.jsp.context.JspTilesRequestContext.dispatch(JspTilesRequestContext.java:82)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:465)
    at org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:140)
    at org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:117)
    at org.apache.tiles.jsp.taglib.RenderTagSupport.execute(RenderTagSupport.java:171)
    at org.apache.tiles.jsp.taglib.RoleSecurityTagSupport.doEndTag(RoleSecurityTagSupport.java:75)
    at org.apache.tiles.jsp.taglib.ContainerTagSupport.doEndTag(ContainerTagSupport.java:80)
    at org.apache.jsp.admin.layout.template_jsp._jspx_meth_tiles_insertAttribute_7(template_jsp.java:539)
    at org.apache.jsp.admin.layout.template_jsp._jspService(template_jsp.java:290)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.wso2.carbon.ui.JspServlet.service(JspServlet.java:161)
    at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:80)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:36)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:690)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:477)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
    at org.eclipse.equinox.http.servlet.internal.RequestDispatcherAdaptor.forward(RequestDispatcherAdaptor.java:30)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor$RequestDispatcherAdaptor.forward(ContextPathServletAdaptor.java:258)
    at org.apache.tiles.servlet.context.ServletTilesRequestContext.forward(ServletTilesRequestContext.java:198)
    at org.apache.tiles.servlet.context.ServletTilesRequestContext.dispatch(ServletTilesRequestContext.java:185)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:419)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:370)
    at org.wso2.carbon.ui.action.ActionHelper.render(ActionHelper.java:52)
    at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:101)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:36)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:172)
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1001)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
4

1 回答 1

1

您可以清理这些 tmp 文件,这些文件是在 Hadoop map reduce 作业执行期间以及特定作业完成后生成的。删除这些文件是安全的。因此,如果在过去 24 小时内未触摸或访问文件,则删除临时文件应该是安全的。您可以使用 cron 作业来执行此操作。见[1]。

[1] http://mail-archives.apache.org/mod_mbox/hadoop-general/201011.mbox/%3CABC24175AFD3BE4DA15F4CD375ED413D0639E23F45@hq-ex-mb02.ad.navteq.com%3E

谢谢,卡松。

于 2012-11-07T06:29:37.270 回答