2

We have server with 35gb memory and Intel® Xeon(R) E5-1620 0 @ 3.60GHz × 8 CPU. I am running a multithreaded program designed with akka actors and written in scala. In the program, there are 4 actors with tasks:

1) Lazy reading from file with Scala's BufferedSource and iterator,

2) Tokenizing sentences,

3) Calculating single and bigram words frequency for a given window size, and putting them into a map (one map for single words [String, Int], one for tuple words[WordTuple,Int),

4) Merging returned hasmaps into one hashmap and when all lines read from file and write them into a file.

My custom jvm settings is as follows:

-Xms34g

-Xmx34g

-XX:ReservedCodeCacheSize=240m

-XX:+UseParallelGC

-XX:ParallelGCThreads=4

-XX:NewSize=12g

-XX:SoftRefLRUPolicyMSPerMB=50

-ea

-Dsun.io.useCanonCaches=false

-Djava.net.preferIPv4Stack=true

-XX:+HeapDumpOnOutOfMemoryError

-XX:-OmitStackTraceInFastThrow

-Dawt.useSystemAAFontSettings=lcd

-Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine

-verbose:gc

-XX:+PrintGCDetails

-Xloggc:gc.log

My application.conf is as follows:

systemParameters {
  linesPerActor = 5
  windowSize = 6
  threadPoolSize = 5
}


akka.actor.deployment {

  /wordTokenizerRouter {
    router = round-robin-pool
    nr-of-instances = 5
  }

  /frequencyCalculatorRouter {
    router = round-robin-pool
    nr-of-instances = 5
  }
}

The problem:

I am processing a text file with size 15gb. Program starts working, after a while, say 2 hours, those tokenizing, calculating operations is almost not working, no operations can perform. The operation that takes 300 milliseconds starts taking 100000 seconds. But the cpu usage is %100 for all processors. I've tried using jvisualvm to motinor it but sampler is not working with this high cpu usage, so I could not identify which process is making cpu %100. I check gc activity from jvisualvm and most of the time it is using about %10 cpu. So, what could be the problem with my program, which process is possibly using all the cpu?

Here some screenshots from jvisualvm when operations in the program is stop but cpu usage is %100:

Garbage collector status screenshot

Overall status screenshot

Hope I explained it clear. Thanks in advance for your answers.

4

1 回答 1

0

我会研究几个领域。

  1. 您的堆看起来已满，包括老一代。另一个提示：在 8 小时 20 分钟的运行时间中，您的应用程序在 olg generation GC 中花费了 5 小时 45 分钟。当你堆满时，它会一个接一个地触发完整的 GC。使用并行 GC，这将在完整 GC 期间使用所有内核。看看你是 gc.log，看看有多少次完整的 GC 被触发。
  2. 在 CPU 负载期间，创建一些线程转储。您可以使用 VisualVM' 或 `jstack' 命令。在 Visual VM 中，它位于“线程”选项卡上的“线程转储”。查看堆栈转储并查找“可运行”线程，这些线程不在某些阻塞/IO API 中。看看他们在做什么。

如果它在垃圾收集上花费时间。我会进行堆转储并分析所持有的内存。您也可以在 VisualVm 的 Monitor 选项卡中进行堆转储，并在那里进行粗略分析。

于 2017-04-26T13:34:59.293 回答