1

As already answered in this question: JVM_FindSignal function continuously allocates native memory

jemalloc reporting leaks from JVM_FindSignal is related to missing debug symbols. I certainly have debugging symbols installed, see:

rbs42@rbs42-VirtualBox:/usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/server$ gdb libjvm.so -ex 'info address UseG1GC'
GNU gdb (Ubuntu 9.1-0ubuntu1) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from libjvm.so...
Reading symbols from /usr/lib/debug/.build-id/16/240e0172c3fc0dd6e974325c8ad1d93723ccac.debug...
(No debugging symbols found in /usr/lib/debug/.build-id/16/240e0172c3fc0dd6e974325c8ad1d93723ccac.debug)
Installing openjdk unwinder
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 52, in <module>
    class Types(object):
  File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 66, in Types
    nmethodp_t = gdb.lookup_type('nmethod').pointer()
gdb.error: No type named nmethod.
Symbol "UseG1GC" is at 0xd189b2 in a file compiled without debugging.

still my jeprof output looks as following:

rbs42@rbs42-VirtualBox:/media/rbs42/data/Gebos/RBS42/run/sms50$ jeprof --show_bytes /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java jeprof.22104.0.f.heap 
Using local file /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java.
Using local file jeprof.22104.0.f.heap.
Welcome to jeprof!  For help, type 'help'.
(jeprof) top
Total: 33502504 B
20958503  62.6%  62.6% 21342909  63.7% JVM_FindSignal
 8388608  25.0%  87.6%  8388608  25.0% SNX11B1A
 1481379   4.4%  92.0%  1481379   4.4% inflate
 1151253   3.4%  95.5%  1151253   3.4% Java_java_util_zip_ZipFile_getZipMessage
  426303   1.3%  96.7%   426303   1.3% SNE00B1A
  404065   1.2%  97.9%   404065   1.2% inflateInit2_
  253077   0.8%  98.7% 20393297  60.9% SUNWprivate_1.1
  176271   0.5%  99.2%   176271   0.5% std::__throw_ios_failure
  131713   0.4%  99.6%   131713   0.4% _dl_new_object
  131328   0.4% 100.0%   131328   0.4% _dl_check_map_versions
(jeprof)

Is there anything else to consider?

4

2 回答 2

3

It turns out that

  1. openjdk-8-dbg package installs files with debug symbols into /usr/lib/debug/.build-id
  2. jeprof looks for debug symbols in /usr/lib/debug/{FULL_SO_PATH}

So, it's a combination of a bug in jeprof that does not parse .note.gnu.build-id section, and a problem of the dbg package that does not include full path symlinks to debug libraries.

To work around this, you may create the corresponding symlink manually:

ln -s /usr/lib/debug/.build-id/16/240e0172c3fc0dd6e974325c8ad1d93723ccac.debug /usr/lib/debug/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so

(the file names are taken from gdb output)

However, even when jeprof is able to read debug symbols, this doesn't often help in case of Java applications. The problem is that jemalloc doesn't know how to unwind Java stacks. See this presentation for an example.

Consider trying async-profiler that can show mixed Java+native stacks. Profiling malloc, mprotect and mmap calls with async-profiler can be helpful in finding native memory leaks in Java applications. See this answer for details.

于 2020-09-24T00:44:59.167 回答
0

Here is a patch for jeprof that makes it smarter about finding debug symbols. It's based on @apangin's answer.

https://github.com/jemalloc/jemalloc/pull/2059/files

于 2021-04-19T04:05:47.007 回答