4

我正在做一个项目，我将在其中拥有不同的捆绑包。让我们举个例子，假设我有 5 个 Bundles，每个包都有一个方法 name process。

现在，我正在process使用下面的多线程代码并行调用所有这 5 个捆绑包的方法。

但不知何故，每次我运行下面的多线程代码时，它总是给我内存不足的异常。但是如果我按顺序运行它意味着一个一个地调用进程方法，那么它不会给我任何内存不足的异常。

下面是代码-

public void callBundles(final Map<String, Object> eventData) {

    // Three threads: one thread for the database writer, two threads for the plugin processors
    final ExecutorService executor = Executors.newFixedThreadPool(3);

    final Map<String, String> outputs = (Map<String, String>)eventData.get(Constants.EVENT_HOLDER);

    for (final BundleRegistration.BundlesHolderEntry entry : BundleRegistration.getInstance()) {
        executor.submit(new Runnable () {
            public void run() {
                try {
                    final Map<String, String> response = entry.getPlugin().process(outputs);

                    //process the response and update database.
                    System.out.println(response);
                } catch (Exception e) {
                    e.printStackTrace();
                }               
            }
        });
    }
}

下面是一个例外，每当我在多线程代码上运行时，我都会得到。

JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
JVMDUMP032I JVM requested Heap dump using 'S:\GitViews\Stream\goldseye\heapdump.20130904.175256.12608.0001.phd' in response to an event
JVMDUMP010I Heap dump written to S:\GitViews\Stream\goldseye\heapdump.20130904.175256.12608.0001.phd
JVMDUMP032I JVM requested Java dump using 'S:\GitViews\Stream\goldseye\javacore.20130904.175256.12608.0002.txt' in response to an event
UTE430: can't allocate buffer
UTE437: Unable to load formatStrings for j9mm
JVMDUMP010I Java dump written to S:\GitViews\Stream\goldseye\javacore.20130904.175256.12608.0002.txt
JVMDUMP032I JVM requested Snap dump using 'S:\GitViews\Stream\goldseye\Snap.20130904.175256.12608.0003.trc' in response to an event
UTE001: Error starting trace thread for "Snap Dump Thread": -1
JVMDUMP010I Snap dump written to S:\GitViews\Stream\goldseye\Snap.20130904.175256.12608.0003.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
ERROR: Bundle BullseyeModellingFramework [1] EventDispatcher: Error during dispatch. (java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 12)
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 12

JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
JVMDUMP032I JVM requested Heap dump using 'S:\GitViews\Stream\goldseye\heapdump.20130904.175302.12608.0004.phd' in response to an event
JVMDUMP010I Heap dump written to S:\GitViews\Stream\goldseye\heapdump.20130904.175302.12608.0004.phd
JVMDUMP032I JVM requested Java dump using 'S:\GitViews\Stream\goldseye\javacore.20130904.175302.12608.0005.txt' in response to an event

我JDK1.6.0_26在我的eclipse中使用作为安装的JRE。

4

2 回答 2

2

每次调用callBundles()都会通过创建一个自己的执行器来创建一个新的线程池。每个线程都有自己的堆栈空间！所以如果你说你启动 JVM，第一次调用将创建三个线程，总堆为 3M（1024k 是 64 位 JVM 的默认堆栈大小），下一次调用另一个 3M 等等。1000 次调用/s 将需要3GB/秒！

第二个问题是您永远不会shutdown()创建执行程序服务，因此线程将继续存在，直到垃圾收集器删除执行程序（finalize()也调用shutdown()）。但是 GC 永远不会清除栈内存，所以如果栈内存有问题，堆没有满的话，GC 永远也帮不上忙！

您需要使用一个 ExecutorService，假设使用 10 到 30 个线程或自定义ThreadPoolExecutor使用 3-30 个缓存线程和一个LinkedBlockingQueue. shutdown()如果可能，请在您的应用程序停止之前调用该服务。

检查应用程序的物理 RAM、负载和响应时间，以调整参数堆大小、最大线程数和池中线程的保持活动时间。查看代码的其他锁定部分（数据库连接池的大小，...）以及服务器的 CPU/内核数。线程池大小的起点可能是 CPU/核心数加 1。I/O 等待越多越有用。

于 2013-09-18T17:34:18.137 回答
1

主要问题是您并没有真正正确地使用线程池。如果您所有的“进程”线程都具有相同的优先级，那么没有充分的理由不创建一个大型线程池并将您的所有 Runnable 任务提交给它。注意 - 在这种情况下，“大”是通过实验和分析确定的：调整它直到您在速度和内存方面的表现达到您的预期。

这是我所描述的示例：

// Using 10000 purely as a concrete example - you should define the correct number
public static final LARGE_NUMBER_OF_THREADS = 10000;
// Elsewhere in code, you defined a static thread pool
public static final ExecutorService EXECUTOR = 
    Executors.newFixedThreadPool(LARGE_NUMBER_OF_THREADS);

public void callBundles(final Map<String, Object> eventData) {

final Map<String, String> outputs = 
    (Map<String, String>)eventData.get(Constants.EVENT_HOLDER);

for (final BundleRegistration.BundlesHolderEntry entry : BundleRegistration.getInstance()) {
    // "Three threads: one thread for the database writer, 
    // two threads for the plugin processors"
    // so you'll need to repeat this future = E.submit() pattern two more times
    Future<?> processFuture = EXECUTOR.submit(new Runnable() {
        public void run() {
            final Map<String, String> response = 
                entry.getPlugin().process(outputs);
            //process the response and update database.
            System.out.println(response);
        }
    }
    // Note, I'm catching the exception out here instead of inside the task
    // This also allows me to force order on the three component threads
    try {
        processFuture.get();
    } catch (Exception e) {
        System.err.println("Should really do something more useful");
        e.printStackTrace();
    }
    // If you wanted to ensure that the three component tasks run in order, 
    // you could future = executor.submit(); future.get(); 
    // for each one of them
}

为了完整起见，您还可以使用缓存线程池来避免重复创建短期线程。但是，如果您已经担心内存消耗，那么固定池可能会更好。

当您使用 Java 7 时，您可能会发现Fork-Join是比一系列 Futures 更好的模式。不过，无论哪种方式最适合您的需求。

于 2013-09-18T16:48:32.157 回答