5

我有一个用户上传图片的网络应用程序。我们通过 ImageIO.read() 验证图像数据,并对生成的 BufferedImage 执行一些简单的转换,然后将其保存到磁盘。

在执行负载测试时,我们意识到当许多请求同时进入时,它们会在 ImageIO.read() 调用中被阻塞。深入挖掘,我们注意到 JPEGImageReader 是同步的,并且一次只创建一个 BufferedImage。

有没有其他人遇到过这个?我已经在谷歌上搜索了几天了,还没有遇到另一个遇到这个问题的人,所以也许我做错了什么。我想不出任何合乎逻辑的理由。对于某些内存泄漏问题,这似乎与无法为每个图像创建单独的读取器和写入器有关,但这种解释对我来说似乎相当薄弱。

编辑:是一个性能工具,可以分解花费这么长时间的东西。我相信这是由于所有线程都在等待同步锁,JPEGImageReader源。

编辑:JAI 库本来可以工作,只是 OpenJDK 删除了对它的关键部分的支持,明确地是 JPEG 编解码器。

解决方案:考虑到我花费大量时间尝试寻找替代解决方案并且未能这样做,我最好的解决方案是针对请求异步处理图像。因此,当请求进来时,原始图像数据被存储为假定有效的图像;然后,请求线程之外的异步进程将一次处理每个图像。由于 ImageIO 库的同步性,尝试一次执行多个操作没有任何好处。鉴于库不是同步的,只是效率低下,因此可以并行处理图像。

虽然异步进行处理会增加一定程度的复杂性,但就修改图像而言,这可能是一个好主意。行不通的是我们无法在每个请求中处理原始图像,这意味着我们的系统必须假设每个图像都是有效的图像数据。当异步处理器确实开始处理图像时,如果数据不好,系统中可能会出现不一致。

4

2 回答 2

4

编辑:首先,haraldK 的答案是正确的,而我原来的答案是错误的。但是,它不能被删除,只有编辑过的 OP 才能重新分配刻度。

ThreadLock 机制是用来阻止线程从它们自己的读取器中访问不同的读取器,而不是阻止线程独立于其他线程并与其他线程同时访问它们自己的读取器。

因此 ImageIO 的 JPEGImageReader 应该可以异步工作。

于 2013-07-29T22:30:33.797 回答
2

在执行负载测试时,我们意识到当许多请求同时进入时,它们会在 ImageIO.read() 调用中被阻塞。深入挖掘,我们注意到 JPEGImageReader 是同步的,并且一次只创建一个 BufferedImage。

有没有其他人遇到过这个?

正如我在评论部分提到的:从性能分析来看,这看起来像是 OpenJDK (lcms) 的颜色管理系统的性能问题,因为颜色转换应该不会花这么长时间。我的猜测是,无法进一步调试,即使 ImageIO 能够并行解码,这个问题也会使解码(出现)同步。

这是一个 SSCCE,它显示 ImageIO 完全能够一次解码多个图像。

import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.event.IIOReadProgressListener;
import javax.imageio.stream.ImageInputStream;
import java.io.File;
import java.util.Iterator;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;

class MultipleJPEGDecoding {
    private static int threads = Runtime.getRuntime().availableProcessors();
    private static ExecutorService executorService = Executors.newFixedThreadPool(threads * 4);

    public static void main(final String[] args) throws InterruptedException {
        for (int i = 0; i < 100; i++) {
            final int index = i;

            executorService.submit(new Runnable() {
                public void run() {
                    try {
                        ImageInputStream stream = ImageIO.createImageInputStream(new File(args[index % args.length]));
                        try {
                            Iterator<ImageReader> readers = ImageIO.getImageReaders(stream);
                            if (!readers.hasNext()) {
                                System.err.println("No reader!");
                                return;
                            }

                            ImageReader reader = readers.next();
                            reader.setInput(stream);
                            reader.addIIOReadProgressListener(new ProgressListener(index));

                            try {
                                reader.read(0);
                            }
                            finally {
                                reader.dispose();
                            }
                        }
                        finally {
                            stream.close();
                        }
                    }
                    catch (Exception e) {
                        System.err.printf("Error reading %d\n", index);
                        e.printStackTrace();
                    }
                }
            });
        }

        executorService.shutdown();
    }

    static class ProgressListener implements IIOReadProgressListener {
        final static AtomicInteger simultaneous = new AtomicInteger(0);

        final int index;
        int nextProgress = 25;

        public ProgressListener(int index) {
            this.index = index;
        }

        public void imageStarted(ImageReader source, int imageIndex) {
            int inProgress = simultaneous.incrementAndGet();
            System.err.printf("Started reading image %d (now decoding %d images simultaneous)...\n", index, inProgress);
        }

        public void imageComplete(ImageReader source) {
            int inProgress = simultaneous.decrementAndGet();
            System.err.printf("Done reading image %d%s.\n", index, inProgress > 0 ? String.format(" (still decoding %d other images)", inProgress) : "");
        }

        public void imageProgress(ImageReader source, float percentageDone) {
            if (percentageDone > nextProgress) {
                int inProgress = simultaneous.get();
                System.err.printf("Progress on image %d (now decoding %d images simultaneous)...\n", index, inProgress);
                nextProgress += 25;
            }
        }

        public void sequenceStarted(ImageReader source, int minIndex) {
        }

        public void sequenceComplete(ImageReader source) {
        }

        public void thumbnailStarted(ImageReader source, int imageIndex, int thumbnailIndex) {
        }

        public void thumbnailProgress(ImageReader source, float percentageDone) {
        }

        public void thumbnailComplete(ImageReader source) {
        }

        public void readAborted(ImageReader source) {
        }
    }
}

输出如下所示:

Started reading image 5 (now decoding 2 images simultaneous)...
Started reading image 0 (now decoding 16 images simultaneous)...
Started reading image 14 (now decoding 15 images simultaneous)...
Started reading image 4 (now decoding 14 images simultaneous)...
Started reading image 13 (now decoding 1 images simultaneous)...
Started reading image 11 (now decoding 13 images simultaneous)...
Started reading image 2 (now decoding 12 images simultaneous)...
Started reading image 15 (now decoding 4 images simultaneous)...
Started reading image 8 (now decoding 11 images simultaneous)...
Started reading image 7 (now decoding 10 images simultaneous)...
Started reading image 9 (now decoding 9 images simultaneous)...
Started reading image 6 (now decoding 8 images simultaneous)...
Started reading image 10 (now decoding 7 images simultaneous)...
Started reading image 1 (now decoding 6 images simultaneous)...
Started reading image 3 (now decoding 5 images simultaneous)...
Started reading image 12 (now decoding 3 images simultaneous)...
Progress on image 11 (now decoding 16 images simultaneous)...
Progress on image 15 (now decoding 16 images simultaneous)...
Progress on image 13 (now decoding 16 images simultaneous)...
Progress on image 13 (now decoding 16 images simultaneous)...
Progress on image 9 (now decoding 16 images simultaneous)...
Progress on image 9 (now decoding 16 images simultaneous)...
Progress on image 1 (now decoding 16 images simultaneous)...
Progress on image 3 (now decoding 16 images simultaneous)...
Progress on image 1 (now decoding 16 images simultaneous)...
Progress on image 3 (now decoding 16 images simultaneous)...
Progress on image 1 (now decoding 16 images simultaneous)...
Progress on image 13 (now decoding 16 images simultaneous)...
Progress on image 5 (now decoding 16 images simultaneous)...
Progress on image 9 (now decoding 16 images simultaneous)...
Progress on image 3 (now decoding 16 images simultaneous)...
Done reading image 3 (still decoding 15 other images).
Started reading image 16 (now decoding 15 images simultaneous)...
Progress on image 11 (now decoding 15 images simultaneous)...
Done reading image 13 (still decoding 14 other images).
Started reading image 17 (now decoding 15 images simultaneous)...
Progress on image 5 (now decoding 15 images simultaneous)...
Progress on image 4 (now decoding 15 images simultaneous)...
Progress on image 11 (now decoding 15 images simultaneous)...
Done reading image 9 (still decoding 14 other images).
Progress on image 5 (now decoding 15 images simultaneous)...
Progress on image 17 (now decoding 15 images simultaneous)...
Done reading image 11 (still decoding 14 other images).
Started reading image 19 (now decoding 15 images simultaneous)...
Progress on image 17 (now decoding 15 images simultaneous)...
Done reading image 5 (still decoding 14 other images).
Started reading image 18 (now decoding 15 images simultaneous)...
Progress on image 15 (now decoding 15 images simultaneous)...
Done reading image 1 (still decoding 14 other images).
Started reading image 21 (now decoding 15 images simultaneous)...
Progress on image 15 (now decoding 15 images simultaneous)...
Progress on image 21 (now decoding 15 images simultaneous)...
Done reading image 15 (still decoding 14 other images).
Progress on image 6 (now decoding 14 images simultaneous)...
Progress on image 21 (now decoding 14 images simultaneous)...
Progress on image 7 (now decoding 14 images simultaneous)...
Progress on image 21 (now decoding 14 images simultaneous)...
Progress on image 7 (now decoding 14 images simultaneous)...
Progress on image 19 (now decoding 14 images simultaneous)...
Progress on image 19 (now decoding 14 images simultaneous)...
Progress on image 7 (now decoding 14 images simultaneous)...
Progress on image 19 (now decoding 14 images simultaneous)...
Done reading image 7 (still decoding 13 other images).
Started reading image 23 (now decoding 14 images simultaneous)...
Progress on image 10 (now decoding 14 images simultaneous)...
Progress on image 23 (now decoding 14 images simultaneous)...
Progress on image 14 (now decoding 14 images simultaneous)...
Started reading image 22 (now decoding 15 images simultaneous)...
Progress on image 0 (now decoding 15 images simultaneous)...
Done reading image 21 (still decoding 14 other images).
Started reading image 24 (now decoding 15 images simultaneous)...
Started reading image 20 (now decoding 16 images simultaneous)...
Progress on image 2 (now decoding 16 images simultaneous)...
Progress on image 8 (now decoding 16 images simultaneous)...
Progress on image 17 (now decoding 15 images simultaneous)...
Done reading image 17 (still decoding 14 other images).
Started reading image 25 (now decoding 15 images simultaneous)...
Progress on image 23 (now decoding 15 images simultaneous)...
Done reading image 19 (still decoding 15 other images).
Started reading image 26 (now decoding 16 images simultaneous)...
Progress on image 23 (now decoding 16 images simultaneous)...
Done reading image 23 (still decoding 15 other images).
Started reading image 27 (now decoding 16 images simultaneous)...
Progress on image 4 (now decoding 16 images simultaneous)...
Progress on image 27 (now decoding 16 images simultaneous)...
Progress on image 27 (now decoding 16 images simultaneous)...
Progress on image 6 (now decoding 16 images simultaneous)...
Progress on image 12 (now decoding 16 images simultaneous)...
Progress on image 20 (now decoding 16 images simultaneous)...
Progress on image 0 (now decoding 16 images simultaneous)...
Progress on image 25 (now decoding 16 images simultaneous)...
Progress on image 25 (now decoding 16 images simultaneous)...
Progress on image 25 (now decoding 16 images simultaneous)...
Progress on image 14 (now decoding 16 images simultaneous)...
Progress on image 10 (now decoding 16 images simultaneous)...
Progress on image 8 (now decoding 16 images simultaneous)...
Progress on image 18 (now decoding 16 images simultaneous)...
Done reading image 25 (still decoding 15 other images).

[...]

Progress on image 75 (now decoding 12 images simultaneous)...
Started reading image 73 (now decoding 13 images simultaneous)...
Progress on image 75 (now decoding 13 images simultaneous)...
Progress on image 73 (now decoding 13 images simultaneous)...
Progress on image 75 (now decoding 13 images simultaneous)...
Started reading image 74 (now decoding 14 images simultaneous)...
Progress on image 66 (now decoding 14 images simultaneous)...
Progress on image 64 (now decoding 14 images simultaneous)...
Progress on image 73 (now decoding 14 images simultaneous)...
Progress on image 60 (now decoding 14 images simultaneous)...
Progress on image 74 (now decoding 14 images simultaneous)...
Progress on image 58 (now decoding 14 images simultaneous)...
Done reading image 75 (still decoding 13 other images).
Progress on image 73 (now decoding 13 images simultaneous)...
Started reading image 77 (now decoding 14 images simultaneous)...
Done reading image 73 (still decoding 13 other images).
Progress on image 77 (now decoding 13 images simultaneous)...
Started reading image 78 (now decoding 14 images simultaneous)...
Progress on image 60 (now decoding 14 images simultaneous)...
Done reading image 48 (still decoding 13 other images).
Progress on image 77 (now decoding 13 images simultaneous)...
Started reading image 79 (now decoding 14 images simultaneous)...
Started reading image 70 (now decoding 15 images simultaneous)...
Progress on image 52 (now decoding 15 images simultaneous)...
Progress on image 71 (now decoding 15 images simultaneous)...
Started reading image 72 (now decoding 16 images simultaneous)...
Progress on image 71 (now decoding 16 images simultaneous)...
Progress on image 54 (now decoding 16 images simultaneous)...
Progress on image 68 (now decoding 16 images simultaneous)...
Progress on image 64 (now decoding 16 images simultaneous)...
Progress on image 66 (now decoding 16 images simultaneous)...
Progress on image 62 (now decoding 16 images simultaneous)...
Progress on image 79 (now decoding 16 images simultaneous)...
Progress on image 79 (now decoding 16 images simultaneous)...
Progress on image 79 (now decoding 16 images simultaneous)...
Progress on image 77 (now decoding 16 images simultaneous)...
Progress on image 68 (now decoding 16 images simultaneous)...
Done reading image 79 (still decoding 15 other images).
Done reading image 77 (still decoding 14 other images).
Started reading image 81 (now decoding 15 images simultaneous)...
Progress on image 74 (now decoding 15 images simultaneous)...
Progress on image 81 (now decoding 15 images simultaneous)...
Progress on image 81 (now decoding 15 images simultaneous)...
Progress on image 78 (now decoding 15 images simultaneous)...
Done reading image 60 (still decoding 14 other images).
Started reading image 82 (now decoding 15 images simultaneous)...
Started reading image 80 (now decoding 16 images simultaneous)...
Progress on image 76 (now decoding 14 images simultaneous)...
Progress on image 66 (now decoding 14 images simultaneous)...
Progress on image 70 (now decoding 14 images simultaneous)...
Done reading image 52 (still decoding 14 other images).
Done reading image 71 (still decoding 15 other images).
Progress on image 81 (now decoding 16 images simultaneous)...
Started reading image 84 (now decoding 15 images simultaneous)...
Started reading image 83 (now decoding 16 images simultaneous)...
Progress on image 58 (now decoding 16 images simultaneous)...
Progress on image 83 (now decoding 16 images simultaneous)...
Progress on image 83 (now decoding 16 images simultaneous)...
Progress on image 83 (now decoding 16 images simultaneous)...
Done reading image 81 (still decoding 15 other images).
Started reading image 85 (now decoding 16 images simultaneous)...
Progress on image 85 (now decoding 16 images simultaneous)...
Progress on image 74 (now decoding 16 images simultaneous)...
Done reading image 66 (still decoding 15 other images).
Started reading image 86 (now decoding 16 images simultaneous)...
Progress on image 64 (now decoding 16 images simultaneous)...
Progress on image 70 (now decoding 16 images simultaneous)...
Progress on image 78 (now decoding 16 images simultaneous)...
Progress on image 54 (now decoding 16 images simultaneous)...
Done reading image 58 (still decoding 15 other images).
Started reading image 87 (now decoding 16 images simultaneous)...
Progress on image 87 (now decoding 16 images simultaneous)...
Progress on image 84 (now decoding 16 images simultaneous)...
Progress on image 87 (now decoding 16 images simultaneous)...
Done reading image 64 (still decoding 15 other images).
Started reading image 88 (now decoding 16 images simultaneous)...
Progress on image 76 (now decoding 16 images simultaneous)...
Done reading image 83 (still decoding 15 other images).
Progress on image 62 (now decoding 15 images simultaneous)...
Progress on image 70 (now decoding 16 images simultaneous)...
Progress on image 85 (now decoding 16 images simultaneous)...
Started reading image 89 (now decoding 16 images simultaneous)...
Progress on image 72 (now decoding 16 images simultaneous)...
Progress on image 85 (now decoding 16 images simultaneous)...
Progress on image 89 (now decoding 16 images simultaneous)...
Done reading image 85 (still decoding 15 other images).
Progress on image 89 (now decoding 15 images simultaneous)...
Progress on image 82 (now decoding 15 images simultaneous)...
Progress on image 80 (now decoding 15 images simultaneous)...
Done reading image 74 (still decoding 14 other images).
Started reading image 91 (now decoding 15 images simultaneous)...
Started reading image 90 (now decoding 16 images simultaneous)...
Done reading image 62 (still decoding 15 other images).
Progress on image 87 (now decoding 15 images simultaneous)...
Progress on image 68 (now decoding 15 images simultaneous)...
Done reading image 87 (still decoding 14 other images).
Progress on image 91 (now decoding 14 images simultaneous)...
Started reading image 93 (now decoding 15 images simultaneous)...
Progress on image 93 (now decoding 15 images simultaneous)...
Progress on image 91 (now decoding 15 images simultaneous)...
Progress on image 91 (now decoding 15 images simultaneous)...
Progress on image 80 (now decoding 15 images simultaneous)...
Started reading image 92 (now decoding 16 images simultaneous)...
Done reading image 91 (still decoding 15 other images).
Started reading image 94 (now decoding 16 images simultaneous)...
Progress on image 93 (now decoding 16 images simultaneous)...
Progress on image 72 (now decoding 16 images simultaneous)...
Progress on image 93 (now decoding 16 images simultaneous)...
Done reading image 93 (still decoding 15 other images).
Started reading image 95 (now decoding 16 images simultaneous)...
Progress on image 95 (now decoding 16 images simultaneous)...
Progress on image 95 (now decoding 16 images simultaneous)...
Progress on image 76 (now decoding 16 images simultaneous)...
Progress on image 95 (now decoding 16 images simultaneous)...
Progress on image 72 (now decoding 16 images simultaneous)...
Done reading image 95 (still decoding 15 other images).
Started reading image 96 (now decoding 16 images simultaneous)...
Progress on image 94 (now decoding 16 images simultaneous)...
Progress on image 89 (now decoding 15 images simultaneous)...
Done reading image 89 (still decoding 14 other images).
Done reading image 54 (still decoding 15 other images).
Started reading image 97 (now decoding 14 images simultaneous)...
Started reading image 98 (now decoding 15 images simultaneous)...
Done reading image 70 (still decoding 13 other images).
Started reading image 99 (now decoding 16 images simultaneous)...
Progress on image 82 (now decoding 16 images simultaneous)...
Progress on image 99 (now decoding 16 images simultaneous)...
Progress on image 99 (now decoding 16 images simultaneous)...
Progress on image 97 (now decoding 16 images simultaneous)...
Progress on image 97 (now decoding 16 images simultaneous)...
Progress on image 97 (now decoding 16 images simultaneous)...
Done reading image 68 (still decoding 15 other images).
Done reading image 97 (still decoding 14 other images).
Progress on image 78 (now decoding 14 images simultaneous)...
Progress on image 99 (now decoding 14 images simultaneous)...
Done reading image 99 (still decoding 13 other images).
Progress on image 86 (now decoding 13 images simultaneous)...
Done reading image 72 (still decoding 12 other images).
Progress on image 82 (now decoding 12 images simultaneous)...
Progress on image 98 (now decoding 12 images simultaneous)...
Progress on image 84 (now decoding 12 images simultaneous)...
Progress on image 90 (now decoding 12 images simultaneous)...
Done reading image 76 (still decoding 11 other images).
Progress on image 92 (now decoding 11 images simultaneous)...
Progress on image 80 (now decoding 11 images simultaneous)...
Progress on image 94 (now decoding 11 images simultaneous)...
Progress on image 88 (now decoding 11 images simultaneous)...
Progress on image 84 (now decoding 11 images simultaneous)...
Progress on image 90 (now decoding 11 images simultaneous)...
Progress on image 92 (now decoding 11 images simultaneous)...
Progress on image 86 (now decoding 11 images simultaneous)...
Progress on image 94 (now decoding 10 images simultaneous)...
Done reading image 84 (still decoding 9 other images).
Progress on image 92 (now decoding 9 images simultaneous)...
Done reading image 78 (still decoding 10 other images).
Progress on image 88 (now decoding 9 images simultaneous)...
Progress on image 90 (now decoding 9 images simultaneous)...
Done reading image 80 (still decoding 8 other images).
Done reading image 82 (still decoding 7 other images).
Progress on image 86 (now decoding 7 images simultaneous)...
Progress on image 96 (now decoding 7 images simultaneous)...
Progress on image 88 (now decoding 7 images simultaneous)...
Done reading image 90 (still decoding 6 other images).
Done reading image 92 (still decoding 5 other images).
Progress on image 98 (now decoding 5 images simultaneous)...
Done reading image 94 (still decoding 4 other images).
Done reading image 86 (still decoding 3 other images).
Progress on image 96 (now decoding 3 images simultaneous)...
Done reading image 88 (still decoding 2 other images).
Progress on image 98 (now decoding 2 images simultaneous)...
Progress on image 96 (now decoding 2 images simultaneous)...
Done reading image 98 (still decoding 1 other images).
Done reading image 96.
于 2013-08-05T14:20:15.480 回答