3

在 Android 上，我想提取视频的所有帧并在每一帧上从 boofcv运行对象跟踪功能。因此，我使用了ExtractMpegFramesTest示例，并对其稍作调整以在每个帧上运行跟踪，而不是将帧保存为 png。即不是调用 outputSurface.saveFrame()，而是调用 outputSurface.processFrame()，我实现如下：

public void processFrame() {
    mPixelBuf.rewind();
    GLES20.glReadPixels(0, 0, mWidth, mHeight, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE,
                    mPixelBuf);

    Bitmap bmp = Bitmap.createBitmap(mWidth, mHeight, Bitmap.Config.ARGB_8888);
    mPixelBuf.rewind();
    bmp.copyPixelsFromBuffer(mPixelBuf);
    GrayU8 img = ConvertBitmap.bitmapToGray(bmp, (GrayU8)null, null);
    Log.d(TAG, "Running tracker...");
    tracker.process(img, location);
    barPath.add(getCenterOf(location));
    Log.d(TAG, "Finished running tracker.");
            bmp.recycle();
}

一旦我注释掉 tracker.process(img, location) 行，代码就可以完美运行。一旦我包含跟踪，我就没有收到错误消息，但也没有任何反应。应用程序在跟踪第一张图像时卡住了，logcat 输出如下所示：

2021-10-19 13:01:05.778 32449-32449/org.boofcv.android D/ExtractMpegFrames: Initialized
2021-10-19 13:01:05.780 32449-32449/org.boofcv.android I/Choreographer: Skipped 30 frames!  The application may be doing too much work on its main thread.
2021-10-19 13:01:05.789 32449-32449/org.boofcv.android W/Looper: Slow Looper main: doFrame is 514ms late because of 3 msg, msg 2 took 510ms (seq=242 running=153ms runnable=2ms late=18ms h=android.view.ViewRootImpl$ViewRootHandler c=android.view.View$PerformClick)
2021-10-19 13:01:05.793 32449-32726/org.boofcv.android D/ExtractMpegFrames: Extractor selected track 0 (video/avc): {track-id=1, file-format=video/mp4, level=8192, mime=video/avc, frame-count=624, profile=8, language=, color-standard=1, display-width=1920, csd-1=java.nio.HeapByteBuffer[pos=0 lim=9 cap=9], color-transfer=3, durationUs=10395844, display-height=1080, width=1920, color-range=2, rotation-degrees=90, max-input-size=1555201, frame-rate=60, height=1080, csd-0=java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]}
2021-10-19 13:01:05.794 32449-32726/org.boofcv.android D/ExtractMpegFrames: Video size is 1920x1080
2021-10-19 13:01:05.824 32449-32726/org.boofcv.android D/ExtractMpegFrames: textureID=1
2021-10-19 13:01:05.830 32449-32728/org.boofcv.android I/OMXClient: IOmx service obtained
2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: connecting to surface 0x7864f16010, reason connectToSurface
2021-10-19 13:01:05.888 32449-32727/org.boofcv.android I/MediaCodec: [OMX.qcom.video.decoder.avc] setting surface generation to 33227777
2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: disconnecting from surface 0x7864f16010, reason connectToSurface(reconnect)
2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: connecting to surface 0x7864f16010, reason connectToSurface(reconnect)
2021-10-19 13:01:05.890 32449-32728/org.boofcv.android I/ExtendedACodec: setupVideoDecoder()
2021-10-19 13:01:05.892 32449-32728/org.boofcv.android I/ExtendedACodec: Decoder will be in frame by frame mode
2021-10-19 13:01:05.929 32449-32728/org.boofcv.android D/SurfaceUtils: set up nativeWindow 0x7864f16010 for 1920x1080, color 0x7fa30c06, rotation 90, usage 0x20002900
2021-10-19 13:01:05.939 32449-32728/org.boofcv.android W/Gralloc3: allocator 3.x is not supported
2021-10-19 13:01:05.939 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
2021-10-19 13:01:05.946 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 0 to dec, size=148096
2021-10-19 13:01:05.958 32449-32726/org.boofcv.android D/ExtractMpegFrames: no output from decoder available
2021-10-19 13:01:05.958 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
2021-10-19 13:01:05.959 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 1 to dec, size=14112
2021-10-19 13:01:05.970 32449-32726/org.boofcv.android D/ExtractMpegFrames: no output from decoder available
2021-10-19 13:01:05.970 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
2021-10-19 13:01:05.972 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 2 to dec, size=17632
2021-10-19 13:01:05.974 32449-32726/org.boofcv.android D/ExtractMpegFrames: decoder output format changed: {crop-right=1919, color-format=2141391878, slice-height=1088, mime=video/raw, hdr-static-info=java.nio.HeapByteBuffer[pos=0 lim=25 cap=25], stride=1920, color-standard=2, color-transfer=3, crop-bottom=1079, crop-left=0, width=1920, color-range=1, crop-top=0, height=1088}
2021-10-19 13:01:05.974 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
2021-10-19 13:01:05.975 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 3 to dec, size=30208
2021-10-19 13:01:05.976 32449-32726/org.boofcv.android D/ExtractMpegFrames: surface decoder given buffer 16 (size=8)
2021-10-19 13:01:05.976 32449-32726/org.boofcv.android D/ExtractMpegFrames: awaiting decode of frame 0
2021-10-19 13:01:05.978 32449-32449/org.boofcv.android D/ExtractMpegFrames: new frame available
2021-10-19 13:01:05.992 32449-32726/org.boofcv.android D/ExtractMpegFrames: Running tracker...
2021-10-19 13:01:06.003 32449-32727/org.boofcv.android D/SurfaceUtils: disconnecting from surface 0x7864f16010, reason disconnectFromSurface

tracker.process() 函数本身可以完美运行，例如在使用 MediaMetadataRetriever 提取帧时（但是 MediaMetadataRetriever 太慢，这就是我使用 ExtractMpegFrames 的原因）。在我的设备上，tracker.process() 函数本身通常每帧需要大约 30-40 毫秒。只需使用 ExtractMpegFrames 将帧提取为位图，无需进一步处理，每帧大约需要 3-4 毫秒。

所以我猜这个问题可能与线程有关？我会感谢每一个帮助。

编辑

在我的 UI 线程上，我正在调用new ExtractMpegFrames().run()ExtractMpegFrames 类，如下所示：

public class ExtractMpegFrames {

    public void run() throws Throwable {
        this.initialize();
        ExtractMpegFramesWrapper.execute(this);
    }
    private static class ExtractMpegFramesWrapper implements Runnable {
        private Throwable mThrowable;
        private ExtractMpegFrames mExtractor;

        private ExtractMpegFramesWrapper(ExtractMpegFrames test) {
            mExtractor = test;
        }

        @Override
        public void run() {
            try {
                mExtractor.extractMpegFrames();
            } catch (Throwable th) {
                mThrowable = th;
            }
        }

        public static void execute(ExtractMpegFrames obj) throws Throwable {
            ExtractMpegFramesWrapper wrapper = new ExtractMpegFramesWrapper(obj);
            Thread th = new Thread(wrapper, "main");
            th.start();
            //th.join();
            if (wrapper.mThrowable != null) {
                throw wrapper.mThrowable;
            }
        }
    }
    ... 
    // The rest (except for the processFrame() function above) 
    // looks the same as in ExtractMpegFramesTest 
    // See here: 
    // https://bigflake.com/mediacodec/#ExtractMpegFramesTest
}

编辑 2

我仍然没有解决这个问题。我试图用一个 opencv （org.opencv.tracking.TrackerMOSSE）替换 boofcv 跟踪器。但是，这也会导致死锁（与上面的 logcat 输出相同）。最后，我尝试用这个 opencv 模板匹配算法替换跟踪功能。这有效并且不会导致死锁，但是它太慢了（在我的设备上每帧大约需要一秒钟）。

4

0 回答 0