4

我想在逐帧模式下播放视频（主要是带有 Motion JPEG 的 .mov）并改变帧速率。我有一个函数给我一个帧号，然后我必须跳到那里。它主要是在一个方向上，但有时会跳过几帧；速度也不是恒定的。所以我有一个计时器每 40 毫秒询问一个新的帧号并设置新的位置。我现在的第一种方法是使用 DirectShow.Net (Interop.QuartzTypeLib)。因此，我渲染并打开视频并将其设置为暂停以在图表中绘制图片

    FilgraphManagerClass media = new FilgraphManagerClass();
    media.RenderFile(FileName);
    media.pause();

现在我将设置一个新位置

    media.CurrentPosition = framenumber * media.AvgTimePerFrame;

由于视频处于暂停模式，因此它将绘制每个请求的新位置（帧）。工作得很好，但真的很慢......视频一直卡顿和滞后，而不是视频源；录制的帧数足以播放流畅的视频。通过一些性能测试，我发现 LAV-Codec 是这里的瓶颈。这不直接包含在我的项目中，因为它是一个 DirectShow-Player，它将通过我安装在我的 PC 上的编解码器包进行投射。

想法：

  • 我自己直接在 C# 中使用 LAV-Codec。我进行了搜索，但似乎每个人都在使用 DirectShow，构建自己的过滤器，而不是直接在项目中使用现有的过滤器。
  • 我可以仅通过帧号获取单帧并简单地绘制它们，而不是寻找或设置时间吗？
  • 有没有其他完整的方法来归档我想做的事情？

背景：

这个项目必须是一个火车模拟器。我们录制了从驾驶舱内行驶的火车的实时视频，并知道哪个框架是哪个位置。现在我的 C# 程序根据时间和加速度计算火车的位置，返回适当的帧号并绘制该帧。

附加信息：

在 C/C++ 中还有另一个项目（不是我写的），它直接使用 DirectShow 和 avcodec-LAV，其方式与我类似，而且效果很好！那是因为我有想法自己使用像 avrcodec-lav 这样的编解码器/过滤器。但我找不到与 C# 一起使用的互操作或接口。

感谢大家阅读本文并提供帮助！:)

4

2 回答 2

0

Obtaining specific frame by seeking filter graph (the entire pipeline) is pretty slow since every seek operation involves the following on its backyard: flushing everything, possibly re-creating worker threads, seeking to first key frame/splice point/clean point/I-Frame before the requested time, start of decoding starting from found position skipping frames until originally requested time is reached.

Overall, the method works well when you scrub paused video, or retrieve specific still frames. When however you try to play this as smooth video, it eventually causes significant part of the effort to be wasted and spent on seeking within video stream.

Solutions here are:

  • re-encode video to remove or reduce temporal compression (e.g. Motion JPEG AVI/MOV/MP4 files)
  • whenever possible prefer to skip frames and/or re-timestamp them according to your algorithm instead of seeking
  • have a cached of decoded video frames and pick from there, populate them as necessary in worker thread

The latter two are unfortunately hard to achieve without advanced filter development (where continuous decoding without interruption by seeking operations is the key to achieving decent performance). With basic DirectShow.Net you only have basic control over streaming and hence the first item from the list above.

于 2015-06-17T09:49:39.987 回答
0

想发表评论而不是答案，但没有声誉。我认为你在 Direct Show 上的方向是错误的。几年来，我一直在 C# 和 Android 之间使用 motion-jpeg，并且通过内置的 .NET 代码（用于将字节数组转换为 Jpeg 帧）和一些多线程获得了出色的性能。我可以很容易地从多个设备上实现超过 30fps 的速度，每个设备都在它自己的线程中运行。

下面是我的 C# 应用程序“OmniView”中的motion-jpeg 解析器的旧版本。要使用，只需将网络流发送到构造函数，并接收 OnImageReceived 事件。然后，您可以轻松地将帧保存到硬盘驱动器以供以后使用（可能将文件名设置为时间戳以便于查找）。但是，为了获得更好的性能，您需要将所有图像保存到一个文件中。

using OmniView.Framework.Helpers;
using System;
using System.IO;
using System.Text;
using System.Windows.Media.Imaging;

namespace OmniView.Framework.Devices.MJpeg
{
    public class MJpegStream : IDisposable
    {
        private const int BUFFER_SIZE = 4096;
        private const string tag_length = "Content-Length:";
        private const string stamp_format = "yyyyMMddHHmmssfff";

        public delegate void ImageReceivedEvent(BitmapImage img);
        public delegate void FrameCountEvent(long frames, long failed);
        public event ImageReceivedEvent OnImageReceived;
        public event FrameCountEvent OnFrameCount;

        private bool isHead, isSetup;
        private byte[] buffer, newline, newline_src;
        private int imgBufferStart;

        private Stream data_stream;
        private MemoryStream imgStreamA, imgStreamB;
        private int headStart, headStop;
        private long imgSize, imgSizeTgt;
        private bool useStreamB;

        public volatile bool EnableRecording, EnableSnapshot;
        public string RecordPath, SnapshotFilename;

        private string boundary_tag;
        private bool tagReadStarted;
        private bool enableBoundary;

        public volatile bool OututFrameCount;
        private long FrameCount, FailedCount;


        public MJpegStream() {
            isSetup = false;
            imgStreamA = new MemoryStream();
            imgStreamB = new MemoryStream();
            buffer = new byte[BUFFER_SIZE];
            newline_src = new byte[] {13, 10};
        }

        public void Init(Stream stream) {
            this.data_stream = stream;
            FrameCount = FailedCount = 0;
            startHeader(0);
        }

        public void Dispose() {
            if (data_stream != null) data_stream.Dispose();
            if (imgStreamA != null) imgStreamA.Dispose();
            if (imgStreamB != null) imgStreamB.Dispose();
        }

        //=============================

        public void Process() {
            if (isHead) processHeader();
            else {
                if (enableBoundary) processImageBoundary();
                else processImage();
            }
        }

        public void Snapshot(string filename) {
            SnapshotFilename = filename;
            EnableSnapshot = true;
        }

        //-----------------------------
        // Header

        private void startHeader(int remaining_bytes) {
            isHead = true;
            headStart = 0;
            headStop = remaining_bytes;
            imgSizeTgt = 0;
            tagReadStarted = false;
        }

        private void processHeader() {
            int t = BUFFER_SIZE - headStop;
            headStop += data_stream.Read(buffer, headStop, t);
            int nl;
            //
            if (!isSetup) {
                byte[] new_newline;
                if ((nl = findNewline(headStart, headStop, out new_newline)) >= 0) {
                    string tag = Encoding.UTF8.GetString(buffer, headStart, nl - headStart);
                    if (tag.StartsWith("--")) boundary_tag = tag;
                    headStart = nl+new_newline.Length;
                    newline = new_newline;
                    isSetup = true;
                    return;
                }
            } else {
                while ((nl = findData(newline, headStart, headStop)) >= 0) {
                    string tag = Encoding.UTF8.GetString(buffer, headStart, nl - headStart);
                    if (!tagReadStarted && tag.Length > 0) tagReadStarted = true;
                    headStart = nl+newline.Length;
                    //
                    if (!processHeaderData(tag, nl)) return;
                }
            }
            //
            if (headStop >= BUFFER_SIZE) {
                string data = Encoding.UTF8.GetString(buffer, headStart, headStop - headStart);
                throw new Exception("Invalid Header!");
            }
        }

        private bool processHeaderData(string tag, int index) {
            if (tag.StartsWith(tag_length)) {
                string val = tag.Substring(tag_length.Length);
                imgSizeTgt = long.Parse(val);
            }
            //
            if (tag.Length == 0 && tagReadStarted) {
                if (imgSizeTgt > 0) {
                    finishHeader(false);
                    return false;
                }
                if (boundary_tag != null) {
                    finishHeader(true);
                    return false;
                }
            }
            //
            return true;
        }

        private void finishHeader(bool enable_boundary) {
            int s = shiftBytes(headStart, headStop);
            enableBoundary = enable_boundary;
            startImage(s);
        }

        //-----------------------------
        // Image

        private void startImage(int remaining_bytes) {
            isHead = false;
            imgBufferStart = remaining_bytes;
            Stream imgStream = getStream();
            imgStream.Seek(0, SeekOrigin.Begin);
            imgStream.SetLength(imgSizeTgt);
            imgSize = 0;
        }

        private void processImage() {
            long img_r = (imgSizeTgt - imgSize - imgBufferStart);
            int bfr_r = Math.Max(BUFFER_SIZE - imgBufferStart, 0);
            int t = (int)Math.Min(img_r, bfr_r);
            int s = data_stream.Read(buffer, imgBufferStart, t);
            int x = imgBufferStart + s;
            appendImageData(0, x);
            imgBufferStart = 0;
            //
            if (imgSize >= imgSizeTgt) processImageData(0);
        }

        private void processImageBoundary() {
            int t = Math.Max(BUFFER_SIZE - imgBufferStart, 0);
            int s = data_stream.Read(buffer, imgBufferStart, t);
            //
            int nl, start = 0;
            int end = imgBufferStart + s;
            while ((nl = findData(newline, start, end)) >= 0) {
                int tag_length = boundary_tag.Length;
                if (nl+newline.Length+tag_length > BUFFER_SIZE) {
                    appendImageData(start, nl+newline.Length - start);
                    start = nl+newline.Length;
                    continue;
                }
                //
                string v = Encoding.UTF8.GetString(buffer, nl+newline.Length, tag_length);
                if (v == boundary_tag) {
                    appendImageData(start, nl - start);
                    int xstart = nl+newline.Length + tag_length;
                    int xsize = shiftBytes(xstart, end);
                    processImageData(xsize);
                    return;
                } else {
                    appendImageData(start, nl+newline.Length - start);
                }
                start = nl+newline.Length;
            }
            //
            if (start < end) {
                int end_x = end - newline.Length;
                if (start < end_x) {
                    appendImageData(start, end_x - start);
                }
                //
                shiftBytes(end - newline.Length, end);
                imgBufferStart = newline.Length;
            }
        }

        private void processImageData(int remaining_bytes) {
            if (EnableSnapshot) {
                EnableSnapshot = false;
                saveSnapshot();
            }
            //
            try {
                BitmapImage img = createImage();
                if (EnableRecording) recordFrame();
                if (OnImageReceived != null) OnImageReceived.Invoke(img);
                FrameCount++;
            }
            catch (Exception) {
                // output frame error ?!
                FailedCount++;
            }
            //
            if (OututFrameCount && OnFrameCount != null) OnFrameCount.Invoke(FrameCount, FailedCount);
            //
            useStreamB = !useStreamB;
            startHeader(remaining_bytes);
        }

        private void appendImageData(int index, int length) {
            Stream imgStream = getStream();
            imgStream.Write(buffer, index, length);
            imgSize += (length - index);
        }

        //-----------------------------

        private void recordFrame() {
            string stamp = DateTime.Now.ToString(stamp_format);
            string filename = RecordPath+"\\"+stamp+".jpg";
            //
            ImageHelper.Save(getStream(), filename);
        }

        private void saveSnapshot() {
            Stream imgStream = getStream();
            //
            imgStream.Position = 0;
            Stream file = File.Open(SnapshotFilename, FileMode.Create, FileAccess.Write);
            try {imgStream.CopyTo(file);}
            finally {file.Close();}
        }

        private BitmapImage createImage() {
            Stream imgStream = getStream();
            imgStream.Position = 0;
            return ImageHelper.LoadStream(imgStream);
        }

        //-----------------------------

        private Stream getStream() {return useStreamB ? imgStreamB : imgStreamA;}

        private int findNewline(int start, int stop, out byte[] data) {
            for (int i = start; i < stop; i++) {
                if (i < stop-1 && buffer[i] == newline_src[0] && buffer[i+1] == newline_src[1]) {
                    data = newline_src;
                    return i;
                } else if (buffer[i] == newline_src[1]) {
                    data = new byte[] {newline_src[1]};
                    return i;
                }
            }
            data = null;
            return -1;
        }

        private int findData(byte[] data, int start, int stop) {
            int data_size = data.Length;
            for (int i = start; i < stop-data_size; i++) {
                if (findInnerData(data, i)) return i;
            }
            return -1;
        }

        private bool findInnerData(byte[] data, int buffer_index) {
            int count = data.Length;
            for (int i = 0; i < count; i++) {
                if (data[i] != buffer[buffer_index+i]) return false;
            }
            return true;
        }

        private int shiftBytes(int start, int end) {
            int c = end - start;
            for (int i = 0; i < c; i++) {
                buffer[i] = buffer[end-c+i];
            }
            return c;
        }
    }
}
于 2015-07-18T20:43:49.257 回答