长时间的stackoverflow阅读器，第一次海报。
我正在尝试创建一个名为CloudWriter的 iPad 应用程序。该应用程序的概念是绘制您在云中看到的形状。下载应用程序后，在启动CloudWriter时，用户将看到一个实时视频背景（来自后置摄像头），其顶部有一个 OpenGL 绘图层。用户将能够打开应用程序，将 iPad 指向天空中的云，然后在显示屏上绘制他们所看到的内容。
该应用程序的一个主要功能是让用户记录会话期间显示器上发生的情况的视频屏幕截图。实时视频源和“绘图”视图将成为平面（合并）视频。
关于目前如何运作的一些假设和背景信息。
- 使用 Apple 的 AVCamCaptureManager（AVCam示例项目的一部分）作为大部分与相机相关的代码的基础。
- 使用AVCaptureSessionPresetMedium作为预设来初始化 AVCamCapture 会话。
- 开始通过 videoPreviewLayer 将摄像头馈送作为背景输出。
- 使用允许使用 openGL 进行“绘图”（指画风格）的视图覆盖该实时 videoPreviewLayer。“绘图”视图背景是 [UIColor clearColor]。
在这一点上，想法是用户可以将 iPad3 相机对准天空中的一些云并绘制他们看到的形状。此功能完美无缺。当我尝试对用户会话进行“平面”视频屏幕捕获时，我开始遇到性能问题。生成的“平面”视频将使相机输入与用户绘图实时叠加。
与我们正在寻找的功能相似的应用程序的一个很好的例子是Board Cam，可在 App Store 中找到。
要启动该过程，视图中始终可见一个“记录”按钮。当用户点击录制按钮时，期望在再次点击录制按钮之前，会话将被录制为“平面”视频屏幕截图。
当用户点击“记录”按钮时，代码中会发生以下情况
AVCaptureSessionPreset从AVCaptureSessionPresetMedium更改为AVCaptureSessionPresetPhoto，允许访问
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection- isRecording值设置为YES。
didOutputSampleBuffer 开始获取数据并从当前视频缓冲区数据创建图像。它通过调用
- (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer- self.currentImage 设置为此
应用程序根视图控制器开始覆盖 drawRect 以创建平面图像，用作最终视频中的单个帧。
- 该帧被写入平面视频
为了创建一个平面图像，用作单独的帧，在根 ViewController 的 drawRect 函数中，我们抓取 AVCamCaptureManager 的didOutputSampleBuffer代码接收到的最后一帧。也就是下面
- (void) drawRect:(CGRect)rect {
NSDate* start = [NSDate date];
CGContextRef context = [self createBitmapContextOfSize:self.frame.size];
//not sure why this is necessary...image renders upside-down and mirrored
CGAffineTransform flipVertical = CGAffineTransformMake(1, 0, 0, -1, 0, self.frame.size.height);
CGContextConcatCTM(context, flipVertical);
if( isRecording)
[[self.layer presentationLayer] renderInContext:context];
CGImageRef cgImage = CGBitmapContextCreateImage(context);
UIImage* background = [UIImage imageWithCGImage: cgImage];
CGImageRelease(cgImage);
UIImage *bottomImage = background;
if(((AVCamCaptureManager *)self.captureManager).currentImage != nil && isVideoBGActive )
{
UIImage *image = [((AVCamCaptureManager *)self.mainContentScreen.captureManager).currentImage retain];//[UIImage
CGSize newSize = background.size;
UIGraphicsBeginImageContext( newSize );
// Use existing opacity as is
if( isRecording )
{
if( [self.mainContentScreen isVideoBGActive] && _recording)
{
[image drawInRect:CGRectMake(0,0,newSize.width,newSize.height)];
}
// Apply supplied opacity
[bottomImage drawInRect:CGRectMake(0,0,newSize.width,newSize.height) blendMode:kCGBlendModeNormal alpha:1.0];
}
UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
self.currentScreen = newImage;
[image release];
}
if (isRecording) {
float millisElapsed = [[NSDate date] timeIntervalSinceDate:startedAt] * 1000.0;
[self writeVideoFrameAtTime:CMTimeMake((int)millisElapsed, 1000)];
}
float processingSeconds = [[NSDate date] timeIntervalSinceDate:start];
float delayRemaining = (1.0 / self.frameRate) - processingSeconds;
CGContextRelease(context);
//redraw at the specified framerate
[self performSelector:@selector(setNeedsDisplay) withObject:nil afterDelay:delayRemaining > 0.0 ? delayRemaining : 0.01];
}
createBitmapContextOfSize如下
- (CGContextRef) createBitmapContextOfSize:(CGSize) size {
CGContextRef context = NULL;
CGColorSpaceRef colorSpace = nil;
int bitmapByteCount;
int bitmapBytesPerRow;
bitmapBytesPerRow = (size.width * 4);
bitmapByteCount = (bitmapBytesPerRow * size.height);
colorSpace = CGColorSpaceCreateDeviceRGB();
if (bitmapData != NULL) {
free(bitmapData);
}
bitmapData = malloc( bitmapByteCount );
if (bitmapData == NULL) {
fprintf (stderr, "Memory not allocated!");
CGColorSpaceRelease( colorSpace );
return NULL;
}
context = CGBitmapContextCreate (bitmapData,
size.width ,
size.height,
8, // bits per component
bitmapBytesPerRow,
colorSpace,
kCGImageAlphaPremultipliedFirst);
CGContextSetAllowsAntialiasing(context,NO);
if (context== NULL) {
free (bitmapData);
fprintf (stderr, "Context not created!");
CGColorSpaceRelease( colorSpace );
return NULL;
}
//CGAffineTransform transform = CGAffineTransformIdentity;
//transform = CGAffineTransformScale(transform, size.width * .25, size.height * .25);
//CGAffineTransformScale(transform, 1024, 768);
CGColorSpaceRelease( colorSpace );
return context;
}
- (void)captureOutput:didOutputSampleBuffer fromConnection
// Delegate routine that is called when a sample buffer was written
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
// Create a UIImage from the sample buffer data
[self imageFromSampleBuffer:sampleBuffer];
}
- (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer下面
// Create a UIImage from sample buffer data - modifed not to return a UIImage *, rather store it in self.currentImage
- (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer
{
// unlock the memory, do other stuff, but don't forget:
// Get a CMSampleBuffer's Core Video image buffer for the media data
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
// Lock the base address of the pixel buffer
CVPixelBufferLockBaseAddress(imageBuffer, 0);
// uint8_t *tmp = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
int bytes = CVPixelBufferGetBytesPerRow(imageBuffer); // determine number of bytes from height * bytes per row
//void *baseAddress = malloc(bytes);
size_t height = CVPixelBufferGetHeight(imageBuffer);
uint8_t *baseAddress = malloc( bytes * height );
memcpy( baseAddress, CVPixelBufferGetBaseAddress(imageBuffer), bytes * height );
size_t width = CVPixelBufferGetWidth(imageBuffer);
// Create a device-dependent RGB color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Create a bitmap graphics context with the sample buffer data
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8,
bytes, colorSpace, kCGBitmapByteOrderDefault | kCGImageAlphaPremultipliedFirst);
// CGContextScaleCTM(context, 0.25, 0.25); //scale down to size
// Create a Quartz image from the pixel data in the bitmap graphics context
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
// Unlock the pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
// Free up the context and color space
CGContextRelease(context);
CGColorSpaceRelease(colorSpace);
free(baseAddress);
self.currentImage = [UIImage imageWithCGImage:quartzImage scale:0.25 orientation:UIImageOrientationUp];
// Release the Quartz image
CGImageRelease(quartzImage);
return nil;
}
最后，我使用writeVideoFrameAtTime:CMTimeMake将其写入磁盘，代码如下：
-(void) writeVideoFrameAtTime:(CMTime)time {
if (![videoWriterInput isReadyForMoreMediaData]) {
NSLog(@"Not ready for video data");
}
else {
@synchronized (self) {
UIImage* newFrame = [self.currentScreen retain];
CVPixelBufferRef pixelBuffer = NULL;
CGImageRef cgImage = CGImageCreateCopy([newFrame CGImage]);
CFDataRef image = CGDataProviderCopyData(CGImageGetDataProvider(cgImage));
if( image == nil )
{
[newFrame release];
CVPixelBufferRelease( pixelBuffer );
CGImageRelease(cgImage);
return;
}
int status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, avAdaptor.pixelBufferPool, &pixelBuffer);
if(status != 0){
//could not get a buffer from the pool
NSLog(@"Error creating pixel buffer: status=%d", status);
}
// set image data into pixel buffer
CVPixelBufferLockBaseAddress( pixelBuffer, 0 );
uint8_t* destPixels = CVPixelBufferGetBaseAddress(pixelBuffer);
CFDataGetBytes(image, CFRangeMake(0, CFDataGetLength(image)), destPixels); //XXX: will work if the pixel buffer is contiguous and has the same bytesPerRow as the input data
if(status == 0){
BOOL success = [avAdaptor appendPixelBuffer:pixelBuffer withPresentationTime:time];
if (!success)
NSLog(@"Warning: Unable to write buffer to video");
}
//clean up
[newFrame release];
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );
CVPixelBufferRelease( pixelBuffer );
CFRelease(image);
CGImageRelease(cgImage);
}
}
}
一旦 isRecording 设置为 YES，iPad 3 的性能就会从大约 20FPS 变为可能 5FPS。使用Insturments，我可以看到以下代码块（来自drawRect:）是导致性能下降到无法使用的水平的原因。
if( _recording )
{
if( [self.mainContentScreen isVideoBGActive] && _recording)
{
[image drawInRect:CGRectMake(0,0,newSize.width,newSize.height)];
}
// Apply supplied opacity
[bottomImage drawInRect:CGRectMake(0,0,newSize.width,newSize.height) blendMode:kCGBlendModeNormal alpha:1.0];
}
我的理解是，因为我正在捕获全屏，我们失去了“drawInRect”应该提供的所有好处。具体来说，我说的是更快的重绘，因为理论上，我们只更新显示的一小部分（在 CGRect 中传递）。同样，捕获全屏，我不确定drawInRect是否能提供几乎一样多的好处。
为了提高性能，我想如果我缩小imageFromSampleBuffer提供的图像和绘图视图的当前上下文，我会看到帧速率增加。不幸的是，CoreGrapics.Framework 不是我过去使用过的东西，所以我不知道我是否能够有效地将性能调整到可接受的水平。
任何 CoreGraphics 大师有意见吗？
此外，某些代码的 ARC 已关闭，分析器显示有一次泄漏，但我认为这是误报。
即将推出，CloudWriter ™，天空无限！