I just post a example on QCAR forum, you can check the detail here
The idea is simple.
1.I don't use avassetreader to read the video pixel, instead I use AVPlayer and AVPlayerLayer which can play remote video file.
2.I need to convert opengl modelview matrix to CATransform3D so the AVPlayerLayer will attach on the trackable image.Thanks for Hammer on stackoverflow, he shared an example about how to make this.
3.I tried to render camera background use opengl and AVPlayerLayer at same time, but the performace is not good enough, so I use another calayer to render the camera background. There is a bug in 1.5.8 when getting camera frame, thank for andersfrank the problem was solved here.
I put my code on pastebin:
http://pastebin.com/kmeVZ7jy
There are still some problem need to solve, but I think this is a right approach.
PS. For question 3 in my original post, you can fix it by set
cameraLayer.contentsGravity = kCAGravityResizeAspectFill;
This will fix the problem that the aspect ratio of the camera image is different with the iphone screen's.
Update
All the problem are solved, please check my original post on vuforia and pastebin.