The output format should refer to the container, not the codecs used. What you are doing is equivalent to encoding (on the commandline) to a filename with ".h264" extension. This will produce a raw H.264 file (no container). Many players should be able to handle this, but may not be able to probe it, they would rely on the extension to figure out what it is. If you have a raw file which is saved with .3gp extension, that will fail. Try setting the format to ".3gp" or better yet, leave format NULL as well and let ffmpeg guess from the filename :)