Here is my experience: I'm building a robotic smart toy for autistic children using Arduino, sensors, motors, led and bluetooth. I wrote my own libraries to do exactly what I needed using C++. But I found out that the Arduino IDE Compiler is an older version that does not support the new C++11 features.
So I had to find myself a way to compile C++11 code and upload it to my Arduino. It turns out to be "pretty" basic: I needed a Makefile, the avr-gcc 4.8 toolchain and voilà!
The makefile job is done by Sudar (https://github.com/sudar/Arduino-Makefile) and it works great. I had to customise it a bit to make it work for my project though.
Here is some documentation I've written for my project. You should take a look, it might be useful for you. https://github.com/WeAreLeka/moti/blob/master/INSTALL.md
Hope it helps!
Cheers :)
EDIT 08/16/2014:
Because I got a lot a requests similar to this one from friends and other devs, I decided to set up some kind of framework to get up and running with your Arduino projects quickly and easily.
This is the Bare Arduino Project
Hope it could be of any help! If you find bugs or stuff that I could make better, feel free to fill and issue. :)