I might be wrong but I think your problem is not an AngularJS one, it's more fundamental than that.
Your issue is one where you are loading HTML content via AJAX. And how does Google etc. crawl AJAX loaded content if it can't execute JavaScript?
This might help:
https://developers.google.com/webmasters/ajax-crawling/
This might help too but its geared towards .NET: http://gregorypratt.github.com/Ajax-Crawling/
If you provide a solution where you let AngularJS do the routing but you still are able to serve static content from the server when ?_escaped_fragment_=
is present in the URL then you're good to go. You get single page app benefits whilst still being crawlable.
The following is an example site using AngularJS routing and static content being served for Google et al.
http://artelier.co.uk/#!/about
http://artelier.co.uk/?_escaped_fragment_=/about (turn off JS to see it work)