I have discovered a work-around.
1) Go to the location in which Visual Studio caches the app_offline.htm file
C:\Users[user]\AppData\Roaming\Microsoft\VisualStudio\11.0\ (replace [user] with your username)
2) Delete the app_offline.htm file
(NOTE: this alone will not solve it, as VS regenerates the file again when it's not found)
3) Create a directory called app_offline.htm (a DIRECTORY!)
This essentially causes an error with the process that attempts to use the file because it's not a file and the name clash means VS cannot try to recreate the file. Luckily the error has no unwanted side-effects.
Now continue with using VS and Team Foundation Server and enjoy not constantly having the appdomain reloaded!
This does of course means that any Publish process will probably not use the app_offline.htm file, so you'll need to keep an eye on that.