We have production and test environment as any other company. And I was thinking to put a robots.txt into the UAT root folder that Google web crawler would not do an unwarranted crawl on the uat pages.
But what I found out was surprising. I do not have robots.txt anywhere on my Uat box and still the none of the Uat sites show up in google search. Uat is not behind firewall so any outside computer can see the machine. Some how none of the Google searches are pulling any of the web site on my Uat box out.
Any ideas?