I apologize if this isn't the right place to ask this and appreciate any help in moving it to a more appropriate forum if necessary. My original question was going to be along the lines of what I need to do in the PHP to change this behavior... but I'm not sure if it's really the problem or not so this is a pre-question of sorts to find out whether I even need to bother asking the programming question that I was going to ask.
We have a site that's using too much bandwidth. I was told it's being caused by web crawlers so I checked at sure enough that seems to be the case. One thing I noticed was that 403 errors were responsible for most of the traffic. I didn't see how that was possible since I would expect a 403 error to just send a little bit of informational text, but when I tried purposefully going to a url that didn't exist it redirected me to the homepage.
So I'm assuming every time a non-existent link is hit by a web crawler that it's transferring everything on the homepage... and I wonder if the web crawler thinks this is a new starting point from which it needs to branch out to all the links on the homepage all over again since it was hammering the website for over 24 hours straight before it got taken down?
EDIT: Seems that I made a mistake as halfer pointed out. I saw '403' and immediately thought of the wrong thing. It's is 403 (forbidden access) that is the issue, so maybe that means someone is trying to hack into the website?