Is there any way to find out which urls of my websites are indexed and which not?
(e.g. site:http://example.com/site1.html)
What i tried:
- Used google ajax api -> Problem here is that the results are totally different from the ones i'm getting from the google search
- Used google custom search api -> Same problem here: The results differ from the ones google is showing (because its actually kind of like a private search)
- Used Jsoup to crawl google -> Since its against there Terms its really hard to do -> I set a timeout between every request (between 30s-90s) and used proxies. Still i can't crawl for a long time and google blocks the ip.
What to do? :)