I have a group of about 500 websites, and I would like to search them for some key phrase (from 5 to 10 phrases, I think) with the help of Python.
I have two solutions in mind:
Work with Beautifulsoup, and create a script that for every URL does two things: looking for each phrase within the text, look for the internal URLs, repeat the process for each of them and so on. Considering the fact that I am dealing with big websites, I am not sure that this solution can work.
Work with the Google API instead. I am resisting to this idea because I never worked with that API, and I have never seen this solution in other posts related with pictures here.
Which solution is better?
(I have just posted a similar question regarding the same amount of websites, but focused on picture scraping).