0

Hello good day stackoverflow pips,

Issue: stack and data was never scraped in a russian retailer which is in this case www.vseinstrumenti.ru

code:

import asyncio
from pyppeteer import launch

class PyppeteerRequests:
    def __init__(self):
        self.headers = {}

    def get_url(self, url):
        data = None
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)
        try:
            data = loop.run_until_complete(self.main(url))
            print(data)
        except Exception as e:
            print(str(e))

        return data
        
    async def main(self, url):
        browser = await launch(headless=True)
        page = await browser.newPage()
        await page.goto(url, options={'timeout':1000000, 'waitUntil':['load', 'networkidle2']}),
        loaded_html = await page.content()
        await page.waitForNavigation()
            
        print("closing context...")

        await asyncio.sleep(0.3)
        await page.close()
        await browser.close()
        print("closing browser...")
        await asyncio.sleep(0.3)
        return loaded_html

        

if __name__=="__main__":
    requester = PyppeteerRequests()

    url = 'https://www.vseinstrumenti.ru/ruchnoy-instrument/sadoviy-instrument-i-inventar/topory/fiskars/x11-s-1015640/'
    data = requester.get_url(url)
    print(data)

It just stacked and get ERROR: Navigation Timeout Exceeded: 1000000 ms exceeded. What part of the code should I change? Is it scrape-able on your side? Kindly let me know how to improve my code using asnycio. Thanks!

4

0 回答 0