-1

I need to check several hundred proxy servers and get the number of not working. Script for this

import urllib.request
import socket

net = ['http://192.168.1.1:8080',
'http://192.168.1.2:8080',
'http://192.168.1.3:8080',
'http://192.168.1.4:8080',
'http://192.168.1.5:8080',
'http://192.168.1.6:8080',
'http://192.168.1.7:8080',
'http://192.168.1.8:8080',
'http://192.168.1.9:8080',
'http://192.168.1.10:8080']

fail = 0
socket.setdefaulttimeout(3)

for x in net:
    try:
        print(x)
        proxy = urllib.request.ProxyHandler({'http': (x)})
        opener = urllib.request.build_opener(proxy)
        urllib.request.install_opener(opener)
        urllib.request.urlretrieve('http://google.com')
    except IOError:
        print ("Connection error")
        fail+=1

print(fail)

Proxies in the list, I have given a simple version.

It takes 55 seconds to check 250 working proxies. I can't wait that long, need to increase the execution speed. How can this be done using async?

4

1 回答 1

0

这应该让您了解如何处理它。你必须将各种连接块包装在 try 中，除了你自己。

注意：此代码未经测试，因为我没有任何方法可以这样做。

import asyncio, aiohttp

def returnPartionedList(inputlist, x=100):
    return([inputlist[i:i + x] for i in range(0, len(inputlist), x)])
    # Returns: Original list split into segments of x. 

async def TestProxy(url, proxy, session):
    async with session.get(url, proxy=proxy, timeout=3) as response:
        if response.status == 200:
            _ = await response.text()
            return(proxy)

async def TestProxies(listofproxies):
    returnResults = []
    url = "https://google.com" # Test proxy with this url
    ProxyPartitions = returnPartionedList(listofproxies, 20) # Rate limit 20 per second
    for partition in ProxyPartitions:
        ProxyTasks = []
        async with aiohttp.ClientSession() as session:
            for proxy in partition:
                ProxyTasks.append(asyncio.create_task(TestProxy(url, proxy, session)))
            results = await asyncio.gather(*ProxyTasks, return_exceptions=False)
            if results:
                for result in results:
                    if result:
                        returnResults.append(result)
        await asyncio.sleep(1)
    return(returnResults)

async def main():
    listofproxies = [
        'http://10.10.1.1:8080',
        'http://10.10.1.2:8080',
        'http://10.10.1.3:8080',
        'http://10.10.1.4:8080',
        'http://10.10.1.5:8080',
        'http://10.10.1.6:8080',
        'http://10.10.1.7:8080',
        'http://10.10.1.8:8080',
        'http://10.10.1.9:8080',
        'http://10.10.1.10:8080'
    ]
    test_proxies = await TestProxies(listofproxies)
    print(test_proxies)

if __name__ == "__main__":
    asyncio.run(main())
于 2021-12-27T14:29:01.477 回答