0

尝试编写一个从网站中提取 URL 的程序。输出很好，但是当我尝试将输出写入文件时，只写入最后一条记录。这是代码：

import re
import urllib.request

# Retrieves URLs from the HTML source code of a website
def extractUrls(url, unique=True, sort=True, restrictToTld=None):
    # Prepend "www." if not present
    if url[0:4] != "www.":
        url = "".join(["www.",url])
    # Open a connection
    with urllib.request.urlopen("http://" + url) as h:
        # Grab the headers
        headers = h.info()
        # Default charset
        charset = "ISO-8859-1"
        # If a charset is in the headers then override the default
        for i in headers:
            match = re.search(r"charset=([\w\-]+)", headers[i], re.I)
            if match != None:
                charset = match.group(1).lower()
                break
        # Grab and decode the source code
        source = h.read().decode(charset)
        # Find all URLs in the source code
        matches = re.findall(r"http\:\/\/(www.)?([a-z0-9\-\.]+\.[a-z]{2,6})\b", source, re.I)
        # Abort if no URLs were found
        if matches == None:
            return None
        # Collect URLs
        collection = []
        # Go over URLs one by one
        for url in matches:
            url = url[1].lower()
            # If there are more than one dot then the URL contains
            # subdomain(s), which we remove
            if url.count(".") > 1:
                temp = url.split(".")
                tld = temp.pop()
                url = "".join([temp.pop(),".",tld])
            # Restrict to TLD if one is set
            if restrictToTld:
                tld = url.split(".").pop()
                if tld != restrictToTld:
                    continue
            # If only unique URLs should be returned
            if unique:
                if url not in collection:
                    collection.append(url)
            # Otherwise just add the URL to the collection
            else:
                collection.append(url)
        # Done
        return sorted(collection) if sort else collection

# Test
url = "msn.com"
print("Parent:", url)
for x in extractUrls(url):
    print("-", x)

f = open("f2.txt", "w+", 1)
f.write( x ) 
f.close()

输出是：

Parent: msn.com
- 2o7.net
- atdmt.com
- bing.com
- careerbuilder.com
- delish.com
- discoverbing.com
- discovermsn.com
- facebook.com
- foxsports.com
- foxsportsarizona.com
- foxsportssouthwest.com
- icra.org
- live.com
- microsoft.com
- msads.net
- msn.com
- msnrewards.com
- myhomemsn.com
- nbcnews.com
- northjersey.com
- outlook.com
- revsci.net
- rsac.org
- s-msn.com
- scorecardresearch.com
- skype.com
- twitter.com
- w3.org
- yardbarker.com
[Finished in 0.8s]

只有“yardbarker.com”被写入文件。感谢您的帮助，谢谢。

4

4 回答 4

2 
url = "msn.com"
print("Parent:", url)
f = open("f2.txt", "w",)
for x in extractUrls(url):
    print("-", x)
    f.write( x )
f.close()
于 2013-10-17T07:35:48.983 回答
1

根据其他答案，文件写入需要在循环内，但也尝试\n在之后写一个换行符x：

f = open("f2.txt", "w+")
for x in extractUrls(url):
    print("-", x)
    f.write( x +'\n' ) 
f.close()

此外，该行在return sorted(collection) if sort else collection应该有一个缩进的地方有两个缩进。

此外，您的子域代码可能无法提供您所期望的内容，例如www.something.com.au只会返回.com.au

于 2013-10-17T08:43:11.780 回答
0

您需要打开文件，然后在 for 循环中写入每个 X。

最后，您可以关闭文件。

f = open("f2.txt", "w+",1)

for x in extractUrls(url):
    print("-", x)
    f.write( x ) 

f.close()
于 2013-10-17T06:40:26.737 回答
0 
f = open("f2.txt", "w+", 1)

for x in extractUrls(url):
    print("-", x)
    f.write( x )

f.close()
于 2013-10-17T06:40:41.867 回答