0

所以我写了一个脚本来下载文件，保存文件，然后读取文件并将值存储在我可以进一步使用的变量中。我的问题是原始文件的格式不是很好。我最初认为它是制表符分隔的，但似乎也有一些多余的空格。

这是我的代码的一个子集：

for link in soup.find_all('a', href=re.compile('emailLogins_20130930')):
    parsedURL = str(link.get('href')).strip()
    fileURL = 'https://xxxxx.com'+parsedURL
    request = opener.open(fileURL)
    out = open(fileName, 'a')

    for row in request:
        if re.search('Source', row): #strip out the header on the original file before rewriting the file to disk
            continue
        else:
            if row.strip():
                for column in row:
                    out.write(column)
            else:
                continue
    out.close()
    time.sleep(4)


#This part removes some null lines (if they exist) I found previously in the files  
with open(fileName, 'rb') as f_origin:
    data = f_origin.read()

with open('cleanCSV.csv', 'wb') as f_clean: 
    f_clean.write(data.replace('\x00', ''))

#Attempting to remove the tabs and replace with commas.  My thought was that the spaces would just be included in the strings
#but it looks as though those are being converted to ',' as well.

in_txt = csv.reader(open('cleanCSV.csv', 'rb'), delimiter = '\t')
out_csv = csv.writer(open('new-csv-test.csv', 'wb'))
out_csv.writerows(in_txt)

filereader = open('new-csv-test.csv', 'rb')

reader = csv.reader(filereader, delimiter=',', quoting=csv.QUOTE_NONE)

for row in reader:
    rowlist = list(row)
    source = rowlist[0]
    print '0: ' + source
    #start_date = rowlist[1]
    #print '1: ' + start_date
    #start_time = rowlist[2]
    #print '2: ' + start_time
    #start = start_date + ' ' + start_time
    #print 'START: ' + start
    start = rowlist[1]
    print '1: ' + rowlist[1]
    start_dt = datetime.strptime(start, '%Y-%m-%d %H:%M:%S')
    start_ts = start_ts = start_dt.strftime('%b %d %Y %H:%M:%S')
    upstreamIP = rowlist[2]
    print '2: ' + upstreamIP
    username = rowlist[3]
    print '3: ' + username
    emailLogins = rowlist[4]
    print '4: ' + emailLogins
    emailProvider = rowlist[5]
    print '5: ' + emailProvider + '\n'

    mergedEmail = emailLogins+'@'+emailProvider

这是原始文件的示例：

11.111.111.111_vpn_ 2013-09-29 19:50:35     NULL    Pxxx    aol.com
11.111.111.111_vpn_ 2013-09-29 19:49:50     NULL    Dxxxxxxx    aol.com
11.111.111.111_vpn_ 2013-09-29 19:54:24     NULL    fxxxxxxx_governmentgrant    aol.com
11.111.111.111_vpn__parsed  2013-09-30 10:58:48 98506   mxxxxx05    hxxxxxyen   yahoo.com
mace3_vpn_11.11.111.111     2013-09-30 11:14:48     NULL    mxxxxxys00  aol.com
11.111.111.111_vpn__parsed  2013-09-30 11:10:08 98506   mxxxxx05    hhxxxxxen   yahoo.com
mace3_vpn_111.111.111.1     2013-09-30 11:38:57     NULL    Fndxxxxxa   aol.com
mace3_vpn_11.11.111.111     2013-09-30 11:24:49     NULL    myxxxxxx00  aol.com
mace3_vpn_11.11.111.111     2013-09-30 11:25:16     NULL    mxxxxxxxxxx01   yahoo.com

这是我的代码正在执行的操作（请注意第一列之后的双 ','。我希望第二组双 ',' 因为经常有一列没有数据。

111.111.111.1_vpn_,2013-09-29 19:50:35,,NULL,Pxxxx0,aol.com
111.111.111.1_vpn_,2013-09-29 19:49:50,,NULL,Dxxxxxen,aol.com
111.111.111.1_vpn_,2013-09-29 19:54:24,,NULL,fxxxxxxk_governmentgrant,aol.com
111.111.111.1_vpn__parsed,2013-09-30 10:58:48,98506,mxxxxxx5,hxxxxxxen,yahoo.com
mace3_vpn_111.111.111.1,,2013-09-30 11:14:48,,NULL,mxxxxxxs00,aol.com
111.111.111.1_vpn__parsed,2013-09-30 11:10:08,98506,mxxxxxx5,hxxxxxxen,yahoo.com
mace3_vpn_111.111.111.1,,2013-09-30 11:38:57,,NULL,Fxxxxxxxa,aol.com
mace3_vpn_111.111.111.1,,2013-09-30 11:24:49,,NULL,mxxxxxxs00,aol.com
mace3_vpn_111.111.111.1,,2013-09-30 11:25:16,,NULL,mxxxxxxxxx1,yahoo.com

我回去查看 VI 中的原始文件，查看第二行第一列和第二列之间空白的 ASCII 值，它似乎有两个制表符，而不是一个空格和一个制表符。不知道如何在此处删除额外的选项卡，但保留在列没有数据时应该显示的额外选项卡。

4

0 回答 0