0

I have a function that take list of urls and save in to file based on a first letter of url. First, I removed http:// and filter the first letter and join that first letter with file extension and fine that file to search whether the url exists in the file if it exists skip it if not write in the file and append into diff_url_list Here is source code

def checkDiffUrls(url_list):
import mmap
diff_url_list=[]
file_extension="txt"
for urls in url_list:
     temp_url=urls.replace("http://","")
     url_head=temp_url[0][0]
     path=".".join((url_head,file_extension))
     file=open("urls/"+path,'w')
     file_read = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ)
     if urls not in file_read:
         file.write(urls)
         diff_url_list.append(urls)
     file.close()

return diff_url_list

it gives me an error mmap length is greater than file size Can anybody fix that bug? Thank you. I appreciate that.

4

0 回答 0