1

我正在使用 Selenium 制作一个简单的程序，该程序会转到 Flickr.com，搜索用户输入的术语，然后打印出所有这些图像的 URL。

我在最后一部分苦苦挣扎，只获取图像的 URL。我一直在使用class_=搜索来获取 URL 所在的 HTML 部分。搜索“苹果”时，这会多次返回以下内容：

<div class="view photo-list-photo-view requiredToShowOnServer awake" 
   data-view-signature="photo-list-photo-view__engagementModelName_photo-lite-
   models__excludePeople_false__id_6246270647__interactionViewName_photo-list-
   photo-interaction-    view__isOwner_false__layoutItem_1__measureAFT_true__model_1__modelParams_1_    _parentContainer_1__parentSignature_photolist-
   479__requiredToShowOnClient_true__requiredToShowOnServer_true__rowHeightMod    _1__searchTerm_apples__searchType_1__showAdvanced_true__showSort_true__show    Tools_true__sortMenuItems_1__unifiedSubviewParams_1__viewType_jst"
   style="transform: translate(823px, 970px); -webkit-transform:     translate(823px, 970px); -ms-transform: translate(823px, 970px); width:
   237px; height: 178px; background-image:
   url(//c3.staticflickr.com/7/6114/6246270647_edc7387cfc_m.jpg)">
<div class="interaction-view"></div>

我想要的只是每个图像的 URL 如下所示：

c3.staticflickr.com/7/6114/6246270647_edc7387cfc_m.jpg

由于没有a或hreftrags 我正在努力将它们过滤掉。

我最后也尝试了一些正则表达式，例如：

print(soup.find_all(re.compile(r'^url\.jpg$')))

但这没有用。

无论如何，这是我的完整代码，谢谢。

import os
import re
import urllib.request as urllib2
import bs4
from selenium import webdriver
from selenium.webdriver.common.keys import Keys 

os.makedirs('My_images', exist_ok=True)

browser = webdriver.Chrome()
browser.implicitly_wait(10)

print("Opening Flickr.com")

siteChoice = 'http://www.flickr.com'

browser.get(siteChoice)

print("Enter your search term: ")

term = input("> ")

searchField = browser.find_element_by_id('search-field')
searchField.send_keys(term)
searchField.submit()

url = siteChoice + '/search/?text=' + term

html = urllib2.urlopen(url)

soup = bs4.BeautifulSoup(html, "html.parser")

print(soup.find_all(class_='view photo-list-photo-view requiredToShowOnServer awake', style = re.compile('staticflickr')))

我更改的代码：

p = re.compile(r'url\(\/\/([^\)]+)\)')

test_str = str(soup)

all_urls = re.findall(p, test_str)


print('Exporting to file')


with open('flickr_urls.txt', 'w') as f:
    for i in all_urls:
        f.writelines("%s\n" % i)

print('Done')
4

2 回答 2

2

试试这个

url\(\/\/([^\)]+)\)

演示

import re
p = re.compile(ur'url\(\/\/([^\)]+)\)')
test_str = u"<div class=\"view photo-list-photo-view requiredToShowOnServer awake\" \ndata-view-signature=\"photo-list-photo-view__engagementModelName_photo-lite-\nmodels__excludePeople_false__id_6246270647__interactionViewName_photo-list-\nphoto-interaction-    view__isOwner_false__layoutItem_1__measureAFT_true__model_1__modelParams_1_    _parentContainer_1__parentSignature_photolist-\n479__requiredToShowOnClient_true__requiredToShowOnServer_true__rowHeightMod    _1__searchTerm_apples__searchType_1__showAdvanced_true__showSort_true__show    Tools_true__sortMenuItems_1__unifiedSubviewParams_1__viewType_jst\"\n style=\"transform: translate(823px, 970px); -webkit-transform:     translate(823px, 970px); -ms-transform: translate(823px, 970px); width:\n 237px; height: 178px; background-image:\n url(//c3.staticflickr.com/7/6114/6246270647_edc7387cfc_m.jpg)\">\n<div class=\"interaction-view\"></div>"

m = re.search(p, test_str)
print m.group(1)

输出：

c3.staticflickr.com/7/6114/6246270647_edc7387cfc_m.jpg
于 2016-03-10T04:09:22.493 回答
2

要使用 Selenium 从页面中删除所有 png/jpg 链接：

from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://www.flickr.com/")
links = driver.execute_script("return document.body.innerHTML.match(" \
  "/https?:\/\/[a-z_\/0-9\-\#=&.\@]+\.(jpg|png)/gi)")
print links
于 2016-03-10T06:22:27.040 回答