0

Python 和 BeautifulSoup 的新手。非常感谢任何帮助

我知道如何建立一个公司信息列表，但那是在单击一个链接之后。

import requests 
from bs4 import BeautifulSoup


url = "http://data-interview.enigmalabs.org/companies/"
r = requests.get(url)

soup = BeautifulSoup(r.content)

links = soup.find_all("a")

link_list = []

 for link in links:
    print link.get("href"), link.text

 g_data = soup.find_all("div",{"class": "table-responsive"})

 for link in links:
    print link_list.append(link)

谁能给出一个想法，如何首先抓取链接，然后为该网站构建所有公司列表数据的 JSON？

我还附上了示例图像以获得更好的可视化效果。

我将如何抓取网站并像下面的示例一样构建 JSON，而无需单击每个单独的链接？

示例预期输出：

all_listing = [ {"Dickens-Tillman":{'Company Detail': 
 {'Company Name': 'Dickens-Tillman',
  'Address Line 1   ': '7147 Guilford Turnpike Suit816',
  'Address Line 2   ': 'Suite 708',
  'City': 'Connfurt',
  'State': 'Iowa',
  'Zipcode  ': '22598',
  'Phone': '00866539483',
  'Company Website  ': 'lockman.com',
  'Company Description': 'enable robust paradigms'}}},
`{'"Klein-Powlowski" ':{'Company Detail': 
 {'Company Name': 'Klein-Powlowski',
  'Address Line 1   ': '32746 Gaylord Harbors',
  'Address Line 2   ': 'Suite 866',
  'City': 'Lake Mario',
  'State': 'Kentucky',
  'Zipcode  ': '45517',
  'Phone': '1-299-479-5649',
  'Company Website  ': 'marquardt.biz',
 'Company Description': 'monetize scalable paradigms'}}}]

print all_listing`

在此处输入图像描述

在此处输入图像描述

在此处输入图像描述

4

1 回答 1

1

这是我对我提出的问题的最终解决方案。

import bs4, urlparse, json, requests,csv
from os.path import basename as bn

links = []
data = {}
base = 'http://data-interview.enigmalabs.org/'

#Approach 
#1. Each individual pages, collect the links
#2. Iterate over each link in a list
#3. Before moving on each the list for links if correct move on, if not review step 2 then 1
#4. Push correct data to a JSON file



def bs(r):
    return bs4.BeautifulSoup(requests.get(urlparse.urljoin(base, r).encode()).content, 'html.parser').find('table')

for i in range(1,11):
    print 'Collecting page %d' % i
    links += [a['href'] for a in bs('companies?page=%d' % i).findAll('a')]
# Search a the given range of "a" on each page

# Now that I have collected all links into an list,iterate over each link
# All the info is within a html table, so search and collect all company info in data
for link in links:
    print 'Processing %s' % link
    name = bn(link)
    data[name] = {}
    for row in bs(link).findAll('tr'):
        desc, cont = row.findAll('td')
        data[name][desc.text.encode()] = cont.text.encode()

print json.dumps(data)

# Final step is to have all data formating 
json_data = json.dumps(data, indent=4)
file = open("solution.json","w")
file.write(json_data)
file.close()
于 2017-07-13T22:19:18.513 回答