1

我正在尝试使用 BeautifulSoup 对 Morningstar Financials 进行网络抓取。出于某种原因,我什至找不到包含财务数据的表。

我试过使用 div 标签和 table 标签。任何一个都没有运气。

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

urls= [
'http://financials.morningstar.com/ratios/r.html?t=AAPL&region=usa&culture=en-US'
      ]

for url in urls:

  try:
      html = uReq(url)
      page_soup = soup(html, "html.parser")

      mainPage = (page_soup.find("table",{"class":"r_table1 text2"}))

      print (mainPage)

  except:
      pass

当我搜索整个页面时,它不会返回任何表格。我要抓取的数据表应该在“财务”div 标签下。

4

1 回答 1

3

数据通过 AJAX 加载(您需要检查您的开发人员控制台以获取正确的 URL)。即便如此,数据是 JSONp 格式的,所以需要更多的预处理:

from bs4 import BeautifulSoup
import requests
import re
import json

url1 = 'http://financials.morningstar.com/finan/financials/getFinancePart.html?&callback=xxx&t=AAPL'
url2 = 'http://financials.morningstar.com/finan/financials/getKeyStatPart.html?&callback=xxx&t=AAPL'

soup1 = BeautifulSoup(json.loads(re.findall(r'xxx\((.*)\)', requests.get(url1).text)[0])['componentData'], 'lxml')
soup2 = BeautifulSoup(json.loads(re.findall(r'xxx\((.*)\)', requests.get(url2).text)[0])['componentData'], 'lxml')

def print_table(soup):
    for i, tr in enumerate(soup.select('tr')):
        row_data = [td.text for td in tr.select('td, th') if td.text]
        if not row_data:
            continue
        if len(row_data) < 12:
            row_data = ['X'] + row_data
        for j, td in enumerate(row_data):
            if j==0:
                print('{: >30}'.format(td), end='|')
            else:
                print('{: ^12}'.format(td), end='|')
        print()

print_table(soup1)
print()
print_table(soup2)

打印表格中的数据:

                             X|  2009-09   |  2010-09   |  2011-09   |  2012-09   |  2013-09   |  2014-09   |  2015-09   |  2016-09   |  2017-09   |  2018-09   |    TTM     |
               Revenue USD Mil|   42,905   |   65,225   |  108,249   |  156,508   |  170,910   |  182,795   |  233,715   |  215,639   |  229,234   |  265,595   |  258,490   |
                Gross Margin %|    40.1    |    39.4    |    40.5    |    43.9    |    37.6    |    38.6    |    40.1    |    39.1    |    38.5    |    38.3    |    38.1    |
      Operating Income USD Mil|   11,740   |   18,385   |   33,790   |   55,241   |   48,999   |   52,503   |   71,230   |   60,024   |   61,344   |   70,898   |   65,491   |
            Operating Margin %|    27.4    |    28.2    |    31.2    |    35.3    |    28.7    |    28.7    |    30.5    |    27.8    |    26.8    |    26.7    |    25.3    |
            Net Income USD Mil|   8,235    |   14,013   |   25,922   |   41,733   |   37,037   |   39,510   |   53,394   |   45,687   |   48,351   |   59,531   |   57,170   |
        Earnings Per Share USD|    1.30    |    2.16    |    3.95    |    6.31    |    5.68    |    6.45    |    9.22    |    8.31    |    9.21    |   11.91    |   11.94    |
                 Dividends USD|     —      |     —      |     —      |    0.38    |    1.63    |    1.81    |    1.98    |    2.18    |    2.40    |    2.72    |    2.92    |
              Payout Ratio % *|     —      |     —      |     —      |     —      |    27.4    |    28.5    |    22.3    |    24.8    |    26.5    |    23.7    |    24.5    |
                    Shares Mil|   6,349    |   6,473    |   6,557    |   6,617    |   6,522    |   6,123    |   5,793    |   5,500    |   5,252    |   5,000    |   4,812    |
    Book Value Per Share * USD|    5.02    |    7.45    |   11.78    |   16.99    |   19.60    |   20.62    |   22.53    |   23.71    |   25.83    |   24.17    |   23.01    |
   Operating Cash Flow USD Mil|   10,159   |   18,595   |   37,529   |   50,856   |   53,666   |   59,713   |   81,266   |   65,824   |   63,598   |   77,434   |   71,856   |
          Cap Spending USD Mil|   -1,213   |   -2,121   |   -7,452   |   -9,402   |   -9,076   |   -9,813   |  -11,488   |  -13,548   |  -12,795   |  -13,313   |  -12,026   |
        Free Cash Flow USD Mil|   8,946    |   16,474   |   30,077   |   41,454   |   44,590   |   49,900   |   69,778   |   52,276   |   50,803   |   64,121   |   59,830   |
Free Cash Flow Per Share * USD|    1.41    |    2.54    |    4.59    |    6.31    |    6.46    |    7.73    |   11.82    |    8.97    |    9.66    |   11.52    |     —      |
       Working Capital USD Mil|   20,049   |   20,956   |   17,018   |   19,111   |   29,628   |   5,083    |   8,768    |   27,863   |   27,831   |   14,473   |     —      |

            Margins % of Sales|  2009-09   |  2010-09   |  2011-09   |  2012-09   |  2013-09   |  2014-09   |  2015-09   |  2016-09   |  2017-09   |  2018-09   |    TTM     |
                       Revenue|   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |
                          COGS|   59.86    |   60.62    |   59.52    |   56.13    |   62.38    |   61.41    |   59.94    |   60.92    |   61.53    |   61.66    |   61.95    |
                  Gross Margin|   40.14    |   39.38    |   40.48    |   43.87    |   37.62    |   38.59    |   40.06    |   39.08    |   38.47    |   38.34    |   38.05    |
                          SG&A|    9.67    |    8.46    |    7.02    |    6.42    |    6.34    |    6.56    |    6.13    |    6.58    |    6.66    |    6.29    |    6.80    |
                           R&D|    3.11    |    2.73    |    2.24    |    2.16    |    2.62    |    3.30    |    3.45    |    4.66    |    5.05    |    5.36    |    5.92    |
                         Other|     —      |     —      |     —      |     —      |     —      |     —      |     —      |     —      |     —      |     —      |     —      |
              Operating Margin|   27.36    |   28.19    |   31.22    |   35.30    |   28.67    |   28.72    |   30.48    |   27.84    |   26.76    |   26.69    |   25.34    |
           Net Int Inc & Other|    0.76    |    0.24    |    0.38    |    0.33    |    0.68    |    0.54    |    0.55    |    0.63    |    1.20    |    0.75    |    0.74    |
                    EBT Margin|   28.12    |   28.42    |   31.60    |   35.63    |   29.35    |   29.26    |   31.03    |   28.46    |   27.96    |   27.45    |   26.08    |
                 Profitability|  2009-09   |  2010-09   |  2011-09   |  2012-09   |  2013-09   |  2014-09   |  2015-09   |  2016-09   |  2017-09   |  2018-09   |    TTM     |
                    Tax Rate %|   31.75    |   24.42    |   24.22    |   25.16    |   26.15    |   26.13    |   26.37    |   25.56    |   24.56    |   18.34    |   15.18    |
                  Net Margin %|   19.19    |   21.48    |   23.95    |   26.67    |   21.67    |   21.61    |   22.85    |   21.19    |   21.09    |   22.41    |   22.12    |
      Asset Turnover (Average)|    0.99    |    1.06    |    1.13    |    1.07    |    0.89    |    0.83    |    0.89    |    0.70    |    0.66    |    0.72    |    0.73    |
            Return on Assets %|   18.92    |   22.84    |   27.07    |   28.54    |   19.34    |   18.01    |   20.45    |   14.93    |   13.87    |   16.07    |   16.12    |
  Financial Leverage (Average)|    1.50    |    1.57    |    1.52    |    1.49    |    1.68    |    2.08    |    2.43    |    2.51    |    2.80    |    3.41    |    3.23    |
            Return on Equity %|   31.27    |   35.28    |   41.67    |   42.84    |   30.64    |   33.61    |   46.25    |   36.90    |   36.87    |   49.36    |   49.13    |
  Return on Invested Capital %|   30.22    |   34.69    |   41.04    |   42.01    |   26.08    |   26.20    |   31.32    |   21.95    |   19.86    |   24.41    |   23.83    |
             Interest Coverage|     —      |     —      |     —      |     —      |   369.79   |   140.28   |   99.93    |   43.15    |   28.59    |   23.50    |   19.65    |
                             X|  2009-09   |  2010-09   |  2011-09   |  2012-09   |  2013-09   |  2014-09   |  2015-09   |  2016-09   |  2017-09   |  2018-09   | Latest Qtr |
                             X| Revenue %  |
                Year over Year|   32.10    |   52.02    |   65.96    |   44.58    |    9.20    |    6.95    |   27.86    |   -7.73    |    6.30    |   15.86    |   -5.11    |
                3-Year Average|   30.48    |   39.54    |   49.37    |   53.94    |   37.86    |   19.08    |   14.30    |    8.06    |    7.84    |    4.35    |     —      |
                5-Year Average|   38.97    |   36.17    |   41.16    |   45.49    |   39.39    |   33.63    |   29.08    |   14.78    |    7.93    |    9.22    |     —      |
               10-Year Average|   21.47    |   23.37    |   35.05    |   39.17    |   39.31    |   36.27    |   32.58    |   27.29    |   25.31    |   23.38    |     —      |
                             X|Operating Income %|
                Year over Year|   87.09    |   56.60    |   83.79    |   63.48    |   -11.30   |    7.15    |   35.67    |   -15.73   |    2.20    |   15.57    |   -15.60   |
                3-Year Average|   68.52    |   60.96    |   75.28    |   67.57    |   38.65    |   15.82    |    8.84    |    7.00    |    5.32    |   -0.16    |     —      |
                5-Year Average|   102.01   |   61.96    |   68.97    |   65.80    |   50.84    |   34.93    |   31.11    |   12.18    |    2.12    |    7.67    |     —      |
               10-Year Average|   40.70    |   42.78    |     —      |   102.35   |   113.41   |   65.10    |   45.72    |   37.68    |   30.12    |   27.44    |     —      |
                             X|Net Income %|
                Year over Year|   70.36    |   70.16    |   84.99    |   60.99    |   -11.25   |    6.68    |   35.14    |   -14.43   |    5.83    |   23.12    |     —      |
                3-Year Average|   60.57    |   58.85    |   75.03    |   71.77    |   38.26    |   15.08    |    8.56    |    7.25    |    6.96    |    3.69    |     —      |
                5-Year Average|   97.22    |   60.03    |   67.11    |   64.20    |   50.27    |   36.84    |   30.67    |   12.00    |    2.99    |    9.96    |     —      |
               10-Year Average|   29.92    |   33.39    |     —      |   90.88    |   87.49    |   64.28    |   44.61    |   36.81    |   30.04    |   28.54    |     —      |
                             X|   EPS %    |
                Year over Year|   69.40    |   66.85    |   82.71    |   59.50    |   -9.97    |   13.58    |   42.95    |   -9.87    |   10.83    |   29.32    |   -9.89    |
                3-Year Average|   58.74    |   56.80    |   72.85    |   69.41    |   37.92    |   17.71    |   13.49    |   13.53    |   12.61    |    8.91    |     —      |
                5-Year Average|   91.24    |   57.56    |   64.90    |   62.22    |   49.29    |   37.82    |   33.62    |   16.01    |    7.87    |   15.97    |     —      |
               10-Year Average|   25.97    |   30.11    |     —      |   85.81    |   82.88    |   62.35    |   45.10    |   38.31    |   32.28    |   31.58    |     —      |
              Cash Flow Ratios|  2009-09   |  2010-09   |  2011-09   |  2012-09   |  2013-09   |  2014-09   |  2015-09   |  2016-09   |  2017-09   |  2018-09   |    TTM     |
Operating Cash Flow Growth % YOY|    5.87    |   83.04    |   101.82   |   35.51    |    5.53    |   11.27    |   36.09    |   -19.00   |   -3.38    |   21.76    |     —      |
   Free Cash Flow Growth % YOY|    6.54    |   84.15    |   82.57    |   37.83    |    7.57    |   11.91    |   39.84    |   -25.08   |   -2.82    |   26.22    |     —      |
        Cap Ex as a % of Sales|    2.83    |    3.25    |    6.88    |    6.01    |    5.31    |    5.37    |    4.92    |    6.28    |    5.58    |    5.01    |    4.65    |
        Free Cash Flow/Sales %|   20.85    |   25.26    |   27.79    |   26.49    |   26.09    |   27.30    |   29.86    |   24.24    |   22.16    |   24.14    |   23.15    |
     Free Cash Flow/Net Income|    1.09    |    1.18    |    1.16    |    0.99    |    1.20    |    1.26    |    1.31    |    1.14    |    1.05    |    1.08    |    1.05    |
    Balance Sheet Items (in %)|  2009-09   |  2010-09   |  2011-09   |  2012-09   |  2013-09   |  2014-09   |  2015-09   |  2016-09   |  2017-09   |  2018-09   | Latest Qtr |
 Cash & Short-Term Investments|   49.40    |   34.08    |   22.30    |   16.54    |   19.59    |   10.82    |   14.32    |   20.88    |   19.76    |   18.13    |   23.42    |
           Accounts Receivable|   10.65    |   13.20    |   10.07    |   10.62    |    9.97    |   11.74    |   10.45    |    9.11    |    9.50    |   13.40    |    7.68    |
                     Inventory|    0.96    |    1.40    |    0.67    |    0.45    |    0.85    |    0.91    |    0.81    |    0.66    |    1.29    |    1.08    |    1.43    |
          Other Current Assets|    5.43    |    6.76    |    5.62    |    5.14    |    4.99    |    6.09    |    5.19    |    2.57    |    3.71    |    3.30    |    3.54    |
          Total Current Assets|   66.43    |   55.44    |   38.66    |   32.75    |   35.40    |   29.56    |   30.77    |   33.22    |   34.28    |   35.91    |   36.07    |
                      Net PP&E|    6.22    |    6.34    |    6.68    |    8.78    |    8.02    |    8.90    |    7.74    |    8.40    |    9.00    |   11.29    |   11.33    |
                   Intangibles|    1.18    |    1.44    |    3.81    |    3.04    |    2.78    |    3.78    |    3.10    |    2.68    |    2.14    |     —      |     —      |
        Other Long-Term Assets|   26.17    |   36.78    |   50.85    |   55.43    |   53.80    |   57.77    |   58.39    |   55.70    |   54.59    |   52.79    |   52.60    |
                  Total Assets|   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |
              Accounts Payable|   11.79    |   15.98    |   12.57    |   12.03    |   10.81    |   13.02    |   12.22    |   11.59    |   13.07    |   15.28    |    8.90    |
               Short-Term Debt|     —      |     —      |     —      |     —      |     —      |    2.72    |    3.79    |    3.61    |    4.92    |    5.67    |    6.56    |
                 Taxes Payable|    0.91    |    0.28    |    0.98    |    0.87    |    0.58    |    0.52    |     —      |     —      |     —      |     —      |     —      |
           Accrued Liabilities|    7.20    |    5.87    |    6.46    |    1.45    |    2.06    |    2.80    |    8.67    |    6.85    |    6.86    |     —      |     —      |
  Other Short-Term Liabilities|    4.32    |    5.43    |    4.02    |    7.54    |    7.65    |    8.31    |    3.08    |    2.51    |    2.01    |   11.00    |   11.96    |
     Total Current Liabilities|   24.22    |   27.56    |   24.04    |   21.89    |   21.09    |   27.37    |   27.75    |   24.56    |   26.86    |   31.95    |   27.42    |
                Long-Term Debt|     —      |     —      |     —      |     —      |    8.19    |   12.50    |   18.41    |   23.45    |   25.90    |   25.63    |   26.37    |
   Other Long-Term Liabilities|    9.17    |    8.87    |   10.13    |   10.97    |   11.03    |   12.02    |   12.76    |   12.12    |   11.52    |   13.12    |   15.25    |
             Total Liabilities|   33.39    |   36.43    |   34.16    |   32.86    |   40.31    |   51.89    |   58.91    |   60.13    |   64.28    |   70.70    |   69.05    |
    Total Stockholders' Equity|   66.61    |   63.57    |   65.84    |   67.14    |   59.69    |   48.11    |   41.09    |   39.87    |   35.72    |   29.30    |   30.95    |
    Total Liabilities & Equity|   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |   100.00   |
    Liquidity/Financial Health|  2009-09   |  2010-09   |  2011-09   |  2012-09   |  2013-09   |  2014-09   |  2015-09   |  2016-09   |  2017-09   |  2018-09   | Latest Qtr |
                 Current Ratio|    2.74    |    2.01    |    1.61    |    1.50    |    1.68    |    1.08    |    1.11    |    1.35    |    1.28    |    1.12    |    1.32    |
                   Quick Ratio|    2.48    |    1.72    |    1.35    |    1.24    |    1.40    |    0.82    |    0.89    |    1.22    |    1.09    |    0.99    |    1.13    |
            Financial Leverage|    1.50    |    1.57    |    1.52    |    1.49    |    1.68    |    2.08    |    2.43    |    2.51    |    2.80    |    3.41    |    3.23    |
                   Debt/Equity|     —      |     —      |     —      |     —      |    0.14    |    0.26    |    0.45    |    0.59    |    0.73    |    0.87    |    0.85    |
                    Efficiency|  2009-09   |  2010-09   |  2011-09   |  2012-09   |  2013-09   |  2014-09   |  2015-09   |  2016-09   |  2017-09   |  2018-09   |    TTM     |
        Days Sales Outstanding|   24.60    |   24.82    |   18.34    |   19.01    |   25.66    |   30.51    |   26.79    |   27.59    |   26.77    |   28.21    |   20.76    |
                Days Inventory|    6.85    |    6.95    |    5.17    |    3.26    |    4.37    |    6.30    |    5.81    |    6.22    |    9.04    |    9.82    |   14.30    |
               Payables Period|   79.02    |   81.30    |   75.48    |   74.38    |   74.54    |   85.45    |   85.57    |   101.11   |   111.72   |   116.95   |   73.80    |
         Cash Conversion Cycle|   -47.58   |   -49.53   |   -51.96   |   -52.13   |   -44.50   |   -48.64   |   -52.97   |   -67.29   |   -75.91   |   -78.92   |   -38.74   |
          Receivables Turnover|   14.84    |   14.71    |   19.90    |   19.20    |   14.22    |   11.96    |   13.62    |   13.23    |   13.63    |   12.94    |   17.58    |
            Inventory Turnover|   53.28    |   52.51    |   70.53    |   112.12   |   83.45    |   57.94    |   62.82    |   58.64    |   40.37    |   37.17    |   25.53    |
         Fixed Assets Turnover|   15.86    |   16.89    |   17.26    |   13.48    |   10.67    |    9.82    |   10.85    |    8.71    |    7.54    |    7.07    |    7.00    |
                Asset Turnover|    0.99    |    1.06    |    1.13    |    1.07    |    0.89    |    0.83    |    0.89    |    0.70    |    0.66    |    0.72    |    0.73    |
于 2019-06-20T19:35:34.480 回答