-2

我有以下从网上下载的代码。我已经通过分解它进行了修改,以便在雅虎上更容易。

我的问题是关于 python(一般)和 python+pandas 能够处理比我在这里尝试的更多的数据的能力。当我运行这段代码时,如果我计算了符号之间的所有相关性,它最终会阻塞(参见“相反它显示这个”部分)。如果我删除一些计算,它似乎没问题。我不确定是什么东西,我认为是熊猫?

分解此代码的正确方法是什么,以便它不会失去其简洁性[而不是使用矢量化的循环],并且仍然能够处理更多数据?我希望能够处理存储在文件中的 10 年 1 分钟数据,如果它甚至不能处理一年的日常数据,它就永远无法在该数据集上工作。

所以我的问题是:

修复这个程序的正确方法是什么(希望我可以概括)以便它适用于 DOW 30 符号?

import pandas
from matplotlib.pyplot import show, legend
from datetime import datetime
from matplotlib import finance
import numpy

# 2011 to 2012
start = datetime(2011, 01, 01)
end = datetime(2012, 01, 01)

symbolsAK = ["AA", "AXP", "BA", "BAC", "CAT",
             "CSCO", "CVX", "DD", "DIS", "GE", "HD",
             "HPQ", "IBM", "INTC", "JNJ", "JPM",
             "KO"]
symbolsMP = ["MCD", "MMM", "MRK", "MSFT", "PFE", "PG"]
#symbolsTX = ["T", "TRV", "UNH", "UTX", "VZ", "WMT", "XOM"]

symbols = symbolsAK
symbols = symbols + symbolsMP
#symbols = symbols + symbolsTX

quotesAK = [finance.quotes_historical_yahoo(symbol, start, end, asobject=True)
            for symbol in symbolsAK]
quotesMP = [finance.quotes_historical_yahoo(symbol, start, end, asobject=True)
            for symbol in symbolsMP]
#quotesTX = [finance.quotes_historical_yahoo(symbol, start, end, asobject=True)
#            for symbol in symbolsTX]

quotes = quotesAK
quotes = quotes + quotesMP
#quotes = quotes + quotesTX

close = numpy.array([q.close for q in quotes]).astype(numpy.float)
dates = numpy.array([q.date for q in quotes])

data = {}

for i in xrange(len(symbols)):
   data[symbols[i]] = numpy.diff(numpy.log(close[i]))

df = pandas.DataFrame(data, index=dates[0][:-1], columns=symbols)


print df.corr()

它应该看起来像(某物)的输出[缩短]

#           AA       AXP        BA       BAC       CAT
#AA   1.000000  0.768484  0.758264  0.737625  0.837643
#AXP  0.768484  1.000000  0.746898  0.760043  0.736337
#BA   0.758264  0.746898  1.000000  0.657075  0.770696
#BAC  0.737625  0.760043  0.657075  1.000000  0.657113
#CAT  0.837643  0.736337  0.770696  0.657113  1.000000

相反,它显示了这一点

<class 'pandas.core.frame.DataFrame'>
Index: 23 entries, AA to PG
Data columns (total 23 columns):
AA      23  non-null values
AXP     23  non-null values
BA      23  non-null values
BAC     23  non-null values
CAT     23  non-null values
CSCO    23  non-null values
CVX     23  non-null values
DD      23  non-null values
DIS     23  non-null values
GE      23  non-null values
HD      23  non-null values
HPQ     23  non-null values
IBM     23  non-null values
INTC    23  non-null values
JNJ     23  non-null values
JPM     23  non-null values
KO      23  non-null values
MCD     23  non-null values
MMM     23  non-null values
MRK     23  non-null values
MSFT    23  non-null values
PFE     23  non-null values
PG      23  non-null values
dtypes: float64(23)
4

1 回答 1

5

它认为这不是内存或速度问题,而只是 pandas 控制台输出格式的问题(请参阅http://pandas.pydata.org/pandas-docs/stable/basics.html#working-with-package-options

如果 DataFrame 太大而无法在控制台中显示,pandas 将在您到达此处时提供摘要视图(描述每列中有多少非空值)。在我的计算机上,这是最多 20 列和 60 行。但是您可以更改此设置以显示更大的数据框:

这是您获得的摘要视图:

In [2]: df.corr()
Out[2]: 
<class 'pandas.core.frame.DataFrame'>
Index: 30 entries, AA to XOM
Data columns (total 30 columns):
AA      30  non-null values
AXP     30  non-null values
BA      30  non-null values
BAC     30  non-null values
CAT     30  non-null values
CSCO    30  non-null values
CVX     30  non-null values
DD      30  non-null values
DIS     30  non-null values
GE      30  non-null values
HD      30  non-null values
HPQ     30  non-null values
IBM     30  non-null values
INTC    30  non-null values
JNJ     30  non-null values
JPM     30  non-null values
KO      30  non-null values
MCD     30  non-null values
MMM     30  non-null values
MRK     30  non-null values
MSFT    30  non-null values
PFE     30  non-null values
PG      30  non-null values
T       30  non-null values
TRV     30  non-null values
UNH     30  non-null values
UTX     30  non-null values
VZ      30  non-null values
WMT     30  non-null values
XOM     30  non-null values
dtypes: float64(30)

您可以更改要显示的列数:

In [5]: pandas.options.display.max_columns = 50

现在您将显示整个数据框:

In [6]: df.corr()
Out[6]: 
            AA       AXP        BA       BAC       CAT      CSCO       CVX  \
AA    1.000000  0.768692  0.758487  0.738168  0.838511  0.584911  0.785955   
AXP   0.768692  1.000000  0.746401  0.760255  0.736557  0.553068  0.703163   
BA    0.758487  0.746401  1.000000  0.657093  0.770767  0.540786  0.721736   
BAC   0.738168  0.760255  0.657093  1.000000  0.657254  0.518776  0.620971   
CAT   0.838511  0.736557  0.770767  0.657254  1.000000  0.572002  0.798452   
CSCO  0.584911  0.553068  0.540786  0.518776  0.572002  1.000000  0.577381   
CVX   0.785955  0.703163  0.721736  0.620971  0.798452  0.577381  1.000000   
DD    0.851112  0.759933  0.760574  0.675753  0.851070  0.582742  0.803719   
DIS   0.751486  0.742574  0.782171  0.660317  0.746241  0.554223  0.713484   
GE    0.765963  0.755788  0.752602  0.699929  0.724883  0.576092  0.741695   
HD    0.614588  0.647022  0.661556  0.575308  0.621294  0.499147  0.647491   
HPQ   0.595110  0.509375  0.573672  0.453443  0.594590  0.427063  0.487018   
IBM   0.662302  0.635524  0.701338  0.501511  0.664953  0.525319  0.625367   
INTC  0.610633  0.587957  0.633674  0.463658  0.634462  0.543521  0.580799   
JNJ   0.676678  0.674556  0.668016  0.569427  0.677017  0.562765  0.707574   
JPM   0.799386  0.803539  0.695899  0.843875  0.726496  0.583126  0.701845   
KO    0.632974  0.649398  0.686937  0.504384  0.621104  0.496410  0.684645   
MCD   0.590209  0.619359  0.608288  0.482579  0.557322  0.467640  0.584303   
MMM   0.807796  0.760495  0.760528  0.674455  0.805890  0.598801  0.771571   
MRK   0.683526  0.675286  0.683141  0.594330  0.630897  0.528784  0.669215   
MSFT  0.708997  0.670527  0.675077  0.579668  0.672689  0.629810  0.676658   
PFE   0.692207  0.661628  0.661427  0.580332  0.653362  0.524557  0.695621   
PG    0.533456  0.638269  0.634056  0.461198  0.569150  0.500971  0.628252   
T     0.662901  0.658365  0.647409  0.585394  0.611656  0.454966  0.659306   
TRV   0.697297  0.690480  0.693580  0.692810  0.679448  0.550598  0.707205   
UNH   0.626418  0.645823  0.644480  0.577014  0.642021  0.502656  0.628023   
UTX   0.800084  0.770001  0.818340  0.650226  0.844137  0.611440  0.779919   
VZ    0.613422  0.613442  0.576083  0.536747  0.589583  0.472622  0.627931   
WMT   0.517511  0.575717  0.587670  0.479790  0.538195  0.515317  0.556602   
XOM   0.747023  0.699433  0.734805  0.598516  0.753005  0.581742  0.905136   

            DD       DIS        GE        HD       HPQ       IBM      INTC  \
AA    0.851112  0.751486  0.765963  0.614588  0.595110  0.662302  0.610633   
AXP   0.759933  0.742574  0.755788  0.647022  0.509375  0.635524  0.587957   
BA    0.760574  0.782171  0.752602  0.661556  0.573672  0.701338  0.633674   
BAC   0.675753  0.660317  0.699929  0.575308  0.453443  0.501511  0.463658   
CAT   0.851070  0.746241  0.724883  0.621294  0.594590  0.664953  0.634462   
CSCO  0.582742  0.554223  0.576092  0.499147  0.427063  0.525319  0.543521   
CVX   0.803719  0.713484  0.741695  0.647491  0.487018  0.625367  0.580799   
DD    1.000000  0.773421  0.768493  0.660224  0.587773  0.674010  0.627005   
DIS   0.773421  1.000000  0.768324  0.643008  0.609767  0.678413  0.607358   
GE    0.768493  0.768324  1.000000  0.649000  0.553156  0.656494  0.625745   
HD    0.660224  0.643008  0.649000  1.000000  0.459635  0.575951  0.572010   
HPQ   0.587773  0.609767  0.553156  0.459635  1.000000  0.582698  0.548928   
IBM   0.674010  0.678413  0.656494  0.575951  0.582698  1.000000  0.633732   
INTC  0.627005  0.607358  0.625745  0.572010  0.548928  0.633732  1.000000   
JNJ   0.714763  0.654975  0.683914  0.589519  0.494923  0.602186  0.571545   
JPM   0.767345  0.737792  0.795344  0.601889  0.521005  0.602322  0.569887   
KO    0.696257  0.656332  0.674888  0.631668  0.443318  0.694586  0.574671   
MCD   0.583090  0.569733  0.556076  0.608105  0.337828  0.569540  0.491635   
MMM   0.799806  0.775277  0.797455  0.654009  0.578911  0.676061  0.650945   
MRK   0.671173  0.690316  0.687744  0.574417  0.448651  0.627232  0.547941   
MSFT  0.703819  0.684609  0.679975  0.631967  0.521019  0.682591  0.662063   
PFE   0.690313  0.650876  0.706027  0.638876  0.474586  0.623725  0.550615   
PG    0.617922  0.611371  0.613507  0.556490  0.431871  0.610044  0.551303   
T     0.686551  0.669819  0.680358  0.597554  0.494590  0.678023  0.545211   
TRV   0.710612  0.710623  0.677900  0.624701  0.482071  0.589566  0.608157   
UNH   0.640953  0.651940  0.632988  0.612200  0.407039  0.611192  0.547778   
UTX   0.815454  0.786531  0.777018  0.673500  0.610108  0.748190  0.692028   
VZ    0.630868  0.617529  0.684984  0.567786  0.424424  0.586035  0.508896   
WMT   0.566875  0.581024  0.556110  0.692174  0.374181  0.489173  0.489745   
XOM   0.774908  0.720534  0.761815  0.639149  0.523942  0.675966  0.610824   

           JNJ       JPM        KO       MCD       MMM       MRK      MSFT  \
AA    0.676678  0.799386  0.632974  0.590209  0.807796  0.683526  0.708997   
AXP   0.674556  0.803539  0.649398  0.619359  0.760495  0.675286  0.670527   
BA    0.668016  0.695899  0.686937  0.608288  0.760528  0.683141  0.675077   
BAC   0.569427  0.843875  0.504384  0.482579  0.674455  0.594330  0.579668   
CAT   0.677017  0.726496  0.621104  0.557322  0.805890  0.630897  0.672689   
CSCO  0.562765  0.583126  0.496410  0.467640  0.598801  0.528784  0.629810   
CVX   0.707574  0.701845  0.684645  0.584303  0.771571  0.669215  0.676658   
DD    0.714763  0.767345  0.696257  0.583090  0.799806  0.671173  0.703819   
DIS   0.654975  0.737792  0.656332  0.569733  0.775277  0.690316  0.684609   
GE    0.683914  0.795344  0.674888  0.556076  0.797455  0.687744  0.679975   
HD    0.589519  0.601889  0.631668  0.608105  0.654009  0.574417  0.631967   
HPQ   0.494923  0.521005  0.443318  0.337828  0.578911  0.448651  0.521019   
IBM   0.602186  0.602322  0.694586  0.569540  0.676061  0.627232  0.682591   
INTC  0.571545  0.569887  0.574671  0.491635  0.650945  0.547941  0.662063   
JNJ   1.000000  0.649433  0.661615  0.591725  0.736881  0.720435  0.606554   
JPM   0.649433  1.000000  0.584480  0.520379  0.764575  0.632774  0.665440   
KO    0.661615  0.584480  1.000000  0.659553  0.684177  0.685925  0.630570   
MCD   0.591725  0.520379  0.659553  1.000000  0.639054  0.610580  0.569149   
MMM   0.736881  0.764575  0.684177  0.639054  1.000000  0.688326  0.705497   
MRK   0.720435  0.632774  0.685925  0.610580  0.688326  1.000000  0.620179   
MSFT  0.606554  0.665440  0.630570  0.569149  0.705497  0.620179  1.000000   
PFE   0.710511  0.627674  0.630108  0.599965  0.687126  0.723702  0.620668   
PG    0.664540  0.593982  0.660393  0.566643  0.655894  0.646314  0.579561   
T     0.619650  0.661625  0.637338  0.555407  0.645148  0.642262  0.608858   
TRV   0.625928  0.728347  0.675313  0.598593  0.739503  0.654874  0.600154   
UNH   0.620315  0.593633  0.618663  0.534163  0.610730  0.611829  0.562731   
UTX   0.725406  0.718998  0.710645  0.624908  0.848424  0.694618  0.723456   
VZ    0.634423  0.606947  0.592759  0.522129  0.635813  0.620811  0.564451   
WMT   0.574580  0.552472  0.568968  0.571420  0.610972  0.571786  0.579684   
XOM   0.724311  0.712734  0.710473  0.567184  0.748141  0.699390  0.703494   

           PFE        PG         T       TRV       UNH       UTX        VZ  \
AA    0.692207  0.533456  0.662901  0.697297  0.626418  0.800084  0.613422   
AXP   0.661628  0.638269  0.658365  0.690480  0.645823  0.770001  0.613442   
BA    0.661427  0.634056  0.647409  0.693580  0.644480  0.818340  0.576083   
BAC   0.580332  0.461198  0.585394  0.692810  0.577014  0.650226  0.536747   
CAT   0.653362  0.569150  0.611656  0.679448  0.642021  0.844137  0.589583   
CSCO  0.524557  0.500971  0.454966  0.550598  0.502656  0.611440  0.472622   
CVX   0.695621  0.628252  0.659306  0.707205  0.628023  0.779919  0.627931   
DD    0.690313  0.617922  0.686551  0.710612  0.640953  0.815454  0.630868   
DIS   0.650876  0.611371  0.669819  0.710623  0.651940  0.786531  0.617529   
GE    0.706027  0.613507  0.680358  0.677900  0.632988  0.777018  0.684984   
HD    0.638876  0.556490  0.597554  0.624701  0.612200  0.673500  0.567786   
HPQ   0.474586  0.431871  0.494590  0.482071  0.407039  0.610108  0.424424   
IBM   0.623725  0.610044  0.678023  0.589566  0.611192  0.748190  0.586035   
INTC  0.550615  0.551303  0.545211  0.608157  0.547778  0.692028  0.508896   
JNJ   0.710511  0.664540  0.619650  0.625928  0.620315  0.725406  0.634423   
JPM   0.627674  0.593982  0.661625  0.728347  0.593633  0.718998  0.606947   
KO    0.630108  0.660393  0.637338  0.675313  0.618663  0.710645  0.592759   
MCD   0.599965  0.566643  0.555407  0.598593  0.534163  0.624908  0.522129   
MMM   0.687126  0.655894  0.645148  0.739503  0.610730  0.848424  0.635813   
MRK   0.723702  0.646314  0.642262  0.654874  0.611829  0.694618  0.620811   
MSFT  0.620668  0.579561  0.608858  0.600154  0.562731  0.723456  0.564451   
PFE   1.000000  0.576964  0.597129  0.642421  0.590014  0.675389  0.628915   
PG    0.576964  1.000000  0.668227  0.607292  0.492360  0.677481  0.591762   
T     0.597129  0.668227  1.000000  0.657551  0.604891  0.648988  0.756705   
TRV   0.642421  0.607292  0.657551  1.000000  0.665523  0.683029  0.587940   
UNH   0.590014  0.492360  0.604891  0.665523  1.000000  0.660746  0.486421   
UTX   0.675389  0.677481  0.648988  0.683029  0.660746  1.000000  0.605494   
VZ    0.628915  0.591762  0.756705  0.587940  0.486421  0.605494  1.000000   
WMT   0.552283  0.618861  0.529654  0.619793  0.499349  0.601957  0.549769   
XOM   0.715801  0.666614  0.692532  0.706332  0.654499  0.776531  0.617919   

           WMT       XOM  
AA    0.517511  0.747023  
AXP   0.575717  0.699433  
BA    0.587670  0.734805  
BAC   0.479790  0.598516  
CAT   0.538195  0.753005  
CSCO  0.515317  0.581742  
CVX   0.556602  0.905136  
DD    0.566875  0.774908  
DIS   0.581024  0.720534  
GE    0.556110  0.761815  
HD    0.692174  0.639149  
HPQ   0.374181  0.523942  
IBM   0.489173  0.675966  
INTC  0.489745  0.610824  
JNJ   0.574580  0.724311  
JPM   0.552472  0.712734  
KO    0.568968  0.710473  
MCD   0.571420  0.567184  
MMM   0.610972  0.748141  
MRK   0.571786  0.699390  
MSFT  0.579684  0.703494  
PFE   0.552283  0.715801  
PG    0.618861  0.666614  
T     0.529654  0.692532  
TRV   0.619793  0.706332  
UNH   0.499349  0.654499  
UTX   0.601957  0.776531  
VZ    0.549769  0.617919  
WMT   1.000000  0.550944  
XOM   0.550944  1.000000  

或者,另一种方法是选择部分数据以仅查看该数据。在下面的代码中,我使用以下方法对表的前 10 行和前 10 列进行切片ix

In [7]: df.corr().ix[0:10,0:10]
Out[7]: 
            AA       AXP        BA       BAC       CAT      CSCO       CVX  \
AA    1.000000  0.768692  0.758487  0.738168  0.838511  0.584911  0.785955   
AXP   0.768692  1.000000  0.746401  0.760255  0.736557  0.553068  0.703163   
BA    0.758487  0.746401  1.000000  0.657093  0.770767  0.540786  0.721736   
BAC   0.738168  0.760255  0.657093  1.000000  0.657254  0.518776  0.620971   
CAT   0.838511  0.736557  0.770767  0.657254  1.000000  0.572002  0.798452   
CSCO  0.584911  0.553068  0.540786  0.518776  0.572002  1.000000  0.577381   
CVX   0.785955  0.703163  0.721736  0.620971  0.798452  0.577381  1.000000   
DD    0.851112  0.759933  0.760574  0.675753  0.851070  0.582742  0.803719   
DIS   0.751486  0.742574  0.782171  0.660317  0.746241  0.554223  0.713484   
GE    0.765963  0.755788  0.752602  0.699929  0.724883  0.576092  0.741695   

            DD       DIS        GE  
AA    0.851112  0.751486  0.765963  
AXP   0.759933  0.742574  0.755788  
BA    0.760574  0.782171  0.752602  
BAC   0.675753  0.660317  0.699929  
CAT   0.851070  0.746241  0.724883  
CSCO  0.582742  0.554223  0.576092  
CVX   0.803719  0.713484  0.741695  
DD    1.000000  0.773421  0.768493  
DIS   0.773421  1.000000  0.768324  
GE    0.768493  0.768324  1.000000  

为了说明不是pandas不能处理数据量的问题,关联表的计算只需要一毫秒:

In [3]: %timeit df.corr()
1000 loops, best of 3: 1.18 ms per loop
于 2013-08-25T18:43:00.937 回答