1

我试图破译为什么这只是挂在 modin 上并且适用于普通熊猫：

import modin.pandas as pd

infile1 = 'D:\\test_files\\curves_crosstab.csv'
infile2 = 'D:\\test_files\\8760_crosstab.csv'
infilenames = [infile1, infile2]

outfile1 = 'D:\\test_files\\curves_sample_output.csv'
outfile2 = 'D:\\test_files\\8760_sample_output.csv'

for i in range(len(infilenames)) :
    if 'curves' in infilenames[i] :
        print("in curves")
        df = pd.read_csv(infilenames[i], header=[0,1,2,3])
        print("read curves")
        df.columns = df.columns.to_flat_index()
        print("indexed columns")
        df.columns = ['_'.join(i) for i in df.columns]
        print("joined columns")
        df2 = df.melt(id_vars=['Unnamed: 0_level_0_Unnamed: 0_level_1_Unnamed: 0_level_2_Year',
            'Unnamed: 1_level_0_Unnamed: 1_level_1_Unnamed: 1_level_2_Month',
            'Unnamed: 2_level_0_Unnamed: 2_level_1_Unnamed: 2_level_2_Day',
            'Unnamed: 3_level_0_Unnamed: 3_level_1_Unnamed: 3_level_2_Hour'])
        print("melted")
        df2 = pd.concat([df2,df2.variable.str.split('_',expand=True)],axis=1)
        del df2['variable']
        print("deleted variable column")
        df2.rename(columns={'Unnamed: 0_level_0_Unnamed: 0_level_1_Unnamed: 0_level_2_Year' : 'Year' ,
                            'Unnamed: 1_level_0_Unnamed: 1_level_1_Unnamed: 1_level_2_Month' : 'Month',
                            'Unnamed: 2_level_0_Unnamed: 2_level_1_Unnamed: 2_level_2_Day' : 'Day',
                            'Unnamed: 3_level_0_Unnamed: 3_level_1_Unnamed: 3_level_2_Hour' : 'Hour',
                            0 : 'currency', 
                            1 : 'consultant_or_case', 
                            2 : 'name', 
                            3 : 'hub', 
                            'value' : 'rate_in_local_currency'}, inplace = True)
        print("renamed")
        pd.DataFrame.to_csv(df2, path_or_buf=outfile1,index=False,encoding='utf-8')
        print("created csv")
    else :

        df = pd.read_csv(infilenames[i], encoding='cp1252')
        df2 = df.melt(id_vars=['Month','Day','Hour'])

        pd.DataFrame.to_csv(df2, path_or_buf=outfile2,index=False,encoding='utf-8')

当我在 pandas 下运行它时，它执行，但由于曲线文件的大小（约 36.5MB 输入和约 395MB 输出）平均需要 87 秒，我希望 modin 可以缩短那个时间。交换到 Modin 时的脚本运行，但它只是坐着，不做任何事情。它甚至不给我

Waiting for redis server at 127.0.0.1:14618 to respond...
Waiting for redis server at 127.0.0.1:31410 to respond...
Starting local scheduler with the following resources: {'CPU': 4, 'GPU': 0}.

我不知道这是否应该出现在控制台中，但它没有。当我进入曲线时，脚本开始读取 csv。然后它就坐着。从不做任何其他事情。我怎样才能弄清楚发生了什么？

如果重要的话，操作系统是 Windows10。

4

0 回答 0