0

我正在使用多处理 python 库为机器学习问题并行运行特征选择。此函数接受 pandas 数据框作为输入并返回一些数字。

当我使用mp.pool.map()一切顺利执行此功能时。但是，如果我用它替换它mp.pool.ThreadPool.map()会失败并出现以下错误：

AssertionError: Number of manager items must equal union of block items # manager items: 15, # tot_items: 20.

奇怪的是，ThreadPool直到昨天我都在正常运行代码。然后，我尝试重新运行它并开始收到这些错误。我需要ThreadPool，因为这是一个 IO 绑定作业，与pool.

编辑：代码是这样的（python 2.7）：

import multiprocessing as mp
import pandas as pd (version 0.22.0)

def main_functionality(df, params):
    df = df[params['feature']]
    #Run 5-fold cross-validation
        data_df = pd.DataFrame(....)
        pred_df = pred_df.append(data_df)
    return statistics from pred_df

def a_function(df_init, feature, params_init):

    params = dict(params_init)
    df = df_init.copy()

    params['feature'] = feature
    try:
        results = main_functionality(df, params)
    except:
        results = (0,0,0)

    return results

def b_function(df, features):
    pool = mp.pool.ThreadPool(4)
    params = {...}
    results = pool.map(a_function,(df, feature, params) for f in features))

    results_df = pd.DataFrame(results)
    results_df.to_csv(...)

if __name__ == '__main__':
    df = read.csv(...) # A big CSV file (i.e. few GBs)
    features = [i for i in df.columns if i ....]

    b_function(df, features)
4

0 回答 0