2

尝试按照本文对不平衡分类执行过采样。我的班级比例约为8：1。

https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets/notebook

我对管道+编码结构感到困惑。

  • 您应该在训练/测试拆分后过度采样吗？
    • 如果是这样，您如何处理目标标签从 X 中删除的事实？我尝试保留它，然后执行过采样，然后在 X_train/X_test 上删除标签并替换我的管道中的新训练集但是我得到错误“找到具有不一致数量的样本的输入变量”，因为形状不一致，因为新的结束-sampling df 以 50/50 标签分布加倍。

我理解这个问题但是当想要执行过采样以减少类不平衡时如何解决这个问题？


    X = df
    #X = df.drop("label", axis=1)
    y = df["label"]

    X_train,\
    X_test,\
    y_train,\
    y_test = train_test_split(X,\
                              y,\
                              test_size=0.2,\
                              random_state=11,\
                              shuffle=True,\
                              stratify=target)

    target_count = df.label.value_counts()
    print('Class 1:', target_count[0])
    print('Class 0:', target_count[1])
    print('Proportion:', round(target_count[0] / target_count[1], 2), ': 1')

    target_count.plot(kind='bar', title='Count (target)');

    # Class count
    count_class_index_0, count_class_index_1 = X_train.label.value_counts()

    # Divide by class
    count_class_index_0 = X_train[X_train['label'] == '1']
    count_class_index_1 = X_train[X_train['label'] == '0']

    df_class_1_over = df_class_1.sample(count_class_index_0, replace=True)
    df_test_over = pd.concat([count_class_index_0, df_class_1_over], axis=0)

    print('Random over-sampling:')
    print(df_test_over.label.value_counts())

    Random over-sampling:
    1    12682
    0      12682

    df_test_over.label.value_counts().plot(kind='bar', title='Count (target)')

    # drop label for new X_train and X_test
    X_train_OS = df_test_over.drop("label", axis=1)
    X_test = X_test.drop("label", axis=1)

    print(X_train_OS.shape)
    print(X_test.shape)

    print(y_train.shape)
    print(y_test.shape)

    (25364, 9)
    (3552, 9)
    (14207,)
    (3552,)

    cat_transformer = Pipeline(steps=[
        ('cat_imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('cat_ohe', OneHotEncoder(handle_unknown='ignore'))])

    num_transformer = Pipeline(steps=[
        ('num_imputer', SimpleImputer(strategy='constant', fill_value=0)),
        ('num_scaler', StandardScaler())])

    text_transformer_0 = Pipeline(steps=[
        ('text_bow', CountVectorizer(lowercase=True,\
                                     token_pattern=SPLIT_PATTERN,\
                                     stop_words=stopwords))])
    # SelectKBest()
    # TruncatedSVD()

    text_transformer_1 = Pipeline(steps=[
        ('text_bow', CountVectorizer(lowercase=True,\
                                     token_pattern=SPLIT_PATTERN,\
                                     stop_words=stopwords))])
    # SelectKBest()
    # TruncatedSVD()

    FE = ColumnTransformer(
        transformers=[
            ('cat', cat_transformer, CAT_FEATURES),
            ('num', num_transformer, NUM_FEATURES),
            ('text0', text_transformer_0, TEXT_FEATURES[0]),
            ('text1', text_transformer_1, TEXT_FEATURES[1])])

    pipe = Pipeline(steps=[('feature_engineer', FE),
                         ("scales", MaxAbsScaler()),
                         ('rand_forest', RandomForestClassifier(n_jobs=-1, class_weight='balanced'))])

    random_grid = {"rand_forest__max_depth": [3, 10, 100, None],\
                  "rand_forest__n_estimators": sp_randint(10, 100),\
                  "rand_forest__max_features": ["auto", "sqrt", "log2", None],\
                  "rand_forest__bootstrap": [True, False],\
                  "rand_forest__criterion": ["gini", "entropy"]}

    strat_shuffle_fold = StratifiedKFold(n_splits=5,\
      random_state=123,\
      shuffle=True)

    cv_train = RandomizedSearchCV(pipe, param_distributions=random_grid, cv=strat_shuffle_fold)
    cv_train.fit(X_train_OS, y_train)

    from sklearn.metrics import classification_report, confusion_matrix
    preds = cv_train.predict(X_test)
    print(confusion_matrix(y_test, preds))
    print(classification_report(y_test, preds))
4

2 回答 2

2

您在这里遇到的问题很容易（并且可以说更优雅）由SMOTE. 它易于使用，并且允许您保留X_train, X_test, y_train, y_test语法，train_test_split因为它将同时对 X 和 y 执行过采样。

from imblearn.over_sampling import SMOTE

X_train, X_test, y_train, y_test = train_test_split(X,y)
sm = SMOTE(random_state=42)
X_resampled, y_resampled = sm.fit_resample(X_train, y_train)
于 2019-04-24T00:08:10.547 回答
0

So I believe I solved my own question ... the problem was how I was splitting the data ... I normally always follow the standard X_train, X_test, y_train, y_test train_test_split however it was causing the row count mismatch in the X_train and y_train when over-sampling so I did this instead and everything appears to be working. Please let me know if anyone has any recommendations! Thanks!

features = df_
target = df_l["label"]

train_set, test_set = train_test_split(features, test_size=0.2,\
                          random_state=11,\
                          shuffle=True)

print(train_set.shape)
print(test_set.shape)

(11561, 10)
(2891, 10)

count_class_1, count_class_0 = train_set.label.value_counts()

# Divide by class
df_class_1 = train_set[train_set['label'] == 1]
df_class_0 = train_set[train_set['label'] == 0]

df_class_0_over = df_class_0.sample(count_class_1, replace=True)
df_train_OS = pd.concat([df_class_1, df_class_0_over], axis=0)

print('Random over-sampling:')
print(df_train_OS.label.value_counts())

1      10146
0    10146

df_train_OS.label.value_counts().plot(kind='bar', title='Count (target)');

X_train_OS = df_train_OS.drop("label", axis=1)
y_train_OS = df_train_OS["label"]
X_test = test_set.drop("label", axis=1)
y_test = test_set["label"]

print(X_train_OS.shape)
print(y_train_OS.shape)
print(X_test.shape)
print(y_test.shape)

(20295, 9)
(20295,)
(2891, 9)
(2891,)
于 2019-04-23T20:41:52.350 回答