2

SMOTE由于少数标签，我有一个多类文本数据。我已经这样做了，但是我得到了稀疏矩阵作为我的输出。

有没有办法在 SMOTE 之后取回文本数据？

这是我的代码示例：

X_train = df['transcript']
y_train = df['label']
from imblearn.over_sampling import SMOTE 
sm = SMOTE(random_state = 2) 
X_train_res, y_train_res = sm.fit_sample(X_train, y_train)
4

2 回答 2

2

SMOTE.fit_samplelabel_binarize从 Scikit-learn 内部使用： https ://github.com/scikit-learn-contrib/imbalanced-learn/blob/12b2e0d/imblearn/base.py#L87

您应该在应用之前手动使用sklearn.preprocessing.LabelBinarizer这些y值SMOTE。

from imblearn.over_sampling import SMOTE
from sklearn.preprocessing import LabelBinarizer

sm = SMOTE(random_state = 2)
lb = LabelBinarizer()
y_train_bin = lb.fit_transform(y_train)
X_train_res, y_train_res_bin = sm.fit_sample(X_train, y_train_bin)

LabelBinarizer.inverse_transform然后您可以从拟合方法中恢复文本标签：

y_train_res = lb.inverse_transform(y_train_res_bin)
于 2020-07-17T14:18:25.347 回答
2

实际上SMOTE预计X只是数字数据。这不是标签的问题，标签可以是字符串。

阅读此处了解 SMOTE 如何在内部工作。基本上，它使用所选邻居的凸组合为少数类创建合成数据点。

TfidfVectorizer因此，使用或将您的文本数据（成绩单）转换为数字CountVectorizer。您可以使用inverse_transform这些矢量化器的方法来取回文本，但问题是您会丢失单词的顺序。

import pandas as pd
df = pd.DataFrame({'transcripts': ['I want to check this',
                                    'how about one more sentence',
                                    'hopefully this works well fr you',
                                    'I want to check this',
                                    'This is the last sentence or transcript'],
                    'labels': ['good','bad', 'bad', 'good','bad']})
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.fit_transform(df['transcripts'])

from imblearn.over_sampling import SMOTE 
sm = SMOTE(k_neighbors=1, random_state = 2) 
X_train_res, y_train_res = sm.fit_sample(X, df.labels) 


vec.inverse_transform(X_train_res)
# [array(['this', 'check', 'to', 'want'], dtype='<U10'),
#  array(['sentence', 'more', 'one', 'about', 'how'], dtype='<U10'),
#  array(['you', 'fr', 'well', 'works', 'hopefully', 'this'], dtype='<U10'),
#  array(['this', 'check', 'to', 'want'], dtype='<U10'),
#  array(['transcript', 'or', 'last', 'the', 'is', 'sentence', 'this'],
#        dtype='<U10'),
#  array(['want', 'to', 'check', 'this'], dtype='<U10')]
于 2020-07-17T14:54:33.903 回答