124

我正在学习将分类变量转换为机器学习分类器的数字的不同方法。我遇到了这个pd.get_dummies方法和sklearn.preprocessing.OneHotEncoder()我想看看它们在性能和使用方面有何不同。

OneHotEncoder()我在https://xgdgsc.wordpress.com/2015/03/20/note-on-using-onehotencoder-in-scikit-learn-to-work-on-categorical-features/上找到了有关如何使用的教程文档对此sklearn功能没有太大帮助。我有一种感觉我做得不对……但是

pd.dummies有人可以解释使用over的优缺点，sklearn.preprocessing.OneHotEncoder()反之亦然吗？我知道这OneHotEncoder()为您提供了一个稀疏矩阵，但除此之外，我不确定它是如何使用的以及该pandas方法有什么好处。我使用它效率低下吗？

import pandas as pd
import numpy as np
from sklearn.datasets import load_iris
sns.set()

%matplotlib inline

#Iris Plot
iris = load_iris()
n_samples, m_features = iris.data.shape

#Load Data
X, y = iris.data, iris.target
D_target_dummy = dict(zip(np.arange(iris.target_names.shape[0]), iris.target_names))

DF_data = pd.DataFrame(X,columns=iris.feature_names)
DF_data["target"] = pd.Series(y).map(D_target_dummy)
#sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \
#0                  5.1               3.5                1.4               0.2   
#1                  4.9               3.0                1.4               0.2   
#2                  4.7               3.2                1.3               0.2   
#3                  4.6               3.1                1.5               0.2   
#4                  5.0               3.6                1.4               0.2   
#5                  5.4               3.9                1.7               0.4   

DF_dummies = pd.get_dummies(DF_data["target"])
#setosa  versicolor  virginica
#0         1           0          0
#1         1           0          0
#2         1           0          0
#3         1           0          0
#4         1           0          0
#5         1           0          0

from sklearn.preprocessing import OneHotEncoder, LabelEncoder
def f1(DF_data):
    Enc_ohe, Enc_label = OneHotEncoder(), LabelEncoder()
    DF_data["Dummies"] = Enc_label.fit_transform(DF_data["target"])
    DF_dummies2 = pd.DataFrame(Enc_ohe.fit_transform(DF_data[["Dummies"]]).todense(), columns = Enc_label.classes_)
    return(DF_dummies2)

%timeit pd.get_dummies(DF_data["target"])
#1000 loops, best of 3: 777 µs per loop

%timeit f1(DF_data)
#100 loops, best of 3: 2.91 ms per loop
4

4 回答 4

213

For machine learning, you almost definitely want to use sklearn.OneHotEncoder. For other tasks like simple analyses, you might be able to use pd.get_dummies, which is a bit more convenient.

Note that sklearn.OneHotEncoder has been updated in the latest version so that it does accept strings for categorical variables, as well as integers.

The crux of it is that the sklearn encoder creates a function which persists and can then be applied to new data sets which use the same categorical variables, with consistent results.

from sklearn.preprocessing import OneHotEncoder

# Create the encoder.
encoder = OneHotEncoder(handle_unknown="ignore")
encoder.fit(X_train)    # Assume for simplicity all features are categorical.

# Apply the encoder.
X_train = encoder.transform(X_train)
X_test = encoder.transform(X_test)

Note how we apply the same encoder we created via X_train to the new data set X_test.

Consider what happens if X_test contains different levels than X_train for one of its variables. For example, let's say X_train["color"] contains only "red" and "green", but in addition to those, X_test["color"] sometimes contains "blue".

If we use pd.get_dummies, X_test will end up with an additional "color_blue" column which X_train doesn't have, and the inconsistency will probably break our code later on, especially if we are feeding X_test to an sklearn model which we trained on X_train.

And if we want to process the data like this in production, where we're receiving a single example at a time, pd.get_dummies won't be of use.

With sklearn.OneHotEncoder on the other hand, once we've created the encoder, we can reuse it to produce the same output every time, with columns only for "red" and "green". And we can explicitly control what happens when it encounters the new level "blue": if we think that's impossible, then we can tell it to throw an error with handle_unknown="error"; otherwise we can tell it to continue and simply set the red and green columns to 0, with handle_unknown="ignore".

于 2019-06-12T16:53:47.560 回答
68

OneHotEncoder不能直接处理字符串值。如果您的标称特征是字符串，那么您需要首先将它们映射为整数。

pandas.get_dummies有点相反。默认情况下，它只将字符串列转换为 one-hot 表示，除非指定了列。

于 2016-07-29T05:07:19.750 回答
7

我真的很喜欢卡尔的回答并投了赞成票。我将稍微扩展 Carl 的示例，以便更多人希望能够欣赏 pd.get_dummies 可以处理未知数。下面的两个示例表明 pd.get_dummies 在处理未知时可以完成与 OHE 相同的事情。

# data is from @dzieciou's comment above
>>> data =pd.DataFrame(pd.Series(['good','bad','worst','good', 'good', 'bad']))
# new_data has two values that data does not have. 
>>> new_data= pd.DataFrame(
pd.Series(['good','bad','worst','good', 'good', 'bad','excellent', 'perfect']))

使用 pd.get_dummies

>>> df = pd.get_dummies(data)
>>> col_list = df.columns.tolist()
>>> print(df)
   0_bad  0_good  0_worst
0      0       1        0
1      1       0        0
2      0       0        1
3      0       1        0
4      0       1        0
5      1       0        0
6      0       0        0
7      0       0        0

>>> new_df = pd.get_dummies(new_data)
# handle unknow by using .reindex and .fillna()
>>> new_df = new_df.reindex(columns=col_list).fillna(0.00)
>>> print(new_df)
#    0_bad  0_good  0_worst
# 0      0       1        0
# 1      1       0        0
# 2      0       0        1
# 3      0       1        0
# 4      0       1        0
# 5      1       0        0
# 6      0       0        0
# 7      0       0        0

使用 OneHotEncoder

>>> encoder = OneHotEncoder(handle_unknown="ignore", sparse=False)
>>> encoder.fit(data)
>>> encoder.transform(new_data)
# array([[0., 1., 0.],
#        [1., 0., 0.],
#        [0., 0., 1.],
#        [0., 1., 0.],
#        [0., 1., 0.],
#        [1., 0., 0.],
#        [0., 0., 0.],
#        [0., 0., 0.]])
于 2019-12-25T05:55:16.807 回答
6

为什么不将列缓存或保存为生成的 get_dummies 中的变量 col_list 然后使用 pd.reindex 对齐训练数据集与测试数据集....例如：

df = pd.get_dummies(data)
col_list = df.columns.tolist()

new_df = pd.get_dummies(new_data)
new_df = new_df.reindex(columns=col_list).fillna(0.00)
于 2019-10-21T19:44:17.840 回答