43

I would like to calculate NN model certainty/confidence (see What my deep model doesn't know) - when NN tells me an image represents "8", I would like to know how certain it is. Is my model 99% certain it is "8" or is it 51% it is "8", but it could also be "6"? Some digits are quite ambiguous and I would like to know for which images the model is just "flipping a coin".

I have found some theoretical writings about this but I have trouble putting this in code. If I understand correctly, I should evaluate a testing image multiple times while "killing off" different neurons (using dropout) and then...?

Working on MNIST dataset, I am running the following model:

from keras.models import Sequential
from keras.layers import Dense, Activation, Conv2D, Flatten, Dropout

model = Sequential()
model.add(Conv2D(128, kernel_size=(7, 7),
                 activation='relu',
                 input_shape=(28, 28, 1,)))
model.add(Dropout(0.20))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(Dropout(0.20))
model.add(Flatten())
model.add(Dense(units=64, activation='relu'))
model.add(Dropout(0.25))
model.add(Dense(units=10, activation='softmax'))
model.summary()
model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])
model.fit(train_data, train_labels,  batch_size=100, epochs=30, validation_data=(test_data, test_labels,))

How should I predict with this model so that I get its certainty about predictions too? I would appreciate some practical examples (preferably in Keras, but any will do).

To clarify, I am looking for an example of how to get certainty using the method outlined by Yurin Gal (or an explanation of why some other method yields better results).

4

4 回答 4

33

如果您想实施dropout方法来测量不确定性，您应该执行以下操作：

  1. 实现在测试期间也应用dropout的功能：

    import keras.backend as K
f = K.function([model.layers[0].input, K.learning_phase()],
               [model.layers[-1].output])

  2. 将此函数用作不确定性预测器，例如以下列方式：

    def predict_with_uncertainty(f, x, n_iter=10):
    result = numpy.zeros((n_iter,) + x.shape)

    for iter in range(n_iter):
        result[iter] = f(x, 1)

    prediction = result.mean(axis=0)
    uncertainty = result.var(axis=0)
    return prediction, uncertainty

当然，您可以使用任何不同的函数来计算不确定性。

于 2017-04-27T21:35:00.893 回答
6

对投票最多的答案进行了一些更改。现在它对我有用。

这是一种估计模型不确定性的方法。对于其他不确定性来源，我发现https://eng.uber.com/neural-networks-uncertainty-estimation/很有帮助。

f = K.function([model.layers[0].input, K.learning_phase()],
               [model.layers[-1].output])


def predict_with_uncertainty(f, x, n_iter=10):
    result = []

    for i in range(n_iter):
        result.append(f([x, 1]))

    result = np.array(result)

    prediction = result.mean(axis=0)
    uncertainty = result.var(axis=0)
    return prediction, uncertainty
于 2019-02-27T19:29:25.210 回答
3

一种更简单的方法是training=True在推理期间也设置要运行的任何 dropout 层（本质上是告诉层像始终处于训练模式一样运行 - 因此它始终存在于训练和推理中）。

import keras

inputs = keras.Input(shape=(10,))
x = keras.layers.Dense(3)(inputs)
outputs = keras.layers.Dropout(0.5)(x, training=True)

model = keras.Model(inputs, outputs)

上面的代码来自这个问题。

于 2019-05-17T03:11:45.507 回答
3

您的模型使用 softmax 激活，因此获得某种不确定性度量的最简单方法是查看输出 softmax 概率：

probs = model.predict(some input data)[0]

然后，该probs数组将是 [0, 1] 范围内的数字的 10 元素向量，总和为 1.0，因此可以将它们解释为概率。例如，数字 7 的概率为probs[7]。

然后有了这些信息，您可以进行一些后处理，通常预测的类别是概率最高的类别，但您也可以查看概率第二高的类别，等等。

于 2017-04-20T23:44:53.263 回答