0

我正在尝试将正则化添加到我使用 numpy 和 vanilla Python 创建的 Mnist 数字 NN 分类器中。我目前正在使用带有交叉熵成本函数的 Sigmoid 激活。

在不使用正则化器的情况下，我得到了 97% 的准确率。

然而，一旦我添加了正则化器，尽管使用不同的超参数，我只得到了大约 11%。我尝试了不同的学习率：

.001, .1, 1

以及不同的 lambda 值，例如：

.5、.8、1.0、2.0 等

我似乎无法弄清楚我犯了什么错误。我觉得我可能错过了一步？

我所做的唯一更改是权重的导数。我已经按如下方式实现了渐变：

def calculate_gradients(self,x, y, lambd):


        '''calculate all gradients with respect to
        cost. Here our cost function is cross_entropy

        last_layer_z_error = dC/dZ  (z is logit)
        All weight gradients also include regularization gradients

         x.shape[0]  = len of sample size

        '''



##### First we calculate the output layer gradients #########

        gradients, activations, zs = self.gather_backprop_data(x,y)

        #gradient of cost with respect to  Z of last layer
        last_layer_z_error = ((activations[-1] - y)) 



        #updating the weight_derivatives of final layer
        gradients['w'+ str(self.num_layers -1)] = np.dot(activations[-2].T,last_layer_z_error)/x.shape[0] + (lambd/x.shape[0])*(self.parameters['w'+ str(self.num_layers -1)])

        gradients['b'+ str(self.num_layers -1)] = np.mean(last_layer_z_error, axis =0)
        gradients['b'+ str(self.num_layers -1)] = np.expand_dims(gradients['b'+ str(self.num_layers -1)],0)


###HIDDEN LAYER GRADIENTS###

        z_previous_layer = last_layer_z_error



        for i in reversed(range(1,self.num_layers -1)):
            z_previous_layer =np.dot(z_previous_layer,self.parameters['w'+ str(i+1)].T, )*\
                                 (sigmoid_derivative(zs[i-1]))

            gradients['w'+str(i)] = np.dot((activations[i-1].T),z_previous_layer)/x.shape[0] + (lambd/x.shape[0])*(self.parameters['w'+str(i)])
            gradients['b'+str(i)] = np.mean(z_previous_layer, axis =0) 
            gradients['b'+str(i)] = np.expand_dims(gradients['b'+str(i)],0)


        return gradients

完整的代码可以在这里找到：

如果需要，我已将整个笔记本上传到 Github：

https://github.com/moondra2017/Neural-Networks-from-scratch/blob/master/Neural%20Network%20from%20scratch-Testing%20expanded%20Mnist-Sigmoid%20with%20cross-entroupy-with%20L2%20正则化。 ipynb

4

0 回答 0