python - Fixing a subset of weights in Neural network during training

Question

I am considering creating a customized neural network. The basic structure is the same as usual, but I want to truncate the connections between layers. For example, if I construct a network with two hidden layers, I would like to delete some weights and keep the others, like so:

This is not conventional dropout (to avoid overfitting), since the remaining weights (connections) should be specified and fixed.

Are there any ways in python to do it? Tensorflow, pytorch, theano or any other modules?

Answer 1

是的，你可以在张量流中做到这一点。

你会在你的 tensorflow 代码中有一些层，如下所示：

m = tf.Variable( [width,height] , dtype=tf.float32  ))
b = tf.Variable( [height] , dtype=tf.float32  ))
h = tf.sigmoid( tf.matmul( x,m ) + b )

你想要的是一些新的矩阵，我们称它为k表示 kill。它会杀死特定的神经连接。神经连接在m中定义。这将是您的新配置

k = tf.Constant( kill_matrix , dtype=tf.float32 )
m = tf.Variable( [width,height] , dtype=tf.float32  )
b = tf.Variable( [height] , dtype=tf.float32  )
h = tf.sigmoid( tf.matmul( x, tf.multiply(m,k) ) + b )

您的 kill_matrix 是 1 和 0 的矩阵。为每个要保留的神经连接插入 1，为每个要杀死的神经连接插入 0。