1

I've been working on a Q reinforcement learning implementation, where Q(π, a) is is approximated with a neural network. During trouble-shooting, I reduced the problem down to a very simple first step: train a NN to calculate atan2(y, x).

I'm using FANN for this problem, but the library is largely irrelevant as this question is more about the appropriate technique to use.

I have been struggling to teach the NN, given input = {x, y}, to calculate output = atan2(y, x).

Here is the naïve approach I have been using. It's extremely simplistic, but I'm trying to keep this simple to work up.

#include "fann.h"
#include <cstdio>
#include <random>
#include <cmath>

int main()
{
    // creates a 3 layered, densely connected neural network, 2-3-1
    fann *ann = fann_create_standard(3, 2, 3, 1);

    // set the activation functions for the layers
    fann_set_activation_function_hidden(ann, FANN_SIGMOID_SYMMETRIC);
    fann_set_activation_function_output(ann, FANN_SIGMOID_SYMMETRIC);

    fann_type input[2];
    fann_type expOut[1];
    fann_type *calcOut;

    std::default_random_engine rng;
    std::uniform_real_distribution<double> unif(0.0, 1.0);
    for (int i = 0; i < 100000000; ++i) {
        input[0] = unif(rng);
        input[1] = unif(rng);

        expOut[0] = atan2(input[1], input[0]);

        // does a single incremental training round
        fann_train(ann, input, expOut);
    }


    input[0] = unif(rng);
    input[1] = unif(rng);

    expOut[0] = atan2(input[1], input[0]);
    calcOut = fann_run(ann, input);

    printf("Testing atan2(%f, %f) = %f -> %f\n", input[1], input[0], expOut[0], calcOut[0]);

    fann_destroy(ann);
    return 0;
}

Super simple, right? However, even after 100,000,000 iterations this neural network fails:

Testing atan2(0.949040, 0.756997) = 0.897493 -> 0.987712

I also tried using a linear activation function on the output layer (FANN_LINEAR). No luck. In fact, the results are much worse. After 100,000,000 iterations, we get:

Testing atan2(0.949040, 0.756997) = 0.897493 -> 7.648625

Which is even worse than when the weights were randomly initialized. How could a NN get worse after training?

I found this issue with FANN_LINEAR to be consistent with other tests. When linear output is needed (e.g. in the calculation of the Q value, which corresponds to arbitrarily large or small rewards), this approach fails miserably and error actually appears to increase with training.

So what is going on? Is using a fully-connected 2-3-1 NN inappropriate for this situation? Is a symmetric sigmoid activation function in the hidden layer inappropriate? I fail to see what else could possibly account for this error.

4

1 回答 1

3

The problem you are facing is normal, and the quality of your predictor won't improve by augmenting the number of iterations, you should augment the size of your NN, either by adding some layers or by augemnting the size of the hidden layer. Instead of having 2-3-1 you can try 2-256-128-1 for example. Normally that will work better. If you want have a look on this simple code I wrote in python to do the same task, and it is wroking well 

import numpy as np
from numpy import arctan2

from keras.models import Sequential 
from keras.layers import Dense, InputLayer



nn_atan2 = Sequential()
nn_atan2.add(Dense(256, activation="sigmoid", input_shape=(2,)))
nn_atan2.add(Dense(128, activation="sigmoid"))
nn_atan2.add(Dense(1, activation='tanh'))

nn_atan2.compile(optimizer="adam", loss="mse")
nn_atan2.summary()

N = 100000
X = np.random.uniform(size=(N,2) )
y = arctan2(X[:,0], X[:,1])/(np.pi*0.5)

nn_atan2.fit(X,y, epochs=10, batch_size=128)

def predict(x, y):
    return float(nn_atan2.predict(np.array([[x, y]]))*(np.pi*0.5))

Runnin this code will give 

Epoch 1/10
100000/100000 [==============================] - 3s 26us/step - loss: 0.0289
Epoch 2/10
100000/100000 [==============================] - 2s 24us/step - loss: 0.0104
Epoch 3/10
100000/100000 [==============================] - 2s 24us/step - loss: 0.0102
Epoch 4/10
100000/100000 [==============================] - 2s 24us/step - loss: 0.0096
Epoch 5/10
100000/100000 [==============================] - 2s 24us/step - loss: 0.0082
Epoch 6/10
100000/100000 [==============================] - 2s 23us/step - loss: 0.0051
Epoch 7/10
100000/100000 [==============================] - 2s 23us/step - loss: 0.0027
Epoch 8/10
100000/100000 [==============================] - 2s 23us/step - loss: 0.0019
Epoch 9/10
100000/100000 [==============================] - 2s 23us/step - loss: 0.0014
Epoch 10/10
100000/100000 [==============================] - 2s 23us/step - loss: 0.0010
于 2020-01-14T08:10:18.143 回答