神经网络中反向传播算法的理解问题

Question

我理解反向传播算法有困难。我读了很多书，搜索了很多东西，但我不明白为什么我的神经网络不能工作。我想确认我做的每一部分都是正确的。

下面是我的神经网络，当它初始化时，当第一行输入[1,1]和输出[0]被设置时（如你所见，我正在尝试做异或神经网络）：

我有3层：输入，隐藏和输出。第一层（输入层）和隐藏层包含2个神经元，每个神经元有2个突触。最后一层（输出）包含一个有2个突触的神经元。

一个突触包含一个权重和它之前的delta（在开始时，它是0）。连接到突触的输出可以通过与突触相关联的sourceNeuron找到，如果没有sourceNeuron，则可以在输入阵列中找到（如在输入层中）。

java类包含一个神经元列表。在neuralnetwork.java中，我初始化神经网络，然后在训练集中循环。在每次迭代中，我替换输入和输出值，并调用我的反向传播算法上的训练，算法运行一定的时间（现在是1000次的历元）为当前集。

我使用的激活函数是乙状结肠。

训练集和验证集为(input1,input2,output):

1,1,0
0,1,1
1,0,1
0,0,0

下面是我的neuron.java实现：

public class Neuron {

    private IActivation activation;
    private ArrayList<Synapse> synapses; // Inputs
    private double output; // Output
    private double errorToPropagate;

    public Neuron(IActivation activation) {
        this.activation = activation;
        this.synapses = new ArrayList<Synapse>();
        this.output = 0;
        this.errorToPropagate = 0;
    }

    public void updateOutput(double[] inputs) {
        double sumWeights = this.calculateSumWeights(inputs);

        this.output = this.activation.activate(sumWeights);
    }

    public double calculateSumWeights(double[] inputs) {
        double sumWeights = 0;

        int index = 0;
        for (Synapse synapse : this.getSynapses()) {
            if (inputs != null) {
                sumWeights += synapse.getWeight() * inputs[index];
            } else {
                sumWeights += synapse.getWeight() * synapse.getSourceNeuron().getOutput();
            }

            index++;
        }

        return sumWeights;
    }

    public double getDerivative() {
        return this.activation.derivative(this.output);
    }

    [...]
}

synapse.java包含：

public Synapse(Neuron sourceNeuron) {
    this.sourceNeuron = sourceNeuron;
    Random r = new Random();
    this.weight = (-0.5) + (0.5 - (-0.5)) * r.nextDouble();
    this.delta = 0;
}

[... getter and setter ...]

this.forwardPropagation(neuralNetwork, inputs);

this.backwardPropagation(neuralNetwork, expectedOutput);

this.updateWeights(neuralNetwork);

public void forwardPropagation(NeuralNetwork neuralNetwork, double[] inputs) {

    for (Layer layer : neuralNetwork.getLayers()) {

        for (Neuron neuron : layer.getNeurons()) {
            if (layer.isInput()) {
                neuron.updateOutput(inputs);
            } else {
                neuron.updateOutput(null);
            }
        }
    }
}

public void backwardPropagation(NeuralNetwork neuralNetwork, double realOutput) {

    Layer lastLayer = null;

    // Loop à travers les hidden layers et le output layer uniquement
    ArrayList<Layer> layers = neuralNetwork.getLayers();
    for (int i = layers.size() - 1; i > 0; i--) {
        Layer layer = layers.get(i);

        for (Neuron neuron : layer.getNeurons()) {

            double errorToPropagate = neuron.getDerivative();

            // Output layer
            if (layer.isOutput()) {

                errorToPropagate *= (realOutput - neuron.getOutput());
            }
            // Hidden layers
            else {
                double sumFromLastLayer = 0;

                for (Neuron lastLayerNeuron : lastLayer.getNeurons()) {
                    for (Synapse synapse : lastLayerNeuron.getSynapses()) {
                        if (synapse.getSourceNeuron() == neuron) {
                            sumFromLastLayer += (synapse.getWeight() * lastLayerNeuron.getErrorToPropagate());

                            break;
                        }
                    }
                }

                errorToPropagate *= sumFromLastLayer;
            }

            neuron.setErrorToPropagate(errorToPropagate);
        }

        lastLayer = layer;
    }
}

public void updateWeights(NeuralNetwork neuralNetwork) {

    for (int i = neuralNetwork.getLayers().size() - 1; i > 0; i--) {

        Layer layer = neuralNetwork.getLayers().get(i);

        for (Neuron neuron : layer.getNeurons()) {

            for (Synapse synapse : neuron.getSynapses()) {

                double delta = this.learningRate * neuron.getError() * synapse.getSourceNeuron().getOutput();

                synapse.setWeight(synapse.getWeight() + delta + this.momentum * synapse.getDelta());

                synapse.setDelta(delta);
            }
        }
    }
}

this.forwardPropagation(neuralNetwork, inputs);

以下是我在1000纪元之后的结果：

Real: 0.0
Current: 0.025012156926937503
Real: 1.0
Current: 0.022566830709341495
Real: 1.0
Current: 0.02768416343491415
Real: 0.0
Current: 0.024903432706154027

为什么输入层的突触没有更新？在任何地方，它被写入只更新隐藏层和输出层。

正如你所看到的，这是完全错误的！它不会转到1.0只转到第一个训练集输出（0.0）。

=== Input Layer

== Neuron #1

= Synapse #1
Weight: -0.19283583155573614
Input: 1.0

= Synapse #2
Weight: 0.04023817185601586
Input: 1.0

Sum: -0.15259765969972028
Output: 0.461924442180935

== Neuron #2

= Synapse #1
Weight: -0.3281099260608612
Input: 1.0

= Synapse #2
Weight: -0.4388250065958519
Input: 1.0

Sum: -0.7669349326567131
Output: 0.31714251453174147

=== Hidden Layer

== Neuron #1

= Synapse #1
Weight: 0.16703288052854093
Input: 0.461924442180935

= Synapse #2
Weight: 0.31683996162148054
Input: 0.31714251453174147

Sum: 0.17763999229679783
Output: 0.5442935820534444

== Neuron #2

= Synapse #1
Weight: -0.45330313978424686
Input: 0.461924442180935

= Synapse #2
Weight: 0.3287014377113835
Input: 0.31714251453174147

Sum: -0.10514659949771789
Output: 0.47373754172497556

=== Output Layer

== Neuron #1

= Synapse #1
Weight: 0.08643751629154495
Input: 0.5442935820534444

= Synapse #2
Weight: -0.29715579267218695
Input: 0.47373754172497556

Sum: -0.09372646936373039
Output: 0.47658552081912403

我可能有偏见问题。我将在这个答案的帮助下研究它：偏见在神经网络中的作用。它不会在下一个数据集中移回所以...

Answer 1

我终于发现了问题所在。对于异或，我不需要任何偏差，它收敛到期望值。当你对最终输出进行四舍五入时，我得到了精确的输出。所需要的是训练，然后验证，然后再训练，直到神经网络令人满意。我一直在训练每一套直到满意，但不是一遍又一遍地整套。

// Initialize the Neural Network
algorithm.initialize(this.numberOfInputs);

int index = 0;
double errorRate = 0;

// Loop until satisfaction or after some iterations
do {
    // Train the Neural Network
    algorithm.train(this.trainingDataSets, this.numberOfInputs);

    // Validate the Neural Network and return the error rate
    errorRate = algorithm.run(this.validationDataSets, this.numberOfInputs);

    index++;
} while (errorRate > minErrorRate && index < numberOfTrainValidateIteration);

对于真实的数据，我需要一个偏差，因为输出开始发散。下面是我如何添加偏倚的：

在neuron.java类中，我添加了一个带有权重和输出为1.0的偏置突触。我把它和所有其他的突触相加，然后把它放入我的激活函数中。

public class Neuron implements Serializable {

    [...]

    private Synapse bias;

    public Neuron(IActivation activation) {
        [...]
        this.bias = new Synapse(this);
        this.bias.setWeight(0.5); // Set initial weight OR keep the random number already set
    }

    public void updateOutput(double[] inputs) {
        double sumWeights = this.calculateSumWeights(inputs);

        this.output = this.activation.activate(sumWeights + this.bias.getWeight() * 1.0);
    }

    [...]

public class BackPropagationStrategy implements IStrategy, Serializable {

    [...]

    public void updateWeightsAndBias(NeuralNetwork neuralNetwork, double[] inputs) {

        for (int i = neuralNetwork.getLayers().size() - 1; i >= 0; i--) {

            Layer layer = neuralNetwork.getLayers().get(i);

            for (Neuron neuron : layer.getNeurons()) {

                [...]

                Synapse bias = neuron.getBias();
                double delta = learning * 1.0;
                bias.setWeight(bias.getWeight() + delta + this.momentum * bias.getDelta());

                bias.setDelta(delta);
            }
        }
    }

    [...]