hi guys, my first post here, excited
i`ve been trying to learn some of the BP algorithem
and wrote the simplest code just to be sure i understand the basics, but somehow the net output is always about the same
and i cant figure why ?
if anyone can take a look i would be very gratefull . (should be easy for someone whose familiar with bp).

as input I entered two numbers between 0-1 wich are a and b
the output should be the subtraction between them ,output= (a - b) , might be negative..
i used simple sigmoid function .
was written in VB , i pasted just the hart of the code , its just for understanding , procedural way .
some Technical stuff , i scales the output to the range of -1 to 1 , should the output neuron recive an un scaled value of the error? like i did here
or how should it be?

so can someone explain What is wrong ?

thank you..




e = 2.718281828
alpha = 0.1

'input = two numbers between 0-1


a = Cells(2, 1)
b = Cells(2, 2)



truth = a - b 'correct answer


Sum1 = a + b
Sum2 = a + b



f1 = 1 / (1 + e ^ (-Sum1))
f2 = 1 / (1 + e ^ (-Sum2))

Sum3 = f1 + W13 + f2 * W23
Sum4 = f1 * W14 + f2 * W24

f3 = 1 / (1 + e ^ (-Sum3))
f4 = 1 / (1 + e ^ (-Sum4))

Sum5 = f3 * W35 + f4 * W45
f5 = 1 / (1 + e ^ (-Sum5))



answer = -1 + f5 * 2 ' need to spred over the area : -1 till 1



'backPropagate
err5 = (truth - answer + 1) / 2
err3 = err5 * W35
err4 = err5 * W45
err1 = err3 * W13 + err4 * W14
err2 = err3 * W23 + err4 * W24

'update wieghts


W13 = W13 + alpha * (f3 * (1 - f3)) * (f1 * W13) * err3
W23 = W23 + alpha * (f3 * (1 - f3)) * (f2 * W23) * err3

W14 = W14 + alpha * (f4 * (1 - f4)) * (f1 * W14) * err4
W24 = W24 + alpha * (f4 * (1 - f4)) * (f2 * W24) * err4


W35 = W35 + alpha * (f5 * (1 - f5)) * (f3 * W35) * err5
W45 = W45 + alpha * (f5 * (1 - f5)) * (f4 * W45) * err5

p.s the net has to two input numbers ,
two neourons that recives them ( neourons 1, 2)
two neourons as hidden layer (neouron 3,4)
output neouron 5.

Hello

I haven't look in the details your code, but you must let the output in the range ]0,1[ since you use sigmoid function,
the delta rule is then (with sigmoid function) : delta=alpha*(1-output)*output*(error)*input

If you're in a hidden layer, then 'error' is the sum of the errors of the next layer (else the difference between wanted and effectively get output)

Hope that helps

EDIT : I corrected the delta rule

hi adaline .
thanx but i belive thats not it . first : how do i create an output of boolean values ? wide range values etc ? i output neouron has a 0-1 sigmoid function but it`s translated into -1 till 1 terms .
and even if it must be just 0-1 without scalling , i tried it in deffrent ways , and it didnt improve ..
about the delta , i belive its [color=#1C2837][size=2]alpha*(1-output)*output*(error)*inputi because you mentioned the formula without the input and i saw this while Xij is a certain input .
thus i still dont know what the problem is , and i dont really know if this net structure is sepose to be enough for this task or may be its imposible to solve this way ?
i wil be glad if someone hhas simple written code of BP wich i can learn from as an example ..


[color=#1C2837][size=2][sub] [/sub]

Excuse me you're absolutely right, delta=output*(1-output)*error*input (sorry I made a mistake : I forgot the input)

But you should keep the output in the range ] 0,1 [ so that you can apply it to the delta rule
Afterwards only, you can transform the output like you want


First of all, compute the errors of the output layer
Then retropropagate them (the hidden layers 'receive' the weighted sum of the errors commited by the next layer)


You're training a net so that it computes difference :
You want your net to learn to compute difference between a and b:

1) Compute a,b
2) Present a and b to the input layer
3) Compute output
4) The error committed by the net is : (a-b)-output
5) Compute output layer error
6) Propagate it to previous layers
7) adapt weights for all layers

You use sigmoid function so you have to encod your values accordingly to it
For example output<0.5 means negative ; positive else
The sigmoid function will not give you anyway the difference between a and b, but you can learn your net to detect if the result is positive or negative

Is it helping ?

Edit : I see than alpha =0.1 try just alpha=0.9 (in this case the minimum is absolute, but putting a small alpha value increases the risk to get stuck in local minimum)

I transformed the output into those trerms, if theres closest values to 1 means big positive diffrence , the closest to 0 means negative gape and 0.5 is close to equals . The thing is that the net answers about the same answer for all cases , so i guess its beacuse of the expected values of randonm distribution .. Just a guess .
Can it be that this kind of problem is unsolveable with ANN or the net structure doesnt fit the problem ?
How can i know if iys solveable or not and what structure to choose ?
Do you know a problem wich is solvable for sure and the net structure that i shold apply?
Do you know any other problem wich is easier to solve

The fact is that it doesnt learn .
Can it br that this is a type of problam that cant be solved with ANN? Or that this net structure doesnt fit

Sorry the end is written twice , stupied iphone ..

Can it be that this kind of problem is unsolveable with ANN or the net structure doesnt fit the problem ?

Your net can actually learn to detect if the [s]sum[/s] difference is negative or positive.
Maybe you can try step by step, try first with a single neuron with 2 inputs : it must works even with a single unit
Could you post your complete code, I'll have a look in the details

If the output values are always the same, its probably being Run instead of Trained.
Verify that the initial weights are properly random, and that the output of the first iteration in Training mode shows that randomness.
If your network weights are converging on a consistent result in a single iteration, then something is wrong with your transfer function (nothing to do with the Sigmoid).

Hmm .. Its combaind with the excel datasheet, nothing special to add except refering excel cells , but ill post it anyway when im on my comp .i belive the ann shold answer more than just positive / negatuve ,for It is only a binary question ..