Original Post
Hi! Maybe someone here can help me. I'm programming a long short-term memory (LSTM) neural network for my masters thesis. On paper LSTM sounds good, but from what I've seen I'm getting doubful. I'm having problems replicating the results in the original LSTM research papers. I've implemented an addition test, a periodic function generation test and a gliding addition test, as described in the papers. In the addition test each sequence fed to the network consists of random numbers and the network has to remember two marked numbers and return their sum at the end of a sequence. Gliding addition test is the same thing, but there are no sequences. Instead there is a continuous stream. The network has to keep on adding numbers and learn to periodically reset itself with a forget gate. I haven't yet quite grasped how the equations are derived from RTRL, but I think I understand them well enough to apply them. I found two bugs in the original lstm code and fixed them. When running my network without peephole connections and forget gates it produces exactly same results as the original code. The results I'm getting are qualitatively not as good as the ones described in the research papers. I have the problem even when using the original code. I use old original code without peepholes and forget gates. I haven't yet been able to get the network to learn the periodic function generation task. I've tried with all possible combinations of parameters I could think of. My questions are: is this normal behavior? Is LSTM really this hard to train or is it more likely there is another bug in the original code and my code? Or am I just doing something wrong in the training? I haven't decided what to use the network on. A large part of my thesis will be applying the network on some problem. I would appreciate any tips what to apply the network on. It should be something that is not too hard to implement and get working. Our department has a nice robot with all kinds of sensors and a controllable arm. The only idea I have so far is to try to control the robot or its arm, but it might be too difficult. Sorry about the long rant :) [Edited by - oaks on March 19, 2009 1:21:16 PM]