Recurrent Neural Network
- tags
- Neural Network
See other extensions: LSTM, GRU, (Chandar et al. 2019), (Goudreau et al. 1994), (Sutskever, Martens, and Hinton 2011), (Cho et al. 2014)
Getting Started
Some basic RNN resources. Here is some stuff to get you started: Really early RNN Work:
- http://srsc-mac1.ulb.ac.be/axcWWW/papers/pdf/91-NC.pdf
- (bptt) http://axon.cs.byu.edu/Dan/678/papers/Recurrent/Werbos.pdf
- (rtrl) http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.52.9724&rep=rep1&type=pdf
LSTMs:
- http://www7.informatik.tu-muenchen.de/~hochreit
- https://www.researchgate.net/profile/Felix_Gers/publication/12292425_Learning_to_Forget_Continual_Prediction_with_LSTM/links/5759414608ae9a9c954e84c5/Learning-to-Forget-Continual-Prediction-with-LSTM.pdf
- (In RL) http://papers.nips.cc/paper/1953-reinforcement-learning-with-long-short-term-memory.pdf
Some other useful things:
- (review on more modern learning techniques) https://ieeexplore.ieee.org/abstract/document/6639349
- (On the difficulty of training) http://proceedings.mlr.press/v28/pascanu13.pdf
- (overview on some basic RNN algorithms) https://pdfs.semanticscholar.org/cccd/3fd7a45e7643f26391bd539ffbede0690f36.pdf
- (Colah’s blog) https://colah.github.io/posts/2015-08-Understanding-LSTMs/
Some ideas that I believe could use some more research:
- (Echo State Netwoks) http://www.scholarpedia.org/article/Echo_state_network
- (Bi Directional LSTMs/RNNs) https://ieeexplore.ieee.org/document/4288069
Chandar, Sarath, Chinnadhurai Sankar, Eugene Vorontsov, Samira Ebrahimi Kahou, and Yoshua Bengio. 2019. “Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies.” In AAAI. https://arxiv.org/abs/1902.06704.
Cho, Kyunghyun, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. “On the Properties of Neural Machine Translation: EncoderDecoder Approaches.” In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. Doha, Qatar: Association for Computational Linguistics.
Links to this note:
- LSTM
- Backpropagation Through Time
- chandar2019nonsaturating: Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
- General Value Functions
- wu2016multiplicative: On Multiplicative Integration with Recurrent Neural Networks
- sutskever2011generating: Generating text with recurrent neural networks
- goudreau1994firstorder: First-order versus second-order single-layer recurrent neural networks
- chung2014empirical: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling