Recurrent Neural Network
- tags
- Neural Network
See other extensions: LSTM, GRU, (Chandar et al. 2019), (Goudreau et al. 1994), (Sutskever, Martens, and Hinton 2011), (Cho et al. 2014)
Getting Started
Some basic RNN resources. Here is some stuff to get you started: Really early RNN Work:
- http://srsc-mac1.ulb.ac.be/axcWWW/papers/pdf/91-NC.pdf
- (bptt) http://axon.cs.byu.edu/Dan/678/papers/Recurrent/Werbos.pdf
- (rtrl) http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.52.9724&rep=rep1&type=pdf
LSTMs:
- http://www7.informatik.tu-muenchen.de/~hochreit
- https://www.researchgate.net/profile/Felix_Gers/publication/12292425_Learning_to_Forget_Continual_Prediction_with_LSTM/links/5759414608ae9a9c954e84c5/Learning-to-Forget-Continual-Prediction-with-LSTM.pdf
- (In RL) http://papers.nips.cc/paper/1953-reinforcement-learning-with-long-short-term-memory.pdf
Some other useful things:
- (review on more modern learning techniques) https://ieeexplore.ieee.org/abstract/document/6639349
- (On the difficulty of training) http://proceedings.mlr.press/v28/pascanu13.pdf
- (overview on some basic RNN algorithms) https://pdfs.semanticscholar.org/cccd/3fd7a45e7643f26391bd539ffbede0690f36.pdf
- (Colah’s blog) https://colah.github.io/posts/2015-08-Understanding-LSTMs/
Some ideas that I believe could use some more research:
- (Echo State Netwoks) http://www.scholarpedia.org/article/Echo_state_network
- (Bi Directional LSTMs/RNNs) https://ieeexplore.ieee.org/document/4288069
Chandar, Sarath, Chinnadhurai Sankar, Eugene Vorontsov, Samira Ebrahimi Kahou, and Yoshua Bengio. 2019. “Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies.” In Proceedings of the AAAI Conference on Artificial Intelligence, 33:3280–87.
Cho, Kyunghyun, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. “On the Properties of Neural Machine Translation: Encoder–Decoder Approaches.” In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 103–11. Association for Computational Linguistics. doi:10.3115/v1/W14-4012.
Goudreau, M.W., C.L. Giles, S.T. Chakradhar, and D. Chen. 1994. “First-Order versus Second-Order Single-Layer Recurrent Neural Networks.” IEEE Transactions on Neural Networks 5 (3): 511–13. doi:10.1109/72.286928.
Links to this note:
- wu2016multiplicative: On Multiplicative Integration with Recurrent Neural Networks
- sutskever2011generating: Generating text with recurrent neural networks
- LSTM
- goudreau1994firstorder: First-order versus second-order single-layer recurrent neural networks
- General Value Functions
- chung2014empirical: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- chandar2019nonsaturating: Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
- Backpropagation Through Time