RNN - framework

Just a framework about RNN not a deep understanding of the theory

And maybe include the coding by Kears

RNN - recurrent neural networks

• Concept

Recurrent means the output at the current time step becomes the input to the next time step. At each element of the sequence, the model considers not just the current input, but what it remembers about the preceding elements.

A RNN is designed to mimic the human way of processing sequences: we consider the entire sentence when forming a response instead of words by themselves. 

 Recall, the benefit of a Recurrent Neural Network for sequence learning is it maintains a memory of the entire sequence preventing prior information from being lost.

Example

Convert abstracts from list of strings into list of lists of integers (sequences)

Tokenizer : map word and indexes

• Remove punctuation and split strings into lists of individual words
• Convert the individual words into integer

Create feature and labels from sequences

1. Build LSTM model with Embedding, LSTM, and Dense layers
2. Load in pre-trained embeddings
3. Train model to predict next work in sequence
4. Make predictions by passing in starting sequence

hidden state/layer

ground truth

loss function

back propagation

Solution: 

LSTM

1. forget gate: discarding irrelevant information 
2. input gate: handling the current input
3. output gate: producing predictions at each time step

BiLSTM

GRU