Modules

torch-rnn provides high-peformance, reusable RNN and LSTM modules. These modules have no dependencies other than torch and nn and each lives in a single file, so they can easily be incorporated into other projects.

We also provide a LanguageModel module used for character-level language modeling; this is less reusable, but demonstrates that LSTM and RNN modules can be mixed with existing torch modules.

VanillaRNN

rnn = nn.VanillaRNN(D, H)

VanillaRNN is a torch nn.Module subclass implementing a vanilla recurrent neural network with a hyperbolic tangent nonlinearity. It transforms a sequence of input vectors of dimension D into a sequence of hidden state vectors of dimension H. It operates over sequences of length T and minibatches of size N; the sequence length and minibatch size can change on each forward pass.

Ignoring minibatches for the moment, a vanilla RNN computes the next hidden state vector h[t] (of shape (H,)) from the previous hidden state h[t - 1] and the current input vector x[t] (of shape (D,)) using the recurrence relation

h[t] = tanh(Wh h[t- 1] + Wx x[t] + b)

where Wx is a matrix of input-to-hidden connections, Wh is a matrix of hidden-to-hidden connections, and b is a bias term. The weights Wx and Wh are stored in a single Tensor rnn.weight of shape (D + H, H) and the bias b is stored in a Tensor rnn.bias of shape (H,).

You can use a VanillaRNN instance in two different ways:

h = rnn:forward({h0, x})
grad_h0, grad_x = unpack(rnn:backward({h0, x}, grad_h))
h = rnn:forward(x)
grad_x = rnn:backward(x, grad_h)

h0 is the initial hidden states, of shape (N, H) and x is the sequence of input vectors, of shape (N, T, D). The output h is the sequence of hidden states at each timestep, of shape (N, T, H). In some applications, such as image captioning, it is possible that the initial hidden state will be computed as the output of some other network.

By default, if h0 is not provided on the forward pass then the initial hidden state will be set to zero. This behavior might be useful for applications like sentiment analysis, where you want an RNN to process many independent sequences.

If h0 is not provided and the instance variable rnn.remember_states is set to true, then the first call to rnn:forward will set the initial hidden state to zero; on subsequent calls to forward, the final hidden state from the previous call will be used as the initial hidden state. This behavior is commonly used in language modeling, where we want to train with very long (potentialy infinite) sequences, and compute gradients using truncated back-propagation through time. You cause the model to forget its hidden states by calling rnn:resetStates(); then the next call to rnn:forward will cause h0 to be initialized to zeros.

These behaviors are all exercised in the unit test for VanillaRNN.lua.

As an implementation note, we implement :backward directly to compute both gradients with respect to inputs and accumulate gradients with respect to weights since these two operations share a lot of computation. We override :updateGradInput and :accGradparameters to call into :backward, so to avoid computing the same thing twice you should call :backward directly rather than calling :updateGradInput and then :accGradParameters.

The file VanillaRNN.lua is standalone, with no dependencies other than torch and nn.

LSTM

lstm = nn.LSTM(D, H)

An LSTM (short for Long Short-Term Memory) is a fancy type of recurrent neural network that is much more commonly used than vanilla RNNs. Similar to the VanillaRNN above, LSTM is a torch nn.Module subclass implementing an LSTM. It transforms a sequence of input vectors of dimension D into a sequence of hidden state vectors of dimension H; it operates over sequences of length T and minibatches of size N, which can be different on each forward pass.

An LSTM differs from a vanilla RNN in that it keeps track of both a hidden state and a cell state at each timestep. Ignoring minibatches, the next hidden state vector h[t] (of shape (H,)) and cell state vector c[t] (also of shape (H,)) are computed from the previous hidden state h[t - 1], previous cell state c[t - 1], and current input x[t] (of shape (D,)) using the following recurrence relation:

ai[t] = Wxi x[t] + Whi h[t - 1] + bi  # Matrix / vector multiplication
af[t] = Wxf x[t] + Whf h[t - 1] + bf  # Matrix / vector multiplication
ao[t] = Wxo x[t] + Who h[t - 1] + bo  # Matrix / vector multiplication
ag[t] = Wxg x[t] + Whg h[t - 1] + bg  # Matrix / vector multiplication
i[t] = sigmoid(ai[t])  # Input gate
f[t] = sigmoid(af[t])  # Forget gate
o[t] = sigmoid(ao[t])  # Output gate
g[t] = tanh(ag[t])     # Proposed update
c[t] = f[t] * c[t - 1] + i[t] * g[t]  # Elementwise multiplication of vectors
h[t] = o[t] * tanh(c[t])              # Elementwise multiplication of vectors

The input-to-hidden matrices Wxi, Wxf, Wxo, and Wxg along with the hidden-to-hidden matrices Whi, Whf, Who, and Whg are stored in a single Tensor lstm.weight of shape (D + H, 4 * H). The bias vectors bi, bf, bo, and bg are stored in a single tensor lstm.bias of shape (4 * H,).

You can use an LSTM instance in three different ways:

h = lstm:forward({c0, h0, x})
grad_c0, grad_h0, grad_x = unpack(lstm:backward({c0, h0, x}, grad_h))
h = lstm:forward({h0, x})
grad_h0, grad_x = unpack(lstm:backward({h0, x}, grad_h))
h = lstm:forward(x)
grad_x = lstm:backward(x, grad_h)

In all cases, c0 is the initial cell state of shape (N, H), h0 is the initial hidden state of shape (N, H), x is the sequence of input vectors of shape (N, T, D), and h is the sequence of output hidden states of shape (N, T, H).

If the initial cell state or initial hidden state are not provided, then by default they will be set to zero.

If the initial cell state or initial hidden state are not provided and the instance variable lstm.remember_states is set to true, then the first call to lstm:forward will set the initial hidden and cell states to zero, and subsequent calls to lstm:forward set the initial hidden and cell states equal to the final hidden and cell states from the previous call, similar to the VanillaRNN. You can reset these initial cell and hidden states by calling lstm:resetStates(); then the next call to lstm:forward will set the initial hidden and cell states to zero.

These behaviors are exercised in the unit test for LSTM.lua.

As an implementation note, we implement :backward directly to compute both gradients with respect to inputs and accumulate gradients with respect to weights since these two operations share a lot of computation. We override :updateGradInput and :accGradparameters to call into :backward, so to avoid computing the same thing twice you should call :backward directly rather than calling :updateGradInput and then :accGradParameters.

The file LSTM.lua is standalone, with no dependencies other than torch and nn.

LanguageModel

model = nn.LanguageModel(kwargs)

LanguageModel uses the above modules to implement a multilayer recurrent neural network language model with dropout regularization. Since LSTM and VanillaRNN are nn.Module subclasses, we can implement a multilayer recurrent neural network by simply stacking multiple instance in an nn.Sequential container.

kwargs is a table with the following keys: - idx_to_token: A table giving the vocabulary for the language model, mapping integer ids to string tokens. - model_type: “lstm” or “rnn” - wordvec_size: Dimension for word vector embeddings - rnn_size: Hidden state size for RNNs - num_layers: Number of RNN layers to use - dropout: Number between 0 and 1 giving dropout strength after each RNN layer