Gated Recurrent Unit

A gated recurrent unit is a simplified LSTM. It uses two gates instead of three and drops the separate cell state, which makes it smaller and faster to train.

The interesting result is how little that simplification costs. On most tasks a GRU matches an LSTM, and on smaller datasets the reduced parameter count sometimes helps.

Think of it like this. Think of a camera with two dials instead of five. You lose some fine control and you take the shot faster, and in most conditions the photograph is just as good.

How it works

• The update gate decides how much of the previous state to keep versus replace. It does the job of the LSTM's forget and input gates together.
• The reset gate decides how much past state to use when computing the new candidate.
• No separate cell state. The hidden state carries everything, unlike the LSTM's split design.
• Fewer parameters, which means faster training and less memory.

Trade-offs

• Faster to train, with roughly 25 percent fewer parameters than an equivalent LSTM.
• Slightly less expressive. On very long sequences the LSTM's extra gate occasionally wins.
• Often better on small data, because fewer parameters means less to overfit.

Common mistakes

• "Simpler means worse." GRUs frequently match LSTMs. Simplicity here was a genuine finding, not a compromise.
• "Always pick the GRU." On long sequences with plenty of data, the LSTM sometimes has the edge. It is worth testing both.
• "A GRU is an LSTM." They are different designs. The GRU merges gates and drops the separate cell state.