Long Short-Term Memory
An LSTM is a recurrent network with gates that decide what to keep, forget and output, giving it much better memory over long sequences than a plain RNN.
Key takeaways
- Gates decide what an LSTM keeps, forgets, and outputs at each step.
- A separate cell state gives information a protected path across many steps.
- That path is what fixes the vanishing gradients that cripple plain RNNs.
- They handle long sequences far better than RNNs and still worse than transformers.
An LSTM is a recurrent network with gates that decide what to keep, what to discard, and what to pass on. The gates are what give it usable memory across long sequences.
A plain RNN overwrites its memory at every step. An LSTM maintains a separate cell state that information can travel along largely untouched, so something from step one can still matter at step five hundred.
Think of it like this. Think of a notebook alongside your working memory. Most of what happens gets forgotten, and you deliberately write down the few things worth keeping. The gates are the decision about what gets written.
How it works
- A cell state runs through the whole sequence, modified only where a gate allows it.
- The forget gate decides what to drop from that state.
- The input gate decides what new information to add.
- The output gate decides what to expose at this step.
- Gradients survive the journey, because the cell state offers a path that does not repeatedly shrink them.
Common mistakes
- "LSTMs solved long-range memory." They improved it enormously. Very long sequences still degrade.
- "LSTMs always beat GRUs." GRUs use two gates instead of three and often match them, while training faster.
- "They are obsolete." For long sequences in language, transformers won. For small models, streaming, and modest time series, LSTMs remain a sensible choice.