In an RNN, every word depends on the previous hidden state. This looks natural for sequences, but it creates a bottleneck: the model has to compress everything it has seen so far into one small state vector.
The problem becomes serious when the important word is far away from the current word. For example:
To understand was excellent, the model should connect it back to report. But after many intermediate words, the original signal may fade.
- Vanishing gradient: during training, early tokens receive very weak learning signals.
- Sequential processing: tokens are processed one after another, so training is slow.
- Fixed memory bottleneck: the hidden state has limited capacity to remember long context.
- Long-range dependency issue: connecting distant words becomes difficult.
Basic RNNs had difficulty remembering useful information over long sequences. LSTM and GRU were introduced to solve this by adding a controlled memory mechanism.
LSTM stands for Long Short-Term Memory. It has a memory cell and three main gates:
- Forget gate: decides what old information to remove.
- Input gate: decides what new information to store.
- Output gate: decides what information to expose as output.
Example: In a long paragraph, if the subject is report, the LSTM can keep that information in memory until it is needed later.
GRU stands for Gated Recurrent Unit. It is a simpler version of LSTM with fewer gates:
- Update gate: controls how much past information to keep.
- Reset gate: controls how much past information to forget.
GRUs are usually faster and lighter than LSTMs, while still handling long dependencies better than vanilla RNNs.
Before encoder-decoder models, it was difficult to convert one sequence into another sequence when their lengths were different. For example, in translation, the English sentence and Hindi sentence may not have the same number of words or the same order.
The encoder reads the input and converts it into a meaningful representation. The decoder then uses that representation to generate the output step by step.
- Machine translation
- Text summarization
- Question answering
- Chatbot response generation
- Speech-to-text and text-to-speech style tasks
However, early encoder-decoder models still had one major limitation: the whole input was compressed into one fixed-size vector. For long sentences, this caused information loss. Attention was introduced to solve that limitation.
In early encoder-decoder models, the encoder compressed the full input sentence into a single context vector. This worked for short sentences, but for long sentences, important information could be lost.
Attention changed this. Instead of using only one fixed representation, the decoder can look back at all input tokens and assign importance scores to them.
Example: while translating Amit is preparing for an AI interview, when generating the word related to Amit, the model focuses on Amit. When generating the word related to preparing, it focuses on preparing.
- It reduced information loss in long sequences.
- It improved translation and summarization quality.
- It helped the model capture long-range dependencies.
- It made model behavior more interpretable through attention weights.
- It later became the foundation of Transformers.
The word bank can mean a financial bank or a river bank. Its meaning depends on surrounding words. Self-attention helps the model understand this by allowing each token to look at other tokens.
So the same word can get a different representation depending on context.
For each token, the model asks:
- Which other tokens are important for understanding me?
- How strongly should I connect to them?
- How should my meaning change based on them?
Another example:
This helps with pronoun resolution, word sense disambiguation, grammar relationships, and long-distance dependencies.
A single attention head may focus on one kind of relationship. But language has many relationships at the same time: grammar, meaning, entity reference, position, topic, and long-range dependency.
Multi-head attention solves this by using several attention heads together.
- One head may connect
ittoanimal. - One head may understand the cause-effect relation with
because. - One head may focus on grammatical structure.
- One head may focus on the overall sentence meaning.
The outputs of all heads are combined, giving the model a richer understanding than a single attention head.
RNNs process text sequentially. This means token 10 cannot be processed before token 9. That makes training slow and difficult to parallelize.
Transformers use self-attention, so every token can directly connect with every other token. This makes it easier to learn long-distance relationships.
- Parallel processing: much faster training on GPUs/TPUs.
- Better long-context handling: tokens can directly attend to distant tokens.
- Scalability: performance improves strongly with more data and parameters.
- Transfer learning: pretrained Transformer models can be fine-tuned for many tasks.
- Flexible architecture: encoder-only, decoder-only, and encoder-decoder variants.
This is why models like BERT, GPT, T5, and LLaMA are Transformer-based rather than RNN-based.
Transformers process all tokens in parallel. This is good for speed, but it creates one issue: the model does not automatically know which word came first, second, or third.
Both sentences have the same words, but the meaning is completely different because the order is different. Positional encoding helps the model understand this order.
Each token gets two kinds of information:
Token embedding tells the model what the word means. Positional encoding tells where the word appears in the sequence.
- Sinusoidal positional encoding: fixed mathematical position signals used in the original Transformer.
- Learned positional embeddings: position vectors learned during training.
- Relative positional encoding: focuses on distance between tokens rather than absolute position.
- RoPE: commonly used in modern LLMs to encode relative position using rotation.
Encoder-only models read the full input at once using bidirectional attention. This means every token can look at tokens on both left and right sides.
They are strong when the task is to understand text rather than generate long text.
Decoder-only models generate text autoregressively, one token at a time. They use masked self-attention, so the model cannot look at future tokens while predicting the next token.
This is the main architecture behind modern LLMs.
These models have both an encoder and a decoder. The encoder understands the input, and the decoder generates the output.
The Transformer architecture solved many limitations of RNNs. It allowed models to process tokens in parallel, capture long-range dependencies using attention, and train efficiently on massive datasets.
- Parallel training: made large-scale training practical.
- Self-attention: improved context understanding.
- Scalability: larger models and more data improved performance significantly.
- Pretraining: models could learn general language patterns from huge text corpora.
- Fine-tuning and instruction tuning: made models useful for specific tasks and conversations.
- Decoder-only architecture: enabled next-token prediction at massive scale.
Modern LLMs are mostly Transformer-based. They are trained to predict the next token, then improved using instruction tuning, supervised fine-tuning, reinforcement learning from human feedback, preference optimization, and tool/RAG integrations.