Earlier NLP systems represented text using simple symbolic or statistical features.

Examples:

Suppose we have a vocabulary:

A one-hot representation would look like:

The problem is that one-hot vectors do not capture meaning.

In this representation:

because every pair of different words has the same distance.

So the model cannot naturally understand that:

This created the need for dense vector representations, called embeddings.

An embedding is a dense numerical vector that represents a word, token, sentence, document, image, user, product, or any other object in a continuous vector space.

For NLP:

Example:

The key idea is:

So embeddings convert discrete language into a mathematical form that neural networks can process.

A good embedding space should capture relationships like:

These relationships are not manually programmed. They emerge because words appear in similar contexts during training.

LLMs do not directly understand raw text.

The input pipeline is roughly:

Example:

First tokenization happens:

Then each token is converted into an integer ID:

Then each token ID is mapped to an embedding vector:

These vectors are the real input to the neural network.

So in LLMs:

Important distinction:

Example:

The token “bank” starts with a base embedding. But after transformer layers process the full sentence, the representation of “bank” becomes different in each sentence.

That is the bridge from static meaning to contextual meaning.

Embeddings usually start as random vectors.

During training, the model tries to solve a task. When it makes mistakes, the loss is calculated, and backpropagation updates the model weights, including the embedding vectors.

Basic training flow:

Over millions or billions of examples, embeddings become meaningful.

Why?

Because words used in similar contexts receive similar gradient updates.

Example:

Words like doctor, nurse, and physician appear in similar contexts. Their vectors are repeatedly adjusted in similar directions.

So the model learns:

This is not manually labeled. It emerges from distributional patterns.

The principle is:

Word2Vec was a major breakthrough because it showed that useful word meaning could be learned from raw text.

It introduced dense word vectors trained from local context.

Word2Vec has two main architectures:

After Word2Vec, NLP moved toward models that could process sequences.

Common sequence models:

The idea was to read text step by step and maintain a hidden state.

Example:

An RNN processes this sequentially:

The hidden state acts like a memory of what the model has seen so far.

ELMo was important because it introduced widely used contextual word embeddings.

Unlike Word2Vec:

Example:

In ELMo, “bank” gets different representations in each sentence.

Sequence models like LSTMs compressed information into hidden states.

In translation, this was a big problem.

Example:

A traditional encoder-decoder model had to compress the entire sentence into one fixed vector before decoding.

This creates an information bottleneck.

Attention solved this by allowing the decoder to look back at different parts of the input whenever needed.

Instead of relying only on one compressed vector, the model can dynamically focus on relevant words.

Simple idea:

Attention is a mechanism that calculates how much focus one token or state should give to other tokens or states.

Suppose we are translating:

When generating the translated word for “cat,” the model should focus strongly on “cat.”

When generating the translated word for “mat,” it should focus on “mat.”

So attention produces weights like:

Then it creates a weighted combination of input representations.

The general formula is:

Where weights represent relevance.

Modern attention uses three vectors:

Intuition:

Library analogy:

In attention:

• A token creates a Query.
• Other tokens provide Keys.
• The Query is compared with each Key.
• Similarity scores are calculated.
• Scores are normalized into attention weights.
• Values are combined using those weights.

Simplified:

The actual scaled dot-product attention formula is:

Meaning:

Self-attention means attention within the same sequence.

Each token attends to other tokens in the same input.

Example:

To understand “it,” the model should attend to “cat.”

Self-attention allows:

So every token gets updated based on relevant tokens around it.

In RNNs/LSTMs:

Example:

If token 1 needs to influence token 100, the signal must pass through many steps.

In self-attention:

This gives direct access to long-range dependencies.

Also, self-attention allows parallel computation.

RNN:

Transformer:

This is one of the biggest reasons transformers scale so well.

Main advantages:

One attention mechanism may focus on one type of relationship.

But language has many relationships:

Multi-head attention runs multiple attention heads in parallel.

Each head has its own Q, K, V projections.

Example:

Then the outputs of all heads are combined.

Why useful?

Because the model can look at the sentence from multiple perspectives at the same time.

Interview answer:

Multi-head attention allows the model to learn different types of relationships in parallel, making attention richer and more expressive.

Attention is the general idea of focusing on relevant information.

Self-attention is attention applied within the same sequence.

Difference:

Example of attention in translation:

Example of self-attention:

In modern transformers:

GPT-style LLMs mainly use decoder-only transformer architecture with causal self-attention.

Normal self-attention can allow each token to attend to all tokens.

But GPT-style models generate text left to right.

So while predicting the next token, the model should not look into the future.

Example:

The model predicts the next token.

During training, for a sequence:

When predicting “India,” the model should not see “New Delhi” yet.

So GPT uses causal masking.

Causal self-attention means:

Masking pattern:

This preserves autoregressive generation.

Interview answer:

GPT-style models use causal self-attention so each token can only attend to previous tokens, enabling left-to-right next-token prediction.

A transformer is a neural network architecture built around self-attention.

A transformer block usually has:

Simplified block:

This block is repeated many times.

Example:

Each layer refines token representations.

Early layers may learn lower-level patterns like syntax.

Middle layers may capture phrases and relationships.

Higher layers may capture abstract meaning, reasoning patterns, and task behavior.

There are different transformer variants.

Self-attention does not naturally know order.

Without positional information:

would look too similar because the model sees the same set of tokens.

So transformers add position information.

Input becomes:

This tells the model:

Modern models may use:

Positional information is essential because language meaning depends heavily on order.

After attention mixes information across tokens, each token representation passes through a feed-forward neural network.

This FFN is applied independently to each token.

Attention answers:

Feed-forward network answers:

Typical FFN:

The FFN increases the model’s capacity to transform and store complex patterns.

In modern LLMs, the feed-forward layers often contain a large portion of the model parameters.

Interview answer:

Attention mixes information across tokens, while the feed-forward network transforms each token’s contextual representation.

Transformers are deep networks. Deep networks are hard to train without stabilizing techniques.

Two important components are:

Residual connection means:

Instead of replacing the input completely, the model adds a learned change to it.

This helps gradients flow through many layers.

Layer normalization stabilizes the distribution of activations, making training more stable.

Together, they allow transformers to scale to many layers.

Interview answer:

Residual connections and layer normalization stabilize transformer training and allow very deep models to learn effectively.

A transformer becomes an LLM when it is scaled in:

GPT-style LLM training objective:

Example:

Then:

This is repeated over massive text corpora.

Through this objective, the model learns:

The surprising part is that a simple next-token objective, when scaled massively, leads to general-purpose behavior.

The development path looks like this:

Each step solved a limitation of the previous step.

Word2Vec solved sparse representation.

ELMo solved static meaning.

Attention solved fixed-vector bottlenecks.

Self-attention solved sequential dependency limitations.

Transformers solved scalability.

LLMs scaled transformers with massive data and compute.

A strong interview answer could be:

Modern LLMs evolved from earlier NLP representations. Traditional methods used sparse features like one-hot vectors and TF-IDF, which could not capture semantic meaning. Word2Vec introduced dense static embeddings, where words appearing in similar contexts had similar vectors. However, Word2Vec gave each word only one representation, so it could not handle context-dependent meanings. ELMo improved this by using bidirectional LSTMs to generate contextual embeddings, where the same word could have different representations depending on the sentence.

Attention was introduced to solve the bottleneck of compressing an entire sequence into a single hidden state. It allowed models to dynamically focus on relevant parts of the input. Self-attention extended this idea by allowing every token in a sequence to attend to every other token. This made it easier to capture long-range dependencies and enabled parallel computation.

The Transformer architecture combined self-attention, multi-head attention, feed-forward networks, residual connections, layer normalization, and positional encoding. Because transformers scale well with data and compute, they became the foundation of modern LLMs. GPT-style LLMs use decoder-only transformers with causal self-attention and are trained using next-token prediction. At scale, this allows them to learn grammar, knowledge, reasoning patterns, and general-purpose language behavior.

The most important line: