Every piece of data in ML is a vector. A user is a vector of features. A word is a vector of 768 floats. An image is a vector of pixel values. A model's hidden state is a vector. Before you can understand attention, embeddings, similarity search, or neural networks, you need to know what a vector is and what you can do with it.
Without this, the phrase "cosine similarity between two embeddings" is gibberish. With it, it's immediately obvious what's happening.
A vector is just a list of numbers with a direction. Think of it as an arrow in space. A 2D vector [3, 4] is an arrow that goes 3 steps right and 4 steps up. A 768-dimensional vector is the same idea — just an arrow in a space you can't visualize.
In ML, each number in the vector is a feature, and the entire vector is a compressed description of something — a word, a user, a sentence, an image patch.
A vector is an ordered list of numbers: v = [v₁, v₂, ..., vₙ]. The number of elements is the dimension of the vector. In a 3D vector [0.2, 0.8, 0.5], dimension = 3.
Vectors can be written as a column vector (n×1 matrix — default in most ML literature) or a row vector (1×n matrix). This distinction matters when you multiply with matrices — shape errors are almost always a row/column confusion.
v = [[1], ← column vector (3×1)
[2],
[3]]
The magnitude or length of a vector is how far the arrow reaches from the origin. Computed as the square root of the sum of squared components:
For v = [3, 4]: ||v|| = √(9 + 16) = √25 = 5
This is the L2 norm. The L1 norm (Manhattan distance) is just the sum of absolute values: |v₁| + |v₂| + .... L1 and L2 also appear as regularization penalties (Ridge = L2, Lasso = L1).
A unit vector has magnitude = 1. To convert any vector to a unit vector, divide by its magnitude: û = v / ||v||. This is called normalization.
Why? Sometimes you only care about direction, not magnitude. When comparing word embeddings, you normalize them first so that similarity is purely about direction, not how "large" the embedding is.
- A user in a recommendation system:
[age=0.3, purchase_freq=0.7, avg_spend=0.5, ...] - A word embedding (Word2Vec, BERT): dense vector of 768 floats — the "meaning" of a word
- A model's weight vector: every linear layer is a set of weight vectors, one per output neuron
- Gradient vector: the gradient ∇L is a vector pointing in the direction of steepest loss increase
The dot product is the single most used operation in ML. Every linear layer in a neural network is a dot product. Every attention score in a Transformer is a dot product. Every cosine similarity computation involves a dot product. If you understand nothing else from linear algebra, understand this.
The dot product measures how much two vectors point in the same direction. If they point the same way — large positive number. If they're perpendicular — zero. If they point opposite ways — negative number.
Think of it as a vote: each dimension casts a vote, and the dot product is the weighted total of agreement between the two vectors.
Multiply corresponding elements and sum them up:
Example: a = [1, 2, 3], b = [4, 5, 6]
There's a second formula: a · b = ||a|| × ||b|| × cos(θ), where θ is the angle between the vectors.
This is powerful because it tells you: the dot product is large when vectors are aligned (θ≈0°, cos≈1), zero when perpendicular (θ=90°, cos=0), and negative when opposite (θ=180°, cos=−1).
Cosine similarity normalizes the dot product to remove magnitude effects:
Result is always in [−1, 1]. Used everywhere in NLP: comparing embeddings of sentences, finding similar documents, measuring semantic closeness. It's how vector databases (Pinecone, Weaviate) do retrieval.
Every neuron in a linear layer computes: output = w · x + b — a dot product between the weight vector w and the input x, plus a bias. A layer with 512 neurons computes 512 dot products simultaneously, organized as a matrix multiply.
In Transformer attention: score = Q · Kᵀ — every query vector is dot-producted with every key vector to produce attention scores. High score = "this query should attend to this key".
Your entire training dataset is a matrix. Every weight layer in a neural network is a matrix. Every batch of data you pass through a model gets multiplied by weight matrices. Understanding matrices — what they represent, how they transform data — is non-negotiable for understanding how models work.
A matrix is a function that transforms vectors. Multiply a vector by a matrix, and you get a new vector — possibly in a different dimensional space, stretched, rotated, or projected. The weight matrix in a neural layer is literally a learned transformation: it reshapes the input data into a new representation more useful for the task.
A matrix is a 2D array of numbers with m rows and n columns, called an m×n matrix. Shape notation: (m, n).
[4, 5, 6]]
In ML, a dataset X of 1000 samples with 20 features each is a (1000, 20) matrix. A linear layer that maps 20 inputs to 512 outputs has a weight matrix W of shape (512, 20).
Transpose flips a matrix over its diagonal — rows become columns. An m×n matrix becomes n×m.
[3, 4]] [2, 4]]
Where it matters: in attention, Q · Kᵀ requires transposing K so the dimensions align. In PCA, you compute XᵀX to get the covariance matrix.
When you multiply matrix A (m×n) by vector x (n×1), you get a new vector of shape (m×1). The matrix has transformed x from n-dimensional space into m-dimensional space. Each row of A is a weight vector — it computes one output value as a dot product with x.
A neural network layer: output = W·input + bias. W transforms the input representation into a new one. Stack multiple layers and you get a deep network — each layer learns a progressively more abstract transformation.
The identity matrix I is the "do nothing" matrix — 1s on the diagonal, 0s elsewhere. A·I = A.
The inverse A⁻¹ (only for square matrices) satisfies A·A⁻¹ = I. Conceptually: if A transforms a vector, A⁻¹ undoes that transformation. Not all matrices are invertible — singular matrices (determinant = 0) have no inverse.
In ML this matters conceptually: the closed-form solution to linear regression is θ = (XᵀX)⁻¹Xᵀy. In practice you never compute inverses directly — numerically unstable. You use solvers.
A single number that describes how a square matrix scales space. Geometrically: if a matrix transforms the unit square, the determinant is the area of the resulting parallelogram.
- det = 0 → matrix collapses space to a lower dimension (not invertible, data is lost)
- det = 1 → matrix preserves area/volume (rotation is an example)
- det < 0 → matrix flips orientation
You won't compute determinants in interviews but you need to know: zero determinant = singular matrix = not invertible = some information is lost in the transformation.
Matrix multiplication is what makes neural networks fast. A forward pass through a linear layer is a matrix multiply. Running inference on a batch of 64 samples simultaneously is a matrix multiply. The entire efficiency of GPU-accelerated deep learning comes from GPUs being very good at matrix multiplications. If you don't understand this, you can't reason about model efficiency, memory, or batch processing.
Matrix multiplication is just doing many dot products at once. If A has m rows and B has n columns, then A×B produces an m×n matrix where entry (i,j) is the dot product of row i of A with column j of B. Think of it as: "row i votes on column j."
The critical rule: inner dimensions must match. A(m×k) × B(k×n) → C(m×n). The k must be the same. Shape errors are the most common bug — always check shapes first.
C = A × B where A is (m×k) and B is (k×n), result C is (m×n).
Each element C[i][j] = dot(row_i of A, col_j of B)
[3, 4]] [7, 8]]
C[0][0] = 1×5 + 2×7 = 19
C[0][1] = 1×6 + 2×8 = 22
C[1][0] = 3×5 + 4×7 = 43
C[1][1] = 3×6 + 4×8 = 50
C = [[19, 22], [43, 50]]
↑___↑
inner dims must match
outer dims are result shape
Examples from ML:
- Linear layer: input (batch=32, features=256) × weights (256, 512) → output (32, 512)
- Attention scores: Q(seq=10, d=64) × Kᵀ(d=64, seq=10) → scores(10, 10)
- Matrix multiply is NOT commutative: A×B ≠ B×A in general
In practice, you always process batches. If X is a batch of 32 samples, each of dimension 256, and W is (256, 512), then X @ W produces (32, 512) — all 32 forward passes computed simultaneously, which is why GPUs are so effective: they parallelize thousands of dot products at once.
In PyTorch: torch.matmul(X, W) or the @ operator. Know the shapes before you call it.
PCA — the most common dimensionality reduction technique — is entirely based on eigenvalues and eigenvectors of the covariance matrix. Interviewers ask about PCA constantly, and the follow-up is always "how does it work mathematically?" You need eigenvalues to answer that. They also appear in graph neural networks, spectral clustering, and stability analysis of training dynamics.
Most vectors change direction when you multiply them by a matrix — they get rotated and stretched. But certain special vectors don't change direction — they only get stretched (or shrunk). These are eigenvectors. The amount of stretching is the eigenvalue.
Think of it this way: if a matrix represents "how your data varies," eigenvectors point in the directions of maximum variance. The eigenvector with the largest eigenvalue points in the direction where your data spreads out the most. That's your first principal component in PCA.
For a square matrix A, a non-zero vector v is an eigenvector if multiplying A by v gives back v scaled by a constant λ (lambda):
v is the eigenvector. λ is the eigenvalue. The eigenvector's direction is preserved; it's only scaled. A large λ means the matrix strongly stretches data in that direction. λ = 0 means the matrix collapses data in that direction (loss of information).
[0, 3]]
A · v = [[2×1 + 0×0], = [[2], = 2 × [1, 0]
[0×1 + 3×0]] [0]]
So v = [1, 0] is an eigenvector with eigenvalue λ = 2
And v = [0, 1] is an eigenvector with eigenvalue λ = 3
Interpretation: this matrix stretches the x-direction by 2× and the y-direction by 3×. The y-direction has more variance.
PCA goal: find the directions of maximum variance in your data, then project onto those directions to reduce dimensions.
Step 1: Compute the covariance matrix of your data (C = XᵀX / n). This symmetric matrix captures how each pair of features varies together.
Step 2: Find eigenvectors of C. Each eigenvector is a direction in feature space. Its eigenvalue = variance of the data in that direction.
Step 3: Sort eigenvectors by eigenvalue (largest first). Take the top-k — these are your principal components.
Step 4: Project your data onto these k eigenvectors → you get a compressed representation with maximum information retained.
- Symmetric matrices (like covariance matrices) always have real eigenvalues and orthogonal eigenvectors
- Eigenvalues sum = trace of the matrix (sum of diagonal elements)
- Eigenvalues multiply = determinant of the matrix
- An n×n matrix has at most n eigenvalues
- Zero eigenvalue → matrix is singular (not invertible)
SVD is how modern systems compress information. Image compression, recommendation systems, and — critically for LLMs — LoRA fine-tuning all use SVD or its logic. It generalizes eigendecomposition to non-square matrices, which is most ML weight matrices. Interviewers at ML engineering roles will ask about this, especially if they work on LLMs.
SVD says: any matrix can be decomposed into three simple operations — rotate, stretch, rotate again. The singular values in the middle tell you which directions carry the most information. If you keep only the top-k singular values and set the rest to zero, you get the best possible rank-k approximation of the original matrix — the compressed version that loses the least information.
Any matrix A of shape (m×n) can be factored as:
U: (m×m) — left singular vectors (orthogonal matrix, columns are directions in output space)
Σ: (m×n) — diagonal matrix of singular values σ₁ ≥ σ₂ ≥ ... ≥ 0
Vᵀ: (n×n) — right singular vectors transposed (directions in input space)
Intuition of the three steps when you compute y = Av:
- Vᵀ·v: rotate/reflect the input vector into a new coordinate system
- Σ·(Vᵀ·v): stretch along each axis by the singular value
- U·(Σ·Vᵀ·v): rotate/reflect into the output space
If you keep only the top-k singular values (set the rest to zero), you get the best rank-k approximation of A:
where U_k is (m×k), Σ_k is (k×k), V_kᵀ is (k×n). This is rank-k approximation — you've compressed the original m×n matrix (m·n numbers) into far fewer: m·k + k + k·n numbers.
This is mathematically proven to be the best possible rank-k approximation by the Eckart-Young theorem — no other rank-k matrix is closer to A in Frobenius norm.
- Eigendecomposition only works on square matrices. SVD works on any matrix
- For square symmetric matrices: singular values = absolute eigenvalues; U = V = eigenvectors
- PCA on data matrix X = truncated SVD of X directly (equivalent to eigendecomposition of covariance matrix)
- SVD is numerically more stable — used in all practical implementations of PCA
- PCA: sklearn's PCA uses SVD internally, not eigendecomposition
- Recommendation systems: matrix factorization (user-item rating matrix decomposed into low-rank factors)
- LoRA (Low-Rank Adaptation): fine-tuning LLMs by representing weight updates as two small matrices A·B (rank-r approximation). Instead of updating W (d×d), you update A (d×r) and B (r×d) where r ≪ d. This is SVD intuition in practice
- Image compression: SVD of pixel matrix, keep top-k singular values