Training a model means minimizing a loss function. To minimize it, you need to know which direction makes it decrease — and that direction is given by the derivative. Without derivatives there is no gradient descent, no backprop, no training. Every optimizer (SGD, Adam, RMSProp) is entirely built on derivatives. This is the most foundational concept in ML training.
A derivative answers one question: if I nudge the input by a tiny amount, how much does the output change? Geometrically, it's the slope of the curve at a single point — the steepness of the tangent line there.
Imagine you're standing on a hilly landscape (your loss surface). The derivative at your current position tells you: how steep is the ground right here, and in which direction does it slope? That's all you need to know to take one step downhill.
The derivative of f at point x is the limit of the slope of the secant line as the two points get infinitely close:
You don't need to compute limits in interviews — but understanding this says: derivative = instantaneous rate of change = slope of tangent at x.
If f'(x) > 0 → function is increasing at x. If f'(x) < 0 → decreasing. If f'(x) = 0 → flat — could be a minimum, maximum, or saddle point.
And the two you'll need for activation functions:
The second derivative f''(x) is the derivative of the derivative — it measures how the slope is changing, i.e. curvature.
- f''(x) > 0 at a critical point → local minimum (curve is concave up, like a bowl)
- f''(x) < 0 at a critical point → local maximum (concave down, like a hill)
- f''(x) = 0 → inflection point
In ML: second-order optimizers (like Newton's method) use the Hessian matrix — the matrix of all second partial derivatives — to take more informed steps. Computationally expensive for large models, but theoretically more efficient.
A neural network has millions of parameters. The loss function depends on all of them simultaneously. A regular derivative only handles one variable. Partial derivatives extend this: they let you ask "how does the loss change with respect to this one specific weight, while holding all others fixed?" This is exactly what backprop computes — a partial derivative for every single weight in the network.
Imagine a landscape where your position is described by two coordinates (x, y) and your altitude is f(x, y). A partial derivative ∂f/∂x asks: if I take one step in the x-direction only (keeping y frozen), how much does my altitude change? ∂f/∂y asks the same for the y-direction.
In ML: x and y are two weights. f is the loss. Partial derivatives tell you the slope in each weight's direction independently.
Treat all other variables as constants, then differentiate normally with respect to the target variable.
The notation ∂ (curly d) distinguishes partial from total derivatives. Read ∂f/∂w₁ as "the partial derivative of f with respect to w₁."
In a simple linear model: ŷ = w·x + b, loss = MSE = (y - ŷ)²
These two partial derivatives tell you exactly how to update w and b to reduce the loss. This is the update rule for linear regression gradient descent.
A deep network has parameters w₁, w₂, ..., wₙ (millions of them). Backprop computes ∂L/∂wᵢ for every single weight wᵢ. Each tells you: nudge this weight in this direction by this much to reduce the loss. The collection of all these partial derivatives is the gradient — covered next.
A neural network is a composition of functions — layer 1 feeds into layer 2, feeds into layer 3, and so on. To compute how the loss at the end depends on weights at layer 1, you need the chain rule. Backpropagation is literally "apply the chain rule through a computational graph." This is the single most important rule to understand if you want to explain how neural networks learn.
The chain rule handles composed functions: if y depends on u, and u depends on x, then how does y change when x changes? Answer: multiply the rates. If u doubles when x increases by 1, and y triples when u doubles — then y increases by 6 when x increases by 1. You chain the rates together by multiplying.
In a neural network: loss depends on the output layer, which depends on the hidden layer, which depends on the weights. The chain rule lets you trace this dependency all the way back.
If y = f(u) and u = g(x), then:
The derivative of the outer function times the derivative of the inner function. For deeper compositions:
A chain of multiplications — one factor per function in the composition.
Consider a two-layer network:
To find ∂L/∂w₁ (how loss depends on weight in layer 1):
Each factor in this product is a local derivative — easy to compute at each node. Backprop walks this chain from right to left (loss → output → hidden → input), accumulating products. This is the full mechanism of backpropagation.
Notice that backprop multiplies many numbers together as it propagates backward. Two problems arise:
The gradient is what every optimizer actually uses. When you call loss.backward() in PyTorch, it computes the gradient of the loss with respect to every parameter. The gradient vector is the complete answer to "which direction makes the loss increase the fastest?" — and you walk in the opposite direction to minimize it.
Think of the gradient as a compass for a hilly landscape with millions of dimensions. It points uphill — in the direction of steepest increase. Flip it and you get the direction of steepest descent. Each component of the gradient tells you the slope in one parameter's direction.
The gradient always points perpendicular to contour lines (lines of equal loss). To descend most efficiently, you follow the negative gradient.
For a function f(w₁, w₂, ..., wₙ) with n parameters, the gradient is a vector of all partial derivatives:
The gradient has the same shape as the parameter vector — one number per parameter. For a network with 10 million weights, the gradient is a 10-million-dimensional vector.
- ∇f at a point points in the direction of steepest increase of f
- −∇f points in the direction of steepest decrease → this is the descent direction
- The magnitude ||∇f|| tells you how steep the slope is at that point
- At a minimum (or maximum or saddle point): ∇f = 0 (all partial derivatives are zero)
- The gradient is perpendicular to the level curves (contour lines) of f
For logistic regression with sigmoid output σ(z) and binary cross-entropy loss:
Beautifully simple: the gradient is just how wrong your prediction is. If σ(z) = 0.9 and y = 0, gradient = 0.9 — large, because you were very wrong. This is why cross-entropy + sigmoid is the standard for binary classification.
This is the training algorithm. Everything before this was building up to it. Gradient descent is how weights get updated, how models improve over epochs, how a random initialization becomes a useful model. Understanding it — and why Adam beats vanilla SGD — is expected in every ML interview, from junior to senior.
Imagine you're blindfolded on a hilly landscape and want to reach the lowest point. You can only feel the slope under your feet. Strategy: take a small step in the downhill direction, feel the slope again, take another step. Repeat. That's gradient descent. The learning rate is your step size — too large and you overshoot valleys; too small and you take forever.
At each step, every parameter moves a small amount opposite to its gradient:
If ∂L/∂w is positive (increasing w increases loss) → subtract → decrease w → reduce loss. If negative → add → increase w → reduce loss. The sign always works out correctly.
Mini-batch is the standard. Batch size is a hyperparameter — smaller = noisier gradients (can help generalization), larger = more stable (better GPU utilization, may converge to sharp minima that generalize worse).
Common practice: use a learning rate scheduler — start high, decay over time (step decay, cosine annealing, warmup + decay).
Vanilla SGD uses the same learning rate for every parameter. Adam adapts the learning rate per parameter using running estimates of the gradient's mean and variance:
This section was cut short during authoring — the Adam explanation above stops mid-derivation. Finish it (why the bias correction matters, and how Adam compares to RMSProp/AdaGrad) before treating this topic as complete.
Placeholder — this section is linked from the nav but has no content yet. Needs: the chain-rule walkthrough for a multi-layer network, why gradients vanish/explode, and how autodiff frameworks compute this automatically.
Placeholder — this section is linked from the nav but has no content yet. Needs: what makes a loss surface convex vs non-convex, why neural network loss landscapes are non-convex, and why gradient descent still works well in practice despite that.