Before building any model, you need to understand your data. Interviewers frequently open ML system design rounds with "you have this dataset — how would you approach it?" The expected first answer is always exploratory data analysis, and descriptive statistics are the tools of EDA. Get these wrong and the interviewer loses confidence in your engineering judgment before the real questions even start.
Beyond EDA, these concepts appear in technical depth too: PCA requires the covariance matrix, normalisation decisions require understanding mean and variance, and model evaluation often involves comparing distributions of predictions vs actuals.
Descriptive statistics are summaries. Instead of looking at 1 million data points, you reduce them to a handful of numbers that capture the most important properties: where is the data centred (mean/median), how spread out is it (variance/std), how is it shaped (skewness/kurtosis), and how do different features relate to each other (covariance)?
Think of them as a "health check" for your data — you run these before any modelling to catch problems: skewed distributions, outliers, correlated features, missing patterns.
The mean is the "balance point" of the distribution — it minimises the sum of squared deviations. The median minimises the sum of absolute deviations. The key difference is sensitivity to outliers:
Variance measures how spread out values are around the mean. Squaring the deviations makes all terms positive and penalises large deviations more. Standard deviation is just variance back in the original units — easier to interpret.
- Small variance → values cluster tightly around the mean
- Large variance → values are spread widely
- Variance of a constant = 0
- Adding a constant to all values doesn't change variance; multiplying by c scales variance by c²
- In ML: feature normalisation (subtract mean, divide by std) puts all features on the same scale — essential for gradient-based models and distance-based models (KNN, SVM)
For a dataset with d features, the covariance matrix Σ is a d×d symmetric matrix where entry (i,j) = Cov(Xᵢ, Xⱼ):
The covariance matrix is central to PCA — eigenvectors of Σ are the principal components, eigenvalues are the variance along each component. Large off-diagonal values mean features are correlated and potentially redundant.
Skewness > 0 → right-skewed: long tail on the right, mean > median (e.g. income, house prices)
Skewness < 0 → left-skewed: long tail on the left, mean < median
Highly skewed features often benefit from log-transform before feeding into linear models.
High kurtosis (leptokurtic): heavy tails, more extreme outliers — common in financial data.
Low kurtosis (platykurtic): light tails, fewer extremes.
Heavy-tailed data means outliers are more frequent than Gaussian would predict.
This is the single most important concept in applied ML. Every modelling decision — choosing model complexity, adding regularisation, collecting more data, applying dropout — is a move along the bias-variance spectrum. If you can't explain the tradeoff clearly, interviewers will question your understanding of why models fail and how to fix them.
It's asked directly ("explain bias-variance tradeoff") and indirectly ("why is your model overfitting?", "when would you use L2 regularisation?", "how would you fix underfitting?"). All are the same question in disguise.
Bias is the error from wrong assumptions. A linear model fitting a sine wave has high bias — it's systematically wrong no matter how much data you give it. It's not paying attention to the data closely enough.
Variance is the error from being too sensitive to the training data. A 100-layer decision tree memorises every training point but fails completely on new data. It's paying too much attention — to noise.
The tradeoff: making a model more complex (lower bias) almost always makes it more sensitive to the training data (higher variance). The sweet spot is a model complex enough to capture real patterns but not so complex it captures noise.
For any model, the expected prediction error at a point x can be decomposed as:
The irreducible noise is a floor — even a perfect model cannot go below it. Your job is to minimise Bias² + Variance.
Every time you deploy a new model and ask "is this better than the old one?", you are running a hypothesis test. A/B testing — the standard way to evaluate model changes in production — is applied hypothesis testing. Data science roles test this heavily because it separates engineers who make decisions based on evidence from those who make decisions based on vibes.
You'll be asked about p-values, Type I/II errors, and confidence intervals. The classic trap: "our p-value is 0.03, so our new model is significantly better" — which ignores practical significance, multiple comparisons, and power.
Hypothesis testing starts from scepticism. You assume the boring explanation is true — "there's no effect, any difference I see is just random chance" (null hypothesis). Then you measure how surprising your data would be if that boring explanation were true. If the data is extremely surprising under the null, you reject it.
The p-value answers: "if the null were actually true, how often would I see a result at least this extreme just by luck?" Small p → "this would be very unlikely by luck" → evidence against the null. It does NOT tell you the probability that the null is true.
- Type I (α): you say "the new model is better" when it isn't. Setting α=0.05 means you accept a 5% false positive rate.
- Type II (β): you say "no significant difference" when the new model actually is better. Power = 1-β = probability of correctly detecting a real effect.
- Lowering α reduces Type I errors but increases Type II errors (harder to detect real effects) — the classic tradeoff.
Confidence intervals are more informative than p-values alone — they tell you both statistical significance AND the magnitude and precision of the effect.
If you run 20 hypothesis tests at α=0.05 and nothing is actually different, you'd expect 1 false positive just by chance (0.05 × 20 = 1). This is the multiple comparisons problem — running many tests inflates your effective false positive rate.
- Bonferroni correction: divide α by the number of tests. If running 20 tests, use α = 0.05/20 = 0.0025 per test. Conservative but simple.
- FDR (False Discovery Rate): Benjamini-Hochberg procedure controls the expected fraction of false positives among all rejections. Less conservative than Bonferroni.
- Relevant in ML: testing many features for significance, hyperparameter search, comparing multiple models simultaneously.
Confusing correlation with causation is one of the most common and costly errors in applied data science. Models trained on correlations will fail the moment the correlation breaks — and it always eventually breaks. Understanding this distinction is the difference between a data scientist who builds robust systems and one who keeps being surprised when models degrade in production.
It also comes up in feature selection (correlated features vs features that actually drive the target), model interpretation (a feature having a high coefficient doesn't mean it causes the outcome), and business recommendations ("our model says X correlates with churn — should we change X?").
Two variables can move together for three reasons: X causes Y, Y causes X, or a third variable Z causes both X and Y. Correlation only tells you they move together — it says nothing about why. Causation requires ruling out the other two explanations, which requires either a randomised experiment or careful causal reasoning.
The ice cream and drowning example: both rise in summer, not because ice cream causes drowning, but because hot weather (a confounder) causes both. Your model will learn this correlation happily — and then give you the wrong recommendation ("ban ice cream to prevent drowning").
Pearson r measures only linear relationships. Two variables can have r=0 and still have a strong non-linear relationship (e.g. Y = X²). This is a critical limitation.
Also, r is scale-invariant — multiplying X or Y by a constant doesn't change r. But unlike covariance, r is unitless and directly comparable across different pairs of variables.
Spearman asks: "when X goes up, does Y tend to go up?" (monotonic), whereas Pearson asks: "when X goes up by 1 unit, does Y go up by exactly r units?" (linear). Spearman is more robust and often more appropriate for ML feature analysis.
For most ML interviews, the key is to demonstrate awareness: correlation ≠ causation, confounders exist, and acting on a correlational finding without causal reasoning can waste resources or cause harm.