Statistics for
Data Science 2

Long-run average, and why the house wins.

A random variable can give a different value every time. Expected value is one single number that tells you where those values settle on average, if you could repeat the experiment again and again. This chapter builds the idea, tests it on dice, and then shows why every casino game is designed to lose you money slowly.

1.1What is expected value?

1.2A fair die

For a fair die $E[X] = 1 \cdot \tfrac16 + 2 \cdot \tfrac16 + \cdots + 6 \cdot \tfrac16 = \tfrac{21}{6} = 3.5$. You can never roll a 3.5, and that is fine: the expected value need not be a possible outcome. It is only the average. If you roll 6000 times, the total will be close to $6000 \times 3.5 = 21{,}000$.

1.3A loaded die

1.4The indicator variable

1.5Casino math: why the house always wins

Casinos use expected value to make sure that, in the long run, they never lose. Every game is designed so that the player's expected gain is slightly negative. A few players win big, but over thousands of bets the average takes over.

The expected gain is $35 \cdot \tfrac1{37} - 1 \cdot \tfrac{36}{37} = -\tfrac1{37} \approx -0.027$. The casino keeps about 2.7 paise out of every rupee bet.

1.6Judging a game

1.7Going deeper

• Mean is not "most likely" (extra). In the loaded die the most likely face is 6, but the mean is 4.5. The mean is the balance point, which can differ from the highest bar.
• Why it works (extra). In n repeated trials a value x appears about n·P(X = x) times, so the total is about Σ x · n · P(X = x), and the average is about Σ x · P(X = x) = E[X].
• Few games, big swings (extra). Expected value tells you the average over many plays. In a single roulette spin you either win ₹35 or lose ₹1; the −₹0.027 only appears as a long-run average.

1.8Exam tips and common mistakes

Practice questions (graded-assignment style)

Linearity, functions (LOTUS) and the continuous case.

Now that you know what expected value means, here are the rules that let you calculate with it quickly, and how it works for continuous random variables.

2.1Four rules you will use all the time

2.2Counting heads with linearity

2.3LOTUS: the expected value of a function

To find $E[g(X)]$ you do not need the distribution of $g(X)$. Take each value $x$, apply $g$, multiply by $P(X = x)$, and add. The most common choice is $g(x) = x^2$, which we need for variance in Chapter 4.

2.4Continuous random variables

For a continuous random variable the "sum" turns into an integral (a smooth sum), and probabilities come from the density function $f(x)$:

2.5Going deeper

• LOTUS for a continuous variable (extra). The same idea works with an integral: $E[g(X)] = \int g(x) f(x)\,dx$. That is how E[X²] = 28 was computed above.
• Why linearity is so powerful (extra). It works even for strongly dependent variables, so you can find the mean of a complicated sum by finding the mean of each simple piece, as in "100 coins = 100 indicators".
• E[g(X)] is not g(E[X]) (extra). For linear $g(x) = ax + b$ they agree, but for curved $g$ they differ. Chapter 7 (Jensen) says which side is bigger.

2.6Means and variances of the named distributions, and conditional expectation

The summary table at the end of the week lists the mean and variance of the uniform, exponential and normal distributions. The discrete distributions of Week 0 (Bernoulli, binomial, geometric, Poisson, hypergeometric) are used just as often. Here is each result, with the reason.

A. The five discrete distributions

Why $E[G] = 1/p$ for the geometric. Look at the first trial. With probability $p$ it succeeds and $G = 1$. With probability $1 - p$ it fails, you have used one trial, and you are back at the start, so the expected total is $1 + E[G]$. Hence

Worked values.

• Binomial$(10, 0.3)$: mean $3$, variance $10(0.3)(0.7) = 2.1$, standard deviation $\sqrt{2.1} \approx 1.449$. (Computing $\sum k\,P(X = k)$ from the full PMF gives exactly $3$ and $2.1$.)
• Geometric$(1/6)$ (throw a die until the first 1): $E[G] = 6$ throws, $\operatorname{Var}(G) = \dfrac{5/6}{1/36} = 30$, standard deviation $\approx 5.48$. The spread is as big as the mean: waiting times are very variable.
• Poisson$(4)$: mean $4$, variance $4$, and $E[X^2] = 4 + 16 = 20$.
• Hypergeometric$(N = 10, r = 4, m = 3)$: mean $3 \cdot 0.4 = 1.2$, variance $3(0.4)(0.6)\dfrac{7}{9} = 0.56$. (Binomial$(3, 0.4)$ would have variance $0.72$; sampling without replacement lowers it.)

B. Sums and products of two variables

• Always: $E[X + Y] = E[X] + E[Y]$, independent or not.
• If $X$ and $Y$ are independent: $E[XY] = E[X]\,E[Y]$ and $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$.
• In general: $E[XY] = E[X]E[Y] + \operatorname{Cov}(X, Y)$ and $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X, Y)$.
• Two independent fair dice: sum has mean $3.5 + 3.5 = 7$ and variance $\tfrac{35}{12} + \tfrac{35}{12} = \tfrac{35}{6} \approx 5.83$; the product has $E[XY] = 3.5 \times 3.5 = 12.25$.

C. Conditional expectation and the law of total expectation

The conditional expectation of $X$ given an event $A$ (with $P(A) > 0$) is the average of $X$ over the conditional PMF:

If $B_1, \dots, B_n$ partition the sample space, then

This is the law of total probability carried over to averages: the overall average is the weighted average of the group averages.

• A die. $E[X \mid \text{even}] = \dfrac{2 + 4 + 6}{3} = 4$, $E[X \mid \text{odd}] = \dfrac{1 + 3 + 5}{3} = 3$, so $E[X] = 4 \cdot \tfrac12 + 3 \cdot \tfrac12 = 3.5$ ✓.
• From a joint PMF. Let $P(X = 0, Y = 0) = 0.2$, $P(0, 1) = 0.3$, $P(1, 0) = 0.1$, $P(1, 1) = 0.4$. Then $P(Y = 0) = 0.3$, $P(Y = 1) = 0.7$, $E[X \mid Y = 0] = \tfrac{0.1}{0.3} = \tfrac13$, $E[X \mid Y = 1] = \tfrac{0.4}{0.7} = \tfrac47$, and $E[X] = \tfrac13(0.3) + \tfrac47(0.7) = 0.1 + 0.4 = 0.5$, which matches $E[X] = P(X = 1) = 0.5$ computed directly ✓.

D. Standardising

For $X$ with mean $\mu$ and standard deviation $\sigma > 0$, the standardised variable $Z = \dfrac{X - \mu}{\sigma}$ has $E[Z] = 0$ and $\operatorname{Var}(Z) = 1$ (by the rules for $aX + b$). The value $z = \dfrac{x - \mu}{\sigma}$ says "how many standard deviations $x$ is from the mean". Example: marks with mean 60 and standard deviation 8: a score of 76 has $z = (76 - 60)/8 = 2$.

2.7Exam tips and common mistakes

Practice questions (graded-assignment style)

Reading patterns, range and IQR.

Before using formulas, it helps to look at data. A scatter plot shows how two variables are related, and measures of spread tell us how scattered the values are.

3.1Scatter plots

A scatter plot draws every observation as one dot at position (x, y). Look at the cloud of dots and ask three questions: Which direction? (up or down) How strong? (tight or loose) What shape? (straight or curved)

3.2Reading the shape

• Linear: dots follow a straight line. A straight-line model works well.
• Curved: dots follow a curve. The correlation can be close to 0 even though the link is strong, because correlation only sees straight-line patterns.
• Fan shape: the spread grows as x grows. Straight-line modelling assumptions may fail.
• Clusters: separate groups can behave differently from the combined data (this is called Simpson's paradox).
• Outliers: one extreme dot can change the correlation a lot, especially with few points. Always look at the plot first.

3.3Measuring spread

The mean says where the centre is, but not how scattered the values are. Three common measures of spread:

3.4Going deeper

• Simpson's paradox (extra). Two groups can each show a downward trend while the combined cloud shows an upward one (or the reverse). That is why clusters in a scatter plot are a warning: analyse the groups separately.
• Which spread when (extra). Range is quick but fragile; IQR is robust to outliers; variance and standard deviation are the ones that fit the formulas of this week.
• Plot before you compute (extra). A correlation number can hide a curve, a fan, clusters or an outlier. The plot takes five seconds and can save the whole analysis.

3.5Exam tips and common mistakes

Practice questions (graded-assignment style)

How far values wander from the mean.

Variance and standard deviation are the standard way to measure how far values wander from their mean. They appear in almost every formula in statistics, so take the time to get a feel for them.

4.1Variance

In simple words: take every value, find how far it is from the mean, square that distance, then take the average. That average is the variance.

Why square? Without squaring, distances above and below the mean cancel out. Squaring also makes big gaps count more. And Var(X) ≥ 0 always, with Var(X) = 0 only when X is a constant (no spread at all).

• Fair die. $E[X^2] = \tfrac{91}{6}$, $E[X] = 3.5$, so $\operatorname{Var}(X) = \tfrac{91}{6} - \tfrac{49}{4} = \tfrac{35}{12} \approx 2.917$.
• Bernoulli variable. If $X$ is 1 with chance $p$ and 0 otherwise, then $E[X^2] = p$ (since $X^2 = X$), so $\operatorname{Var}(X) = p - p^2 = p(1 - p)$.

4.2Standard deviation

$\sigma = \sqrt{\operatorname{Var}(X)}$ is the standard deviation. Taking the square root brings us back to the original units (marks, cm, rupees), so it is easier to understand than the variance (marks², cm², ...). If exam scores have mean 70 and $\sigma = 10$, most students score within $70 \pm 10$, that is between 60 and 80. If $\sigma = 2$, scores are packed between 68 and 72.

4.3Rules for variance

4.4The empirical rule (68-95-99.7)

For a roughly normal (bell-shaped) distribution: about 68% of the values lie within 1σ of the mean, about 95% within 2σ, and about 99.7% within 3σ.

4.5Going deeper

• Two ways to the same number (extra). The definition E[(X − μ)²] and the shortcut E[X²] − (E[X])² always agree. For X = 1, 3, 5 above the definition gives 0.2·(1 − 3.2)² + 0.5·(3 − 3.2)² + 0.3·(5 − 3.2)² = 0.968 + 0.02 + 0.972 = 1.96.
• Why E[X²] ≥ (E[X])² (extra). Their gap is the variance, which is never negative. Chapter 7 shows the same fact as a case of Jensen's inequality.
• Units (extra). If X is in cm, Var(X) is in cm² and σ is in cm. That is why we quote σ when talking to people and use Var in formulas.
• Max variance (extra). Among Bernoulli variables the largest variance is 0.25, at p = 0.5, as the curve shows.

4.6Exam tips and common mistakes

Practice questions (graded-assignment style)

Do two variables move together?

So far we looked at one variable at a time. Now we look at two variables together: do they rise and fall together, or do they pull in opposite directions?

5.1Covariance: the direction of a relationship

Covariance tells us whether X and Y tend to move in the same direction (positive), in opposite directions (negative), or show no tendency (zero). For each pair, multiply the two deviations from their means, then average.

Examples: temperature and ice-cream sales are positively related (both are high in summer). Temperature and the heating bill are negatively related (hot days mean small bills).

5.2Rules for covariance

5.3Correlation: covariance without units

5.4Properties of correlation

• $-1 \le \rho \le 1$ always (proved in Chapter 7).
• $\rho = \pm 1$ exactly when $Y = aX + b$ (a perfect straight line). The sign of $\rho$ is the sign of $a$.
• Symmetric: $\rho(X, Y) = \rho(Y, X)$.
• Independent gives $\rho = 0$, but $\rho = 0$ does not give independent (except for the bivariate normal).
• Under a change of scale: $\rho(aX + b, cY + d) = \operatorname{sign}(ac)\cdot\rho(X, Y)$. Flipping one variable flips the sign.

5.5Zero correlation does not mean no relationship

$\rho$ only detects straight-line relationships. Take $X \sim N(0, 1)$ and $Y = X^2$. Here $E[X] = 0$, $E[Y] = 1$ and $E[XY] = E[X^3] = 0$ (odd powers of a symmetric variable average to 0). So $\operatorname{Cov} = 0 - 0\cdot 1 = 0$, yet $Y$ is completely decided by $X$.

5.6Correlation is not causation

If two things move together, one need not cause the other. A third hidden factor (a confounder) can drive both. Even pure coincidence can produce a high correlation: the PDF quotes a funny example, the number of films a famous actor appears in per year, once reported to correlate (about 0.66) with pool drownings, which is obviously a coincidence.

5.7Going deeper

• Covariance as an average product (extra). Cov(X, Y) is the average of (X − μX)(Y − μY). Above-right and below-left dots give positive products, the other corners negative ones, which is exactly the picture of the quadrants.
• Correlation is symmetric but not about causes (extra). ρ(X, Y) = ρ(Y, X), so it cannot say which variable drives which.
• Special case (extra). For two jointly normal variables, ρ = 0 does give independence; that is the exception mentioned above.

5.8Exam tips and common mistakes

Practice questions (graded-assignment style)

Bounds from just the mean and spread.

Sometimes we do not know the full distribution, only a couple of numbers such as the mean and the variance. Probability inequalities still give a guaranteed upper limit on how likely something extreme can be.

6.1Markov's inequality: using only the mean

The idea behind the proof. Suppose too many (more than E[X]/a of them) are at or above a. Their total alone is already above E[X], and nobody is negative. That is a contradiction: the mean would be too big. So at most E[X]/a of the population can be that large.

6.2Chebyshev's inequality: using mean and variance

Where it comes from. Apply Markov to the non-negative variable $(X - \mu)^2$ with $a = k^2\sigma^2$: $P((X - \mu)^2 \ge k^2\sigma^2) \le \dfrac{E[(X-\mu)^2]}{k^2\sigma^2} = \dfrac{\sigma^2}{k^2\sigma^2} = \dfrac1{k^2}$.

Can it be improved? Not without extra assumptions: there are distributions where it holds with equality, so it is the best possible bound for general data.

6.3Going deeper

• Both are loose but valid (extra). They use very little information, so they must cover the worst case. That is why they are so safe and so cautious.
• Choosing the tool (extra). Only the mean and X ≥ 0: Markov. Mean and standard deviation: Chebyshev. A known shape (normal): the exact 68–95–99.7 figures.
• k below 1 says nothing (extra). For k = 1 the bound 1/k² = 1 is trivial, as the table shows (0% inside).

6.4Exam tips and common mistakes

Practice questions (graded-assignment style)

Curves, averages and the limit of correlation.

Two more famous inequalities: Jensen (what happens when you apply a curved function to a random variable) and Cauchy-Schwarz (why correlation can never go beyond ±1).

7.1Jensen's inequality