Statistics for
Data Science 2

Most experiments produce several numbers at once, and those numbers are often related. This week builds the tools to describe such relationships.

1.1Why one random variable is not enough

• Most real experiments produce several related quantities at once.
• This week builds tools to describe several random variables on the same probability space, and how they relate to each other.

Example: toss a coin thrice

$X_i = 1$ if toss $i$ is Heads, $0$ if Tails, $i = 1, 2, 3$. All three are defined on the same three tosses, so they live on the same probability space. Together $X_1, X_2, X_3$ fully describe the outcome. Here they are independent of each other; this will not always be true.

Example: a random 2-digit number

Pick a 2-digit number $00$–$99$. $X$ = units digit, $Y$ = remainder mod 4. Direct counting shows $X \sim \text{Uniform}\{0, \dots, 9\}$ and $Y \sim \text{Uniform}\{0, 1, 2, 3\}$.

Example: an IPL powerplay over

$X$ = runs in an over, $Y$ = wickets in the over. Intuition: if $Y = 0$ (no wicket), $X$ tends to be larger, because the batting side was not interrupted. Modelling exactly this kind of dependency is the goal of the chapter.

1.2Going deeper: random variables on one experiment

• A random variable is a function of the outcome. Several random variables on one experiment are several functions of the same outcome. For three tosses, $X_1, X_2, X_3$ are the tosses, and $S = X_1 + X_2 + X_3$ (number of Heads) is one more random variable on the same space.
• Independent means no information is carried. $X$ and $Y$ are independent when learning $X$ never changes what you can say about $Y$. In terms of the PMFs, $f_{XY} = f_X f_Y$ in every cell.
• If one determines the other, they are dependent. If $Y$ is a function of $X$ (like "even or odd" of a die roll), then knowing $X$ fixes $Y$, so they are dependent unless $Y$ is a constant.
• The fastest proof of dependence is one impossible combination: $P(X = a, Y = b) = 0$ although $P(X = a) > 0$ and $P(Y = b) > 0$.
• Three or more variables. Mutual independence means the whole joint factorises: $f(t_1, t_2, t_3) = f(t_1) f(t_2) f(t_3)$. The joint PMF, not the separate PMFs, carries the relationships.

1.3Exam tips and common mistakes

Practice questions (graded-assignment style)

The joint PMF gives the probability of every pair of values at once. It is the complete description of two random variables on the same experiment.

2.1The joint PMF

Example: toss a fair coin twice

$X_1, X_2$ mark Heads (1) / Tails (0) on two independent tosses. $f_{X_1X_2}(0, 0) = P(X_1 = 0)\,P(X_2 = 0) = \frac{1}{2} \times \frac{1}{2} = \frac{1}{4}$, and the same logic gives every other cell.

Example: a random 2-digit number

• $f_{XY}(0, 0)$: units digit 0 and divisible by 4 gives $\{00, 20, 40, 60, 80\}$, so $\frac{5}{100} = \frac{1}{20}$.
• $f_{XY}(1, 0)$: units digit 1 can never make a multiple of 4, so $0$.
• $f_{XY}(4, 2)$: $\{14, 34, 54, 74, 94\}$, so $\frac{5}{100} = \frac{1}{20}$.

Every cell works out to $0$ or $\frac{1}{20}$. There are $10 \times 4 = 40$ cells in the table, and exactly 20 of them are nonzero: for each units digit, the tens digit can only produce two different remainders mod 4. And $20 \times \frac{1}{20} = 1$: a valid joint PMF.

2.2Going deeper: using a joint PMF table

• Reading regions. The probability of any event about $(X, Y)$ is the sum of the cells it contains. For $P(X + Y = 3)$, add the cells on the diagonal where the coordinates sum to 3; for $P(X < Y)$, add the cells above the diagonal.
• Finding unknown constants. Step 1: all cells add to 1. Step 2: if a probability is given, write it as a second sum. Step 3: solve the two equations together (as in GA Q7).
• A function of two variables. To get the PMF of $Z = g(X, Y)$, group the cells by the value of $g$ and add each group. For two fair coins, $Z = X_1 + X_2$ gives $P(Z = 0) = \frac{1}{4}$, $P(Z = 1) = \frac{1}{2}$, $P(Z = 2) = \frac{1}{4}$.
• Building a table. If the variables are independent, multiply marginals. If the experiment is a selection without replacement, count: $f_{XY}(x, y) = \frac{\binom{a}{x}\binom{b}{y}\binom{c}{n - x - y}}{\binom{N}{n}}$.
• Size of the table. $|T_X| \times |T_Y|$ cells. Many may be 0, and the nonzero ones must still add up to 1.

2.3Exam tips and common mistakes

Practice questions (graded-assignment style)

From the joint PMF we can always recover each variable on its own by summing out the other. The reverse is not possible.

3.1Marginal PMF

Example: toss a fair coin twice

Sum down the columns for $f_{X_1}$, across the rows for $f_{X_2}$:

Example: marginals from a joint PMF

Same marginals, different joints

Both tables below give marginals $\frac{1}{2}, \frac{1}{2}$ for $X_1$ and $X_2$. Case I (independent): every cell $= \frac{1}{4}$. Case II: for any $x$ between $0$ and $\frac{1}{2}$, $f(0, 0) = x$, $f(0, 1) = \frac{1}{2} - x$, $f(1, 0) = \frac{1}{2} - x$, $f(1, 1) = x$.

Example: marginals and an independence check

For the random 2-digit number, summing the table confirms $X \sim \text{Uniform}\{0, \dots, 9\}$ and $Y \sim \text{Uniform}\{0, 1, 2, 3\}$, matching the direct counting. Independence would need $f_{XY} = f_X f_Y$ in every cell. At $(0, 0)$: $f_{XY}(0, 0) = \frac{1}{20}$ but $f_X(0) f_Y(0) = \frac{1}{10} \cdot \frac{1}{4} = \frac{1}{40}$. Since $\frac{1}{20} \neq \frac{1}{40}$, they are not independent, matching the earlier intuition.

3.2Going deeper: marginals and independence from a table

• Margins of the table. Add each column to get one marginal and each row to get the other. They are written in the margins, hence the name, and each must add to 1.
• One-way street. The joint gives the marginals uniquely, but the marginals alone cannot give the joint. Two tables can share the same margins and differ inside (Case I and Case II).
• Independence gives the joint back. Only when $X$ and $Y$ are independent can the joint be rebuilt from the marginals, by multiplying: $f_{XY} = f_X f_Y$.
• Quick dependence test. A zero cell whose row and column marginals are both positive proves dependence at once.
• Full independence test. Compute $f_X(t) f_Y(t^{\prime})$ for every cell and compare with $f_{XY}$. Stop at the first mismatch.

3.3Exam tips and common mistakes

Practice questions (graded-assignment style)

Once we know something about one variable, the distribution of another changes. The conditional PMF tells us exactly how: take the relevant slice of the joint and rescale it to total 1.

4.1Conditional PMF

Given an event A

The range of $(X \mid A)$ can differ from the range of $X$, depending on what $A$ restricts.

Given another random variable

Example: conditional PMFs from a joint PMF

• $f_{Y \mid X = 0}(0) = \frac{1/4}{3/8} = \frac{2}{3}$, $f_{Y \mid X = 0}(1) = \frac{1/8}{3/8} = \frac{1}{3}$ ($\frac{2}{3} + \frac{1}{3} = 1$ ✓: still a valid PMF).
• $f_{X \mid Y = 1}(0) = \frac{1/8}{1/2} = \frac{1}{4}$.

4.2Worked problems

Example: throw a die and toss coins

$X \sim \text{Uniform}\{1, \dots, 6\}$ and $(Y \mid X = t) \sim \text{Binomial}(t, \tfrac{1}{2})$: throw a die, then toss that many fair coins and let $Y$ be the number of heads. Then $f_{Y \mid X = t}(t^{\prime}) = \binom{t}{t^{\prime}}(\tfrac{1}{2})^t$, and joint $=$ marginal $\times$ conditional:

Example: a Poisson number of coin tosses

$N \sim \text{Poisson}(\lambda)$; given $N = n$, toss a fair coin $n$ times and let $X$ = number of Heads. Then

4.3Going deeper: conditioning as filtering and rescaling

• The recipe. To get $f_{Y \mid X = t}$, take the column (or row) of the joint where $X = t$, and divide every entry by that column total $f_X(t)$. The new entries add up to 1.
• Conditioning on an event. Keep only the outcomes in $A$ and divide by $P(A)$. The range of $(X \mid A)$ can be smaller than the range of $X$.
• Bayes with PMFs. $f_{X \mid Y = y}(x) = \frac{f_{Y \mid X = x}(y)\, f_X(x)}{f_Y(y)}$, where the denominator comes from total probability. This turns "given the cause, the effect" into "given the effect, the cause".
• Hierarchical experiments. When stage 2 depends on stage 1 (die, then that many coins), build the joint as marginal $\times$ conditional.
• Thinning (extra). If $N \sim \text{Poisson}(\lambda)$ and each of the $N$ items is kept independently with probability $p$, then the number kept is $\text{Poisson}(\lambda p)$.
• Undefined case. Conditioning on a value with probability 0 divides by 0, so it is undefined (the assignment convention reports 0).

4.4Exam tips and common mistakes

Practice questions (graded-assignment style)

When an experiment happens in stages, the distribution of the final quantity is a weighted mixture of the conditional distributions. This is the law of total probability applied to PMFs.

5.1Law of total probability with distributions

If we know the marginal of $Y$ and every conditional $(X \mid Y = y)$, we can rebuild the marginal of $X$. This is the law of total probability, written with PMFs:

Example: IPL powerplay over

• $(X \mid Y = 0) \sim \text{Uniform}\{6, \dots, 12\}$
• $(X \mid Y = 1) \sim \text{Uniform}\{2, \dots, 8\}$
• $(X \mid Y = 2) \sim \text{Uniform}\{0, \dots, 6\}$

Each conditional has 7 equally likely values, so each conditional PMF equals $\frac{1}{7}$ where it applies. Combining with $f_X(t) = \sum f_Y(y)\, f_{X \mid Y = y}(t)$:

• $f_X(0) = \frac{1}{16} \cdot \frac{1}{7} = \frac{1}{112}$ (only $Y = 2$ allows 0 runs)
• $f_X(2) = \frac{1}{8} \cdot \frac{1}{7} + \frac{1}{16} \cdot \frac{1}{7} = \frac{3}{112}$ ($Y = 1$ and $Y = 2$)
• $f_X(12) = \frac{13}{16} \cdot \frac{1}{7} = \frac{13}{112}$ (only $Y = 0$ allows 12 runs)

5.2Going deeper: a PMF as a weighted mixture

• Picture. $f_X$ is a weighted average of the conditional PMFs $f_{X \mid Y = y}$, with weights $f_Y(y)$. The weights add up to 1, so the result is again a PMF.
• Method. Draw a tree: first stage branches with probabilities $f_Y(y)$, then for each branch the chance of the value you want. Multiply along each path and add the paths that lead to that value.
• Support. A value is possible for $X$ if it is possible under at least one branch. The support of the mixture is the union of the conditional supports.
• Overlap adds. A value allowed by several branches collects a contribution from each (like $f_X(2)$ in the IPL example).
• Then reverse it. Once you have $f_X(t)$, Bayes gives the probability of a branch given the outcome: $P(Y = y \mid X = t) = \frac{f_Y(y) f_{X \mid Y = y}(t)}{f_X(t)}$.