Statistics for
Data Science 1

Weeks 1 to 3 described one variable at a time. Week 4 asks whether two variables are connected. The lecture opens with a review of the course so far.

1.1Where this fits and the three objectives

1.2The contingency table

1.3Lecture examples

Example 1: gender versus use of a smartphone

A market research firm wants to find out whether ownership of a smartphone is associated with the gender of a student: do more females own a smartphone than males, or is owning one independent of gender? A group of 100 college-going children were surveyed.

• There are 44 female and 56 male students.
• 76 students owned a smartphone and 24 did not.
• 34 female students and 42 male students owned a smartphone.

1. Identify the variables. Gender (Male, Female): 2 categories, nominal. Own a smartphone (Yes, No): 2 categories, nominal.
2. Fill in what is given. Row totals 44 and 56; column totals 24 (No) and 76 (Yes); the Yes cells are 34 and 42.
3. Find the missing cells by subtraction. Female No = 44 - 34 = 10; Male No = 56 - 42 = 14.
4. Check. Column No: 10 + 14 = 24; column Yes: 34 + 42 = 76; grand total 24 + 76 = 44 + 56 = 100.

Example 2: income versus use of a smartphone

The firm now asks whether ownership of a smartphone is associated with income. A group of 100 randomly picked individuals were surveyed.

• 20 High, 66 Medium and 14 Low income participants.
• 62 owned a smartphone and 38 did not.
• 18 High, 39 Medium and 5 Low income participants owned a smartphone.

1. Identify the variables. Income level (Low, Medium, High): 3 categories, ordinal. Own a smartphone (Yes, No): 2 categories, nominal.
2. Missing cells. High No = 20 - 18 = 2; Medium No = 66 - 39 = 27; Low No = 14 - 5 = 9.
3. Check. Column No: 2 + 27 + 9 = 38; column Yes: 18 + 39 + 5 = 62; 38 + 62 = 20 + 66 + 14 = 100.

1.4Building a contingency table in Google Sheets

1. Choose the columns of the variables for which you seek an association.
2. Go to Data and click the Pivot table option.
3. Click Create; this opens the pivot table editor.
4. Under Rows, click the first categorical variable.
5. Under Columns, click the second categorical variable.
6. Under Values, click either of the variables, then choose COUNTA under “Summarize by”.

1.5Going deeper: reading a contingency table

1. Every case sits in exactly one cell. That is why the cells add up to the grand total (here 100).
2. Totals are the one-variable tables. The row-total column is the frequency table of the row variable alone; the column-total row is the frequency table of the column variable alone. These two are called the marginal totals (an extra term, not from the lecture).
3. Subtraction is the shortcut. When a total and one cell are given, the other cell in that row is the difference. Always finish with the column check.
4. Which variable goes on the rows? Either works. In these notes the variable whose groups you want to compare (gender, income) goes down the rows.

Raw counts are hard to compare when the groups have different sizes. Two questions show why: What proportion of all participants own a smartphone? and What proportion of the female participants own a smartphone? The second question divides by the number of females, not by 100.

2.1Row relative frequencies: lecture examples

Example 3: gender and smartphone

1. Divide every cell by the total of its row (44 for Female, 56 for Male). The column-total row is divided by the grand total, 100.
2. Convert to percentages.

Example 4: income and smartphone

1. Divide by the row totals 20, 66 and 14 (and the column-total row by 100).
2. As percentages.

2.2Column relative frequencies: lecture examples

Now the questions turn around: What proportion of all participants are female? and What proportion of smartphone owners are female? The second divides by the number of owners.

Example 5: gender and smartphone

1. Divide by the column totals 24 (No) and 76 (Yes); the row-total column is divided by 100.
2. As percentages.

Example 6: income and smartphone

1. Divide by the column totals 38 and 62 (and the row totals by 100).
2. As percentages.

2.3Going deeper

1. Each row (or column) adds to 100%. In a row-percentage table every row sums to 100% (22.73% + 77.27%). In a column-percentage table every column does (41.67% + 58.33%). Use this as the check.
2. Pick the denominator from the question. “Of the females, how many own one?” divides by the females (row). “Of the owners, how many are female?” divides by the owners (column). The word after “of the” tells you what to divide by.
3. The two are not the same number. 77.27% of females own a smartphone, but 44.74% of owners are female. Mixing them up reverses the meaning.
4. Compare with the overall split. The overall row (24.00% No, 76.00% Yes) is the baseline each group is compared against in the next chapter.

To decide whether two categorical variables are associated, we use the row relative frequencies and the column relative frequencies of the previous chapter.

3.1Lecture examples

Example 7: gender and smartphone, associated?

1. Row relative frequencies. Female: 22.73% No, 77.27% Yes. Male: 25.00% No, 75.00% Yes. The two rows are nearly identical, and both are close to the overall split of 24.00% and 76.00%.
2. Column relative frequencies. Among non-owners 41.67% are female and 58.33% male; among owners 44.74% female and 55.26% male. These are close to the overall 44.00% and 56.00%.
3. Conclusion. Gender and smartphone ownership are not associated: knowing a student's gender tells us almost nothing about whether the student owns a smartphone.

Example 8: income and smartphone, associated?

1. Row relative frequencies. High: 10.00% No, 90.00% Yes. Medium: 40.91% No, 59.09% Yes. Low: 64.29% No, 35.71% Yes. The rows are clearly different from one another.
2. Column relative frequencies. Among owners 29.03% are High, 62.90% Medium and 8.06% Low; among non-owners the split is 5.26%, 71.05% and 23.68%. The columns also differ.
3. Conclusion. Income and smartphone ownership are associated. The higher the income level, the larger the share who own a smartphone.

3.2Going deeper

1. Two checks, one answer. Rows and columns tell the same story. If the row percentages are the same across rows, the column percentages are the same across columns too, so one of the two is enough.
2. “Close enough” is a judgement. 77.27% against 75.00% is a gap of about 2 points, small next to the 55-point gap between High (90%) and Low (35.71%). The lecture gives no numerical cut-off; compare the size of the gaps.
3. Association is not cause. Income and smartphone ownership go together, but the table alone does not say that income causes ownership. This remark is an extra, not from the lecture.
4. Ordinal order helps. Because income is ordinal we can say “the higher the income, the larger the share”, a direction. For a nominal variable we can only say the groups differ.

4.1Making one in Google Sheets

1. Select the data you want to include in the contingency table.
2. Click Insert, then Chart, and choose the stacked bar option.

4.2Lecture examples

Example 9: gender and smartphone

Draw the stacked bar chart and the 100% stacked bar chart for the gender and smartphone data.

1. Stacked bars of counts. One bar per gender. The bar height is the number of students (44 and 56); it is split into No (10, 14) and Yes (34, 42).
2. 100% stacked bars. Rescale every bar to the same height, 100%, so each segment is the row percentage from the table.

• Read the chart. The Yes segment is almost the same fraction of both 100% bars, which matches the earlier conclusion that gender and ownership are not associated.

Example 10: income and smartphone

Draw the stacked bar chart and the 100% stacked bar chart for the income and smartphone data.

1. Stacked bars of counts. One bar per income level (14, 66, 20 participants), split into No and Yes.
2. 100% stacked bars. Each bar rescaled to 100%.

• Read the chart. The Yes segment grows from 35.71% (Low) to 59.09% (Medium) to 90.00% (High), so the bars differ: income and ownership are associated.

4.3Going deeper: which chart to use

5.1Which variable goes on which axis

5.2Lecture examples

Example 11: age and height of children

The ages and heights of five children are given. Draw the scatter plot.

1. Age explains height, so age is the explanatory variable (x) and height the response variable (y).
2. Plot the five points (1, 75), (2, 85), (3, 94), (4, 101), (5, 108).
3. The points rise from left to right: taller heights go with greater ages.

Example 12: prices of homes

A real estate agent collected the prices of different sizes of homes. He wanted to see the relationship between the price of a home and its size; in particular, whether prices increase linearly with size or in some other way. He collected data on 15 homes: the size in 1000 square feet and the price in lakhs of rupees.

1. Size is the explanatory variable (x) and price the response variable (y).
2. Plot the 15 pairs. Two homes of size 2.3 have prices 83 and 112, and two of size 2.5 have prices 93 and 98.
3. The cloud climbs from lower left to upper right, so larger homes tend to cost more. The points are scattered around the climb, not exactly on a line.

5.3The visual test for association

5.4Scatter plot in Google Sheets

1. Highlight the data you want to plot.
2. Click Insert, then Chart, and choose the scatter chart.
3. Under the X-axis tab, choose your explanatory variable.
4. Under the Series tab, choose the response variable.
5. Label the title of the chart and the axes appropriately.

5.5Going deeper

1. Which variable is x? Ask: which one would I use to explain or predict the other? Age explains height; size explains price. Swapping them gives a different picture.
2. Each point is one case. Home 7 is the point (2.3, 83); home 8 is (2.3, 112). Two points can share an x-value.
3. Pattern, not exactness. The homes rise with size but do not lie on a line: the price also depends on other things. Chapters 6 to 9 give ways to describe and measure that.
4. Axes need not start at zero. The home plot starts at 0.5 on x so the points fill the picture; check the axis labels before reading.

When describing the association between two variables in a scatter plot, four key questions need to be answered:

6.1Direction

An upward pattern means larger values of x go with larger values of y; a downward pattern means larger values of x go with smaller values of y.

6.2Curvature

Look at whether the points follow a straight line or bend. The lecture shows four plots: linear patterns trending up and down, and two that curve, one bending upward and one levelling off.

6.3Variation

6.4Outliers

6.5Going deeper: a sample description

For the 15 homes (Chapter 5): Direction: upward, larger homes tend to cost more. Curvature: roughly linear. Variation: moderate, the points are scattered around the climb. Outliers: none stands far from the cloud. (This is a reading of the plot, not a number from the lecture.)

How do we measure the strength of the association between two variables? The lecture introduces two measures: covariance and correlation.

7.1Deviations from the means

The lecture builds covariance from two columns of deviations: $x_i - \bar{x}$ and $y_i - \bar{y}$.

Example 13: age and height, the deviations

1. Means. $\bar{x} = \frac{1+2+3+4+5}{5} = 3$ and $\bar{y} = \frac{75+85+94+101+108}{5} = \frac{463}{5} = 92.6$.
2. Deviations. Subtract the means:

Example 14: age and price of a car, the deviations

1. Means. $\bar{x} = 3$ and $\bar{y} = \frac{6+5+4+3+2}{5} = 4$.
2. Deviations.

7.2Multiplying the deviations

Example 15: age and height, products of deviations

Step 3. Multiply $(x_i - \bar{x})(y_i - \bar{y})$ for each person and add the products.

Every product is zero or positive: when age is above average so is height, and when age is below average so is height.

Example 16: age and price of a car, products of deviations

Every product is zero or negative: when age is above average the price is below average, and the other way round.

7.3Definition of covariance

Example 17: covariance of age and height

1. From Section 7.2 the sum of the products is 82.
2. There are 5 observations.

Example 18: covariance of age and price of a car

1. The sum of the products is -10.
2. There are 5 observations.

7.4Going deeper

1. The sign tells the direction. Positive covariance (16.4, 20.5): upward pattern. Negative (-2, -2.5): downward pattern. Near zero: no linear pattern.
2. The size depends on units. Measuring height in metres instead of cm would divide the covariance by 100 without changing the pattern. So 20.5 versus -2.5 cannot be compared directly. This is an extra remark that follows from the definition.
3. Same recipe as variance. The variance divides $\sum (x_i - \bar{x})^2$ by $n-1$; the covariance replaces one of the squares by the other variable's deviation. So the covariance of x with itself is its variance (extra remark).
4. Population or sample? Use $N$ for the whole population and $n - 1$ for a sample, as for the variance.

A more easily interpreted measure of linear association between two numerical variables is correlation. It is derived from covariance.

8.1Lecture examples

Example 19: correlation of age and height

1. Square the deviations and add them:

1. Standard deviations. $s_x = \sqrt{\frac{10}{4}} = 1.58$ and $s_y = \sqrt{\frac{677.2}{4}} = 13.01$.
2. Correlation.

Example 20: correlation of age and price of a car

1. Square the deviations and add them:

1. Standard deviations. $s_x = s_y = \sqrt{\frac{10}{4}} = 1.58$.
2. Correlation.

8.2Correlation in Google Sheets

The function CORREL(series1, series2) returns the value of the correlation. If the x-variable (series1) is in cells A2:A6 and the y-variable (series2) in cells B2:B6, then CORREL(A2:A6,B2:B6) returns the Pearson correlation coefficient.

8.3Going deeper: how to read r

1. Sign and size. The sign gives the direction; the distance from 0 gives the strength of the linear association.
2. Only linear. A curved pattern can have $r$ near 0 and still be a strong association. That is why Chapter 6 asks about curvature before computing anything.
3. Units do not matter. Because $r$ has no units, measuring height in metres or in cm gives the same $r$ (extra remark).
4. Order does not matter. The formula is symmetric in x and y, so $r$ is the same if you swap them (extra remark).
5. Never beyond $\pm 1$. If you get 1.0016 or -1.2, there is a calculation or rounding mistake.

The strength of linear association was measured with covariance and correlation. The linear association can also be described with the equation of a line. This chapter summarises a scatter plot with a line and introduces $R^2$.

9.1Equation of the line in Google Sheets

1. Open the scatter plot.
2. Under the Customize tab, click on Series.
3. Click on Trendline.
4. Under the Label tab, choose Use equation and click the Show R² button.

9.2Lecture examples

Example 21: size versus price of homes

For the 15 homes of Chapter 5, find the equation of the line, $R^2$ and $r$.

1. Fit. Put size on the x-axis, price on the y-axis, and add the trendline with its equation and $R^2$.
2. Reading the line. The slope 30.5 says that the line rises by 30.5 lakh for every extra 1000 square feet.
3. Check. $\sqrt{0.646} = 0.804$ and $0.804^2 \approx 0.647 = R^2$.

Example 22: age versus price of cars

For a data set of the age and price of cars, find the equation of the line, $R^2$ and $r$.

1. Fit the trendline in the scatter plot of price against age.
2. Reading the line. The slope is negative: the line falls by 0.694 lakh for each extra year of age. The points lie close to the line.
3. Check. $(-0.9247)^2 = 0.855 = R^2$.

Example 23: size versus price of homes, a weak line

For a second data set of the size and price of homes, find the equation of the line, $R^2$ and $r$.

1. Fit the trendline.
2. Reading the line. The line is almost flat (a slope of only 7.77) and the points are scattered widely around it.
3. Check. $\sqrt{0.022} \approx 0.149 = r$.

9.3All the lines side by side

9.4Going deeper

1. Slope and intercept. In Price = 30.5 × Size + 36 the slope 30.5 is the change in price for one more unit of size; the intercept 36 is the line's value at size 0 (here just where the line starts, not a real home).
2. $R^2 = r^2$. For a fitted straight line, $R^2$ is the square of $r$, which is the check used in every example. So $R^2$ is between 0 and 1, and the sign of $r$ comes from the slope.
3. What $R^2$ says. 0.647 means roughly 65% of the ups and downs in price go along with size, and 0.022 means almost none do. This reading is the usual one, an extra to the lecture, which only says $R^2$ goes with the line.
4. Do not predict far outside the data. The homes in the data are 0.8 to 3.4 thousand square feet; the line says little about a 10 000 square foot home (extra remark).

The last lecture asks how a categorical variable is associated with a numerical variable, when the categorical variable has just two categories (it is dichotomous).

10.1Lecture example: gender versus marks

Example 24: gender versus marks

A teacher wants to know whether female students performed better than male students in her class. She collected the marks (out of 100) of twenty students in the subject.

1. Code the categorical variable as 0 and 1. The lecture's scatter plot puts the females at 0 and the males at 1 (the marks at 0 range from 65 to 91, the marks at 1 from 69 to 94).
2. Plot marks (numerical) against the gender code.

1. Group the same marks by gender to compare them.

Female marks: 12 values, sum 944, so the mean is $944/12 = 78.67$. Male marks: 8 values, sum 658, mean $658/8 = 82.25$.

10.2Point bi-serial correlation coefficient

Let $X$ be a numerical variable and $Y$ a categorical variable with two categories (a dichotomous variable). To calculate the point bi-serial correlation between them:

1. Group the data into two sets based on the value of the dichotomous variable $Y$; assume the value of $Y$ is either 0 or 1.
2. Means. Calculate the mean values of the two groups: $\bar{Y}_0$ and $\bar{Y}_1$ for the groups with $Y = 0$ and $Y = 1$.
3. Proportions and spread. Let $p_0$ and $p_1$ be the proportions of observations with $Y = 0$ and $Y = 1$, and $s_X$ the standard deviation of the numerical variable $X$.

Example 25: point bi-serial correlation of gender and marks

1. Groups. $Y = 0$ for the 12 females and $Y = 1$ for the 8 males (as in the lecture's plot).
2. Group means. $\bar{Y}_0 = \frac{944}{12} = 78.67$ and $\bar{Y}_1 = \frac{658}{8} = 82.25$.
3. Proportions. $p_0 = \frac{12}{20} = 0.6$ and $p_1 = \frac{8}{20} = 0.4$, so $\sqrt{p_0 p_1} = \sqrt{0.24} = 0.490$.
4. Standard deviation of all 20 marks. The marks add up to 1602, so the mean is 80.1, and the sample standard deviation is 9.57.
5. Substitute.

• Using the population standard deviation (9.33, dividing by 20) instead gives -0.19; either way the size is about 0.18 to 0.19.
• Interpret. The size 0.18 is small, so the association between gender and marks in this class is weak. The negative sign only reflects the coding: the group coded 1 (males) has the higher mean. Swapping the codes would flip the sign and leave the size unchanged.

10.3Going deeper

1. The formula in words. $r_{pb}$ is the gap between the two group means, measured in standard deviations of all the marks, and then shrunk by $\sqrt{p_0 p_1}$. A big gap compared with the spread gives a big $|r_{pb}|$.
2. Which group is 0 is your choice. The sign changes, the size does not. Always say which group is coded 1 when you state the sign.
3. It is a Pearson correlation. With a 0/1 variable, the ordinary correlation formula of Chapter 8 gives the same number (extra remark; the lecture gives only the formula above).
4. Compare means first. The quickest check is the two group means: 78.67 against 82.25, only about 3.6 marks apart.

Everything in Week 4 on one sheet, for a last-minute revision.

Which tool for which pair of variables

Formulas

Google Sheets steps

Values computed in the lectures

Topic by topic