Statistics for
Data Science 1

To describe numerical data we first group the observations into classes (also called categories or bins) and then treat the classes as the distinct values of qualitative data. Once the data are grouped, the frequency and relative-frequency tables are built exactly as in Week 2.

1.1Where Week 3 starts

The lecture begins with a review of Weeks 1 and 2, and then splits variables by type. Weeks 1 and 2 dealt with categorical variables; this week is about numerical ones.

1.2Organising numerical data

Raw numbers are hard to read. We group them into classes, and then every class plays the role that a category played in Week 2.

1.3Discrete data: single-value classes

If the data set contains only a relatively small number of distinct values, it is convenient to represent it in a frequency table. Each class represents a distinct value (a single value) along with its frequency of occurrence.

Lecture example: number of people in a household

The responses of 15 individuals to the number of people in their household: 2, 1, 3, 4, 5, 2, 3, 3, 3, 4, 4, 1, 2, 3, 4

1. List the distinct values: 1, 2, 3, 4 and 5.
2. Tally each response in the row of its value.
3. Count the tallies and divide each frequency by the total, 15.

Check: $2 + 3 + 5 + 4 + 1 = 15$ and the relative frequencies add to 1 (to three decimals: $0.133 + 0.2 + 0.333 + 0.267 + 0.067 = 1$).

1.4Continuous data: class intervals

For continuous data (or discrete data with many distinct values) the values are organised into class intervals, so that the table becomes understandable. A few guidelines have to be followed.

Guidelines for forming classes

1. Number of classes. The appropriate number is a subjective choice; the rule of thumb is to have between 5 and 20 classes.
2. No overlaps, no gaps. Each observation should belong to some class and no observation should belong to more than one class.
3. Equal lengths. It is common, although not essential, to choose class intervals of equal length.

1.5Lecture example: marks of 50 students

The marks of 50 students in a course are listed below. Group them into class intervals of width 10 and build the frequency and relative-frequency table.

68, 79, 38, 68, 35, 70, 61, 47, 58, 66, 60, 45, 61, 60, 59, 45,
39, 80, 59, 62, 49, 76, 54, 60, 53, 55, 62, 58, 67, 55, 86, 56,
63, 64, 67, 50, 51, 78, 56, 62, 57, 69, 58, 52, 42, 66, 42, 56, 58

1. The smallest mark is 35 and the largest 86, so the classes 30-40, 40-50, 50-60, 60-70, 70-80, 80-90 (six classes, width 10) cover everything. The left-end rule decides the boundary values.
2. Put each mark in its class and tally.
3. Count and divide by 50.

Check: $3 + 6 + 18 + 17 + 4 + 2 = 50$ and $0.06 + 0.12 + 0.36 + 0.34 + 0.08 + 0.04 = 1$.

1.6Going deeper: choosing the classes

1. Start from the extremes. Find the smallest and largest value; the classes must cover both. Here 35 and 86 sit inside 30-90.
2. A handy width. A common way to pick the width is (largest - smallest) ÷ (number of classes wanted), rounded up to a convenient number like 5 or 10. Here (86 - 35) ÷ 6 = 8.5, rounded up to 10. This is a practical tip, not a rule from the lecture.
3. Why the left-end rule matters. Without it a mark such as 60 could be counted twice (in 50-60 and in 60-70). The rule gives every mark exactly one class.
4. Grouping loses detail. Once marks are put in a class we only know the class, not the exact mark. That is why the mean from a grouped table (Chapter 4) is only approximate.

A histogram is the graph of a frequency (or relative-frequency) table of grouped numerical data: one vertical bar per class, with the classes placed on a number line.

2.1Steps to construct a histogram

1. Frequency distribution. Obtain a frequency (relative-frequency) distribution of the data.
2. Axes. Draw a horizontal axis on which to place the classes and a vertical axis on which to display the frequencies (relative frequencies).
3. Bars. For each class, construct a vertical bar whose height equals the frequency (relative frequency) of that class.
4. Labels. Label the bars with the classes, the horizontal axis with the name of the variable, and the vertical axis with “Frequency” (“Relative frequency”).

2.2Lecture example: histogram of the 50 marks

Use the frequency table from Chapter 1.

The lecture gives axes with the horizontal scale 30 to 100 and the vertical scale 1 to 18, ready for the bars. Draw a bar over each class with height 3, 6, 18, 17, 4, 2.

2.3Histogram versus bar chart

2.4Going deeper: relative-frequency histogram and shape

If the bar heights are the relative frequencies instead of the frequencies, the shape is exactly the same, only the vertical scale changes. This is the same idea as in Week 2: relative frequencies let us compare data sets of different sizes.

1. Where is the peak? The tallest bars show where most observations lie. Here, between 50 and 70.
2. How spread out? The bars run from 30 to 90, so the marks are spread over about 60 points.
3. Different class widths change the picture. Very narrow classes give a jagged histogram; very wide classes hide the pattern. That is why the 5 to 20 classes rule of thumb exists.

For example, if the data are all two-digit numbers, we can let the stem of a value be the tens digit and the leaf be the ones digit.

3.1Steps to construct a stemplot

1. Split. Think of each observation as a stem (all but the rightmost digit) and a leaf (the rightmost digit).
2. Stems. Write the stems from smallest to largest in a vertical column to the left of a vertical rule.
3. Leaves. Write each leaf to the right of the vertical rule in the row that contains the appropriate stem.
4. Sort. Arrange the leaves in each row in ascending order.

3.2Lecture example: ages of 11 patients

The ages, to the nearest year, of 11 patients admitted in a hospital are given. Draw a stem-and-leaf plot for this data set. 15, 22, 29, 36, 31, 23, 45, 10, 25, 28, 48

1. Split each age: 15 = stem 1, leaf 5; 22 = stem 2, leaf 2; 29 = stem 2, leaf 9; and so on.
2. The stems run from 1 (ages 10 to 19) to 4 (ages 40 to 49): 1, 2, 3, 4.
3. Write the leaves in the order the data are given.
4. Arrange the leaves in each row in ascending order.

Check: the number of leaves is $2 + 5 + 2 + 2 = 11$, the number of patients.

3.3Going deeper: stemplot versus histogram

• Keeps the data. A histogram shows only how many values fall in a class; a stemplot also keeps every value. From the plot above you can still read the oldest patient (48) and the youngest (10).
• Sorted for free. With the leaves in order, the data are already sorted, which makes the median (Chapter 5) and percentiles (Chapter 9) easy to find.
• Always give the key. Write “2 | 2 means 22” so the reader knows what the stem and leaf stand for.
• Three-digit data. For 135, the stem is 13 (all but the rightmost digit) and the leaf is 5.

The objective is to develop descriptive measures: quantities whose values are determined by the data, used to summarise a data set. We start with the most common one, the mean.

4.1Lecture examples: the mean of small data sets

Example 1. Find the mean of 2, 12, 5, 7, 6, 7, 3.

Example 2: one large value changes the mean. Find the mean of (a) 2, 105, 5, 7, 6, 7, 3 and (b) 2, 105, 5, 7, 6, 3.

Example 3: marks of ten students. Find the sample mean of 68, 79, 38, 68, 35, 70, 61, 47, 58, 66

4.2Mean for grouped data

Discrete single-value data. If the value $x_i$ occurs $f_i$ times, the sum of the data is $f_1x_1 + f_2x_2 + \cdots + f_nx_n$, so

Example 4: household sizes. Find the mean number of people in a household for the 15 responses from Chapter 1.

Continuous data. Every observation in a class is replaced by the mid point $m_i$ of the class, so

Example 5: marks of 50 students (grouped continuous data). Find the mean using the grouped table of Chapter 1.

4.3Manipulating the data

Example 6: adding 5 marks. The ten marks above have mean 59. The teacher adds 5 marks to every student. New data: 73, 84, 43, 73, 40, 75, 66, 52, 63, 71.

The mean moves up by exactly 5.

Example 7: scaling down by 40%. The teacher instead multiplies each mark by 0.4. New data: 27.2, 31.6, 15.2, 27.2, 14, 28, 24.4, 18.8, 23.2, 26.4.

The mean is multiplied by 0.4.

4.4Going deeper

1. The mean is the balance point. Imagine the dots as equal weights on a ruler: the mean is where the ruler balances. That is why one far-away value drags it.
2. Deviations add to zero. The deviations $x_i - \bar{x}$ always add to 0 (you will see this in Chapter 8 for the ten marks).
3. The mean need not be a data value. The mean household size is 2.93 people, and no household has 2.93 people.
4. Both rules together. If every value becomes $c\,x_i + d$, the mean becomes $c\,\bar{x} + d$. Apply the multiplication and the addition one after the other.
5. Grouped mean is only approximate. The exact mean needs the original marks; the grouped table only knows the class of each mark.

5.1Steps to obtain the median

Arrange the data in increasing order. Let $n$ be the total number of observations.

5.2Lecture examples

Example 1: odd number of observations. Find the median of 2, 12, 5, 7, 6, 7, 3.

• Arrange in increasing order: 2, 3, 5, 6, 7, 7, 12.
• $n = 7$ is odd, so the median is the $\frac{7+1}{2} = 4$th observation, which is 6.

Example 2: odd n with a large value. Find the median of 2, 105, 5, 7, 6, 7, 3.

• Ordered: 2, 3, 5, 6, 7, 7, 105.
• $n = 7$ is odd, the 4th observation is 6, the same as before even though 12 became 105.

Example 3: even number of observations. Find the median of 2, 105, 5, 7, 6, 3.

• Ordered: 2, 3, 5, 6, 7, 105.
• $n = 6$ is even, so take the average of the 3rd and 4th observations: $\frac{5 + 6}{2} =$ 5.5.

5.3Mean versus median: outliers

Example 4: sensitivity to outliers. Compare the mean and median of (1) 2, 12, 5, 7, 6, 7, 3 and (2) 2, 117, 5, 7, 6, 7, 3.

5.4Manipulating the data

Example 5: adding 5 marks. Marks: 68, 79, 38, 68, 35, 70, 61, 47, 58, 66. Find the median, then the median after adding 5 to each mark.

• Ascending: 35, 38, 47, 58, 61, 66, 68, 68, 70, 79. $n = 10$ is even, so the median is the average of the 5th and 6th: $\frac{61+66}{2} =$ 63.5.
• After adding 5, ascending: 40, 43, 52, 63, 66, 71, 73, 73, 75, 84. Median $\frac{66+71}{2} =$ 68.5 $= 63.5 + 5$.

Example 6: scaling down by 40%. The marks are multiplied by 0.4. Find the new median.

• New data ascending: 14, 15.2, 18.8, 23.2, 24.4, 26.4, 27.2, 27.2, 28, 31.6.
• The 5th and 6th values are 24.4 and 26.4, so the median is $\frac{24.4 + 26.4}{2} =$ 25.4 $= 0.4 \times 63.5$.

5.5Going deeper

1. Sort first. The median only makes sense on the ordered list. If the data are in a stem-and-leaf plot with sorted leaves, they are already sorted.
2. Positions, not values. $\frac{n+1}{2}$ is a position. For $n = 7$ it is 4, so look at the 4th value. For $n = 10$ it is 5.5, meaning halfway between the 5th and 6th values.
3. When to prefer the median. When a few extreme values would drag the mean (incomes, house prices), the median describes the typical case better.
4. The median is the 50th percentile. Chapter 9 puts this in a bigger picture.
5. Both rules together. If every value becomes $c\,x_i + d$, the median becomes $c\cdot\text{median} + d$.

6.1Steps to obtain the mode

1. If no value occurs more than once, then the data set has no mode.
2. Else, the value that occurs with the greatest frequency is a mode of the data set.

6.2Lecture examples

Example 1: mode of three data sets.

6.3Manipulating the data

Example 2: adding 5 marks. Marks: 68, 79, 38, 68, 35, 70, 61, 47, 58, 66. The mode is 68 (it occurs twice). Find the mode after adding 5 marks to each.

The data are 40, 43, 52, 63, 66, 71, 73, 73, 75, 84. The value 73 occurs twice, so the mode is 73 $= 68 + 5$.

Example 3: scaling down by 40%. Find the mode after multiplying each mark by 0.4.

The data are 27.2, 31.6, 15.2, 27.2, 14, 28, 24.4, 18.8, 23.2, 26.4. The value 27.2 occurs twice, so the mode is 27.2 $= 0.4 \times 68$.

6.4Going deeper

1. Mode can be missing or repeated. No value repeated: no mode. Two values tied for the top: two modes (bimodal), as in Week 2.
2. From a frequency table: the mode is the value with the largest frequency. For grouped continuous data we speak of the modal class, the class with the tallest histogram bar (here 50-60 in the marks table). This is an extension, not a slide from the lecture.
3. Compare the three centres. Mean uses every value (sensitive to outliers), median uses the middle position (not sensitive), mode uses only the counts (works even for categories).
4. The mode is a data value. Unlike the mean (2.93 people), the mode always is one of the observations.

Two data sets can have the same centre and still be different. Compare dataset 1: 3, 3, 3, 3, 3 and dataset 2: 1, 2, 3, 4, 5.

7.1Range

Example 1: range of the two data sets.

The range separates the two data sets, which the mean and median could not.

7.2Range is sensitive to outliers

Example 2: one value changes the range. Compare dataset 1 (1, 2, 3, 4, 5) and dataset 2 (1, 2, 3, 4, 15).

7.3Going deeper

1. Range uses two observations. The values in between can be anything; the range does not see them. Chapter 8 introduces measures that use every observation.
2. Range zero means no variation. Range 0 happens exactly when all the values are equal.
3. Units. The range is in the same units as the data (marks, kilograms, years).
4. Adding a constant to every value moves Max and Min by the same amount, so the range is unchanged. The lecture does not cover this, but it follows from the definition.

In contrast to the range, the variance takes into account all the observations. One way of measuring the variability of a data set is to consider the deviations of the data values from a central value, the mean.

• The numerator is the sum of squared deviations of every observation from its mean.
• The denominator for the population variance is $N$, the total number of observations.
• The denominator for the sample variance is $(n - 1)$. The reason will be clear in forthcoming courses on statistics.

8.1Lecture example: marks of ten students

The marks of ten students are given and the mean is 59. Find the population variance and the sample variance. 68, 79, 38, 68, 35, 70, 61, 47, 58, 66

Step 1. Subtract the mean from each mark (the deviation) and square it.

Step 2. Sum of squared deviations = 1898.

Step 3. Divide:

8.2Standard deviation

Example: standard deviation of the marks.

8.3Manipulating the data

Example: adding 5 marks. New data: 73, 84, 43, 73, 40, 75, 66, 52, 63, 71. Every deviation from the new mean (64) is the same as before, so the sum of squared deviations is still 1898.

In general, adding a constant does not change the variability of a data set.

Example: scaling down by 40%. New data: 27.2, 31.6, 15.2, 27.2, 14, 28, 24.4, 18.8, 23.2, 26.4 with mean 23.6. Sum of squared deviations = 303.68.

Check: $0.4^2 \times 210.89 = 0.16 \times 210.89 = 33.74$. For the standard deviation: $0.4 \times 14.522 = 5.808$.

8.4Going deeper

1. Why square? The plain deviations add to 0. Squaring removes the signs, and it also punishes big deviations more than small ones.
2. Why a standard deviation too? The variance is in square units (marks²); taking the square root brings it back to marks, so it can be compared directly with the data.
3. Sample versus population. Divide by $N$ when you have the whole population, by $n - 1$ when you have a sample. For the ten marks: 189.8 (population) and 210.89 (sample).
4. A shortcut for calculation. The sum of squared deviations also equals $\sum x_i^2 - n\bar{x}^2$. For the ten marks: $36708 - 10 \times 59^2 = 36708 - 34810 = 1898$. This is an extra, not part of the lecture.
5. Zero variance. Variance is 0 only when every value equals the mean (like 3, 3, 3, 3, 3).
6. Negative multiplier. The variance is multiplied by $c^2$, which is never negative; the standard deviation is multiplied by $|c|$. The lecture uses positive $c$.

9.1Computing a percentile: the n p rule

To find the sample 100p percentile of a data set of size $n$:

1. Arrange the data in increasing order.
2. If $np$ is not an integer, find the smallest integer greater than $np$. The data value in that position is the sample 100p percentile.
3. If $np$ is an integer, the average of the values in positions $np$ and $np + 1$ is the sample 100p percentile.

9.2Lecture example: percentiles of the ten marks

Let $n = 10$. Arrange the marks in ascending order: 35, 38, 47, 58, 61, 66, 68, 68, 70, 79. Find the 10th, 25th, 50th, 75th and 100th percentiles.

9.3Percentiles in Google Sheets

For example, if the data are in cells A1:A10 and we want the 90th percentile, enter PERCENTILE(A1:A10,0.9) in a blank cell.

1. Paste the data set in a column.
2. In a blank cell enter PERCENTILE(data, percentile), where data is the range of the data and percentile is the decimal form of the desired percentile.

The algorithm Google Sheets uses

1. Arrange the data in increasing order and write $x[i]$ for the i-th ordered value.
2. Find the rank: $\text{rank} = \text{percentile} \times (n - 1) + 1$
3. Split the rank into its integer part and fractional part.
4. Find the ordered value $x[i]$ for the integer part.
5. Percentile $= x[i] + \text{fractional part} \times (x[i+1] - x[i])$.

Example: the 25th percentile by the rank method. For the ten ordered marks 35, 38, 47, 58, 61, 66, 68, 68, 70, 79, find the 25th percentile.

• $\text{rank} = 0.25 \times (10 - 1) + 1 = 3.25$: integer part 3, fractional part 0.25.
• $x[3] = 47$, $x[4] = 58$.
• $47 + 0.25 \times (58 - 47) = 47 + 0.25 \times 11 = 47 + 2.75 = 49.75$.

The 10th percentile row is worked out here from the same two rules; the lecture states only the 25th percentile difference. Always use the method the question names.

9.4Going deeper

1. Percentile means position, not marks. Being at the 75th percentile means that about 75% of the data are at or below your value.
2. Why two methods? With small data sets there is no value that cuts exactly 25% off, so different rules choose differently. For large data sets the answers are almost the same.
3. Sorted data first. Both methods need the ordered list. A sorted stemplot gives it for free.
4. The median is a special case. For $p = 0.5$ the n p rule is exactly the median rule of Chapter 5 ($np = n/2$).

In other words, the quartiles break up a data set into four parts, with about 25 percent of the data values being less than the first (lower) quartile, about 25 percent between the first and second quartiles, about 25 percent between the second and third (upper) quartiles, and about 25 percent larger than the third quartile.

10.1The five-number summary

10.2The interquartile range

Lecture example: IQR of the ten marks. For the ten marks (ordered: 35, 38, 47, 58, 61, 66, 68, 68, 70, 79) find the interquartile range.

• First quartile (25th percentile): $Q_1 = 49.75$.
• Third quartile (75th percentile): $Q_3 = 68$.

The five-number summary of these marks is Min = 35, Q1 = 49.75, Q2 = 63.5, Q3 = 68, Max = 79.

10.3Going deeper

1. IQR versus range. The range of the ten marks is $79 - 35 = 44$; the IQR is 18.25. The IQR looks only at the middle half, so one extreme value cannot stretch it the way it stretches the range.
2. The middle 50%. About half of the data lie between Q1 and Q3, and the IQR is the width of that stretch.
3. Which method for Q1 and Q3? The lecture example uses the Google Sheets values. With the n p rule the same data would give Q1 = 47 and so IQR = 68 - 47 = 21. Use the method the question names.
4. Centre and spread together. Median with IQR is a good pair for skewed data; mean with standard deviation is the other usual pair.

Everything in Week 3 on one sheet, for a last-minute revision.

Effect of a constant on each measure

The lecture shows the effect of constants on the mean, median, mode, variance and standard deviation. It does not cover range or IQR, so they are not in the table.

Formulas

Which graph or table for which data

Topic by topic