Statistics for
Data Science 1

Week 2 is about describing one categorical variable. Everything starts with a frequency table: a list of the categories and how many times each one occurs.

1.1Where this fits

Lecture 2.1 starts with a review of Week 1. Frequency tables are the first tool for one categorical variable, so three ideas from Week 1 are the background.

1.2Steps to build a frequency distribution

1. List the distinct values. Write each different category once, in the first column.
2. Tally every observation. Go through the data one value at a time and put a tally mark in that value's row.
3. Count the tallies. Turn each row of tally marks into a number. This is the frequency.

1.3Lecture examples: four data sets

The lecture builds a frequency table for four lists of categories. Read each list from left to right; every observation adds one tally to its row. After counting, the frequencies must add up to the number of observations.

Data set 1 (10 observations): A, A, B, C, A, D, A, B, D, C

Check: $4 + 2 + 2 + 2 = 10$, the number of observations.

All four data sets together

• Data set 2: A, A, B, C, A, D, A, B, D, C, A, B, C, D, A
• Data set 3: A, B, B, C, A, D, B, B, D, C, A, B, C, D, B
• Data set 4: A, A, B, C, A, D, A, B, D, C, A, B, C, D, A, C, D, D

1.4Frequency table in a Google Sheet

1. Select (highlight) the cells that have the data you want to summarise.
2. In the formatting bar click the Data option.
3. Go to Pivot table and create it in a new sheet.
4. In the Pivot table editor, first add Rows and then add Values.

1.5Going deeper: reading and checking a frequency table

1. The total is your safety check. The frequencies must add up to the number of observations. If they do not, you skipped or double-counted something.
2. Count from the list, not from the table. Cross out each observation as you tally it, so none is counted twice.
3. Order of the rows does not matter for a nominal variable. Write the categories in any sensible order (here A to D).
4. Frequency is a count, so it is a whole number: 0, 1, 2, ... It is never negative and never a fraction. Fractions come in the next topic.

Practice questions (graded-assignment style)

Raw counts cannot compare two data sets of different sizes. Dividing each count by its own total gives a proportion, and proportions can be compared.

2.1Steps to construct it

1. Obtain a frequency distribution of the data.
2. Divide each frequency by the total number of observations.

A relative frequency is always between 0 and 1, and the relative frequencies of all categories add to 1.

2.2Why use relative frequency?

2.3Lecture examples: relative frequencies

The lecture leaves the last column blank for data sets 3 and 4, so those values are worked out here from the frequencies.

• Data set 1: $0.4 + 0.2 + 0.2 + 0.2 = 1$. Data set 2 gives the same relative frequencies although the counts differ.
• Data set 3: $0.2 + 0.4 + 0.2 + 0.2 = 1$. B is the largest share, 40 percent.
• Data set 4: $0.333 + 0.167 + 0.222 + 0.278 = 1.000$. When a division does not end, the values are rounded to three decimals, so the rounded column can differ from 1 in the last digit (see Chapter 6, round-off errors).

2.4Going deeper: frequency versus relative frequency

1. Use frequency when you want to know how many. Use relative frequency when you want to compare shares or compare two data sets.
2. Percent is relative frequency times 100. A relative frequency of 0.4 is 40 percent.
3. To go back, multiply by the total: $\text{frequency} = \text{relative frequency} \times n$. For $n = 40$ and relative frequency 0.25 the frequency is 10.

Practice questions (graded-assignment style)

The two most common displays of a categorical variable are a bar chart and a pie chart. Both describe the variable by displaying its frequency table.

3.1Steps to construct a pie chart

1. Obtain a relative-frequency distribution of the data.
2. Divide a circle into pieces proportional to the relative frequencies.
3. Label the slices with the distinct values and their relative frequencies.

3.2Finding the angle of each slice

Use a protractor and the fact that a full circle has $360^\circ$. Each slice gets an angle equal to its relative frequency times $360^\circ$.

For the first slice of the lecture example the relative frequency is 0.4, so the angle is $0.4 \times 360^\circ = 144^\circ$.

3.3Lecture examples: slice angles

Data set 1: A, A, B, C, A, D, A, B, D, C

Check: $144 + 72 + 72 + 72 = 360$, the full circle.

Data set 2: same slices

The frequencies are 6, 3, 3, 3 with total 15, so the relative frequencies are again 0.4, 0.2, 0.2, 0.2 and the angles are again $144^\circ, 72^\circ, 72^\circ, 72^\circ$. The pie chart looks identical: a pie chart depends only on the relative frequencies, not on the number of observations.

3.4Pie chart in a Google Sheet

1. Select (highlight) the cells that have the data you want to visualise.
2. Click the Insert chart option in the toolbar.
3. Change the visualisation type in the Chart editor.
4. Set the Chart type to Pie chart.

3.5Going deeper: when to use a pie chart

1. A pie chart shows parts of a whole. It is a good way to show that one category makes up more than half of the total.
2. Angles always add to 360°, and relative frequencies always add to 1. Use whichever sum is easier as a check.
3. Too many thin slices are hard to read. Chapter 6 shows how to group small categories.

Practice questions (graded-assignment style)

4.1Steps to construct a bar chart

1. Obtain a frequency or relative-frequency distribution of the data.
2. Draw a horizontal axis on which to place the bars and a vertical axis to display the frequencies or relative frequencies.
3. For each distinct value, construct a vertical bar whose height equals the frequency or relative frequency of that value.
4. Label the bars with the distinct values, the horizontal axis with the name of the variable, and the vertical axis with “Frequency” or “Relative frequency”.

4.2Anatomy of a bar chart

4.3Lecture example: bar chart from a frequency table

1. The frequency distribution of data set 1 is $A = 4, B = 2, C = 2, D = 2$.
2. Axes: categories A to D along the bottom, frequency from 0 to 5 up the side.
3. Draw bars of height 4, 2, 2 and 2, separated by gaps.
4. Label the bars A to D, the horizontal axis “Category” and the vertical axis “Frequency”.

Using the relative frequencies instead gives the same shape with the vertical axis running 0 to 0.4.

4.4Bar chart in a Google Sheet

1. Select (highlight) the cells that have the data you want to visualise.
2. Click the Insert chart option in the toolbar.
3. Change the visualisation type in the Chart editor.
4. Set the Chart type to Bar chart.

4.5Going deeper: what to remember about bar charts

1. A bar chart shows the frequencies or relative frequencies of a categorical variable.
2. If the variable is ordinal, the order of the categories is preserved. The bars are not re-sorted (Chapter 5).
3. The bars can be vertical or horizontal. The rule that matters is that bar length is proportional to the value.
4. Bars must not touch. That is what separates a bar chart (categories) from a histogram (numbers on a continuous scale, a later topic).

Practice questions (graded-assignment style)

5.1Lecture example: a Pareto chart

Data (15 observations): A, B, B, C, A, D, B, B, A, C, B, B, C, D, A.

The slide prints 0.26, 0.40, 0.20, 0.14 so that the column adds up to exactly 1. The exact values are $4/15 = 0.267$ and $2/15 = 0.133$ (rounded to three decimals), which is why the third column is shown with three decimals.

1. Sort the categories by frequency: B (6), A (4), C (3), D (2).
2. Draw the bars in that order, tallest first.

5.2Lecture example: ordinal data

The T-shirt sizes (Small S, Medium M, Large L) of twenty students are L, M, M, S, L, S, S, M, L, M, M, S, S, L, M, S, M, S, L, M. Make the frequency table and the bar chart.

Check: $7 + 8 + 5 = 20$. Size is ordinal (Small < Medium < Large), so the bars keep this order and are not sorted by height.

5.3Pareto or ordered bars?

Practice questions (graded-assignment style)

6.1Know your purpose

Have a purpose for every table or graph you create, and choose the table or graph to serve that purpose.

6.2Label your data

• Label your chart to show the categories and indicate whether some have been combined or omitted.
• Name the bars in a bar chart.
• Name the slices in a pie chart.
• If you have omitted some of the cases, make sure the label of the plot defines the collection that is summarised.

Example: the same grades, grade A = 7, B = 3, C = 7, D = 2 (19 students), drawn both ways with a title, named slices or bars, and labelled axes. The relative frequencies are $7/19 = 36.8\%$, $3/19 = 15.8\%$, $7/19 = 36.8\%$ and $2/19 = 10.5\%$.

6.3Many categories

A bar chart or pie chart with too many categories might conceal the more important categories. In some cases, grouping the other categories together helps.

Example: a variable has 26 categories A to Z with counts A = 11, B = 6, C = 9, D = 4 and one case in each of E to Z. The 22 single-case categories are drawn as 22 tiny bars that crowd the chart and hide the four categories that matter. Grouping E to Z gives $22 \times 1 = 22$ cases, so the new category Others has count 22 and the chart has five clear bars.

6.4The area principle and decorated graphs

Decorated graphics. Charts decorated to attract attention often violate the area principle. Example: total U.S. wine exports (millions of dollars): United Kingdom 150.3, Canada 146.8, Japan 82.8, Italy 42.5, Netherlands 34.4. A decorated graphic with bottles on top of boxes of various sizes and shapes and no baseline breaks the principle, because the size of a box does not correspond to its value.

6.5Truncated graphs

Another common violation is when the baseline of a bar chart is not at zero. Example: regionwise counts South 146, West 97, North 146, East 111. If the vertical axis starts at 75, the visible bar heights are $146 - 75 = 71$, $97 - 75 = 22$, $146 - 75 = 71$ and $111 - 75 = 36$. The South bar then looks about $71/22 \approx 3.2$ times the West bar, but the true ratio is only $146/97 \approx 1.5$.

6.6Round-off errors

When table entries are percentages or proportions, the total may sum to a value slightly different from 100% or 1. This might result in a pie chart where the total does not add up. Example: Samsung 20%, Apple 15%, Nokia 14%, HTC 11%, RIM 10%, Others 31%. Then $20 + 15 + 14 + 11 + 10 + 31 = 101$, so the labels total 101% instead of 100%.

Each share was rounded to a whole percent, so the total drifted from 100%.

6.7Going deeper: a checklist before you trust a graph

1. Purpose: parts of a whole (pie) or groups against each other (bar)?
2. Labels: title, named slices or bars, labelled axes, and a note on anything combined or omitted.
3. Categories: too many? Group the small ones into Others.
4. Area principle: no decoration that changes sizes, and the baseline at zero (or a break mark).
5. Round-off: do the labels add to 100% (or 1)?

Practice questions (graded-assignment style)

Bar charts and pie charts are graphical summaries of categorical data. Sometimes we want a single label that sums up the data set. Numbers used to describe data sets are called descriptive measures; those that indicate where the center or most typical value lies are called measures of central tendency.

7.1Where the mode shows up in a chart

• It labels the longest bar in a bar chart.
• It labels the widest slice in a pie chart.
• In a Pareto chart, the mode is the first category shown.

7.2Lecture example: finding the mode

Find the mode of A, A, B, C, A, D, A, B, C, C, A, B, C, D, A.

1. Count each category: $A = 6, B = 3, C = 4, D = 2$ (total $6 + 3 + 4 + 2 = 15$).
2. The highest frequency is 6, for A.

The most common category is A, so the mode is A. This is the longest bar in the figure above.

7.3Bimodal and multimodal data

Example: A, A, B, C, A, C, A, B, C, C, A, C, C, D, A, A, C, D, B. The counts are $A = 7, B = 3, C = 7, D = 2$ (total $7 + 3 + 7 + 2 = 19$). The highest frequency, 7, is shared by A and C, so the data are bimodal.

7.4Going deeper: what the mode can and cannot do

1. The mode works for any categorical data, ordinal or not. It only needs counts.
2. The mode is a category, not a number. The answer is “A”, not “6”. The 6 is the frequency of the mode.
3. A data set can have more than one mode (bimodal or multimodal) when categories tie.

Practice questions (graded-assignment style)

Ordinal data offer another summary, the median, that is not available unless the data can be put into order.

8.1Three steps to the median

1. Put in order. Sort the observations by their natural order.
2. Find the middle. Odd count: one middle value. Even count: two middle values.
3. Read its category. The category there is the median. Either side is fine when the count is even.

For 15 observations the middle is the 8th. For 14 observations the two middle observations are the 7th and the 8th.

8.2Lecture examples

15 grades: the median is the 8th observation

Grades: A, B, B, C, A, D, B, B, A, C, B, B, C, D, A. In grade order A, B, C, D the sorted data are A,A,A,A,B,B,B,B,B,B,C,C,C,D,D.

The most common grade is B (6 times), so the mode is also B. In this example the mode and the median are the same.

14 grades: two middle observations

Grades: A, B, B, C, A, D, B, B, A, C, B, B, C, D. Sorted: A,A,A,B,B,B,B,B,B,C,C,C,D,D. With 14 observations the two middle ones are the 7th and the 8th. Both are B, so the median grade is B.

15 grades where the mode and the median differ

Grades: A, B, B, C, A, D, A, B, A, C, B, A, C, D, A. Sorted: A,A,A,A,A,A,B,B,B,B,C,C,C,D,D. The first six are A and the next four are B, so the 8th observation is B and the median is B. The most common grade is A (6 times), so the mode is A.

8.3Going deeper: why only ordinal data?

1. Sorting needs a natural order. Nominal categories (colours, cities) have none, so “the middle one” has no meaning.
2. Always sort first. Reading the 8th item of the unsorted list is the commonest slip.
3. Mode and median answer different questions. The mode is the most common category; the median is the category of the middle observation. They can be the same or different.

Practice questions (graded-assignment style)

Everything in Week 2 on one sheet, topic by topic, for a last-minute revision.

What each tool is for

Formulas and rules

Topic by topic