Statistics for
Data Science 1

Statistics for Data Science 1 is an introductory course in statistics for beginners. You learn to create and handle data sets and to summarise them using graphical and numerical techniques. Then the idea of uncertainty is introduced, and probability is used as the tool to handle it. The course ends with random variables, with a detailed look at the Binomial and Normal distributions.

1.1Course objectives

1. Create, download and manipulate data sets.
2. Learn methods for presenting and describing sets of data, and select an appropriate graphical technique for a given scenario.
3. Learn measures that summarise a data set, and use the appropriate numerical summary for a given question.
4. Understand uncertainty through probability: random experiment, events, probability and conditional probability; and random variables, both discrete (in particular Binomial) and continuous (in particular Normal).

1.2Road map

The twelve weeks fall into three stages. Each stage builds on the one before: you first learn to describe data, then to reason about chance, and finally to model chance with random variables.

1.3Week-wise schedule

Each week is set up by a question that the data from the admissions office can answer. The last column lists that question.

1.4Running example: university admissions

XYZ university has just completed admissions to its undergraduate programme. Every admitted student fills in a form and the information is tabulated. A portion of the data collected by the admissions office is shown below. We will keep coming back to this table.

1.5Week 1 learning objectives

• What is statistics? Descriptive and inferential statistics; population versus sample. (Lecture 1.1)
• How data are collected. Identify variables and cases (observations) in a data set. (Lecture 1.2)
• Types of data. Categorical versus numerical, cross-sectional versus time-series, and measurement scales. (Lectures 1.3 and 1.4)
• Creating data sets, downloading and manipulating them, working on subsets, and framing questions that data can answer. These two objectives are listed in the course slides but are not part of the lectures used for these notes.

1.6What to expect from Week 1

Week 1 has almost no calculation. Nearly every question asks you to classify something. Five question types cover almost everything, and each one is handled in one chapter below.

The only calculations in Week 1 are a proportion (count ÷ total), the table size $n \times k$, a unit conversion, and the Celsius to Fahrenheit formula.

1.7How to study this week

Before collecting or describing anything we need two ideas: what statistics is for, and the difference between the group we care about and the part of it we actually look at.

2.1What is statistics?

The word "art" is a hint that statistics is not only calculation: choosing what to collect and how to read it matters just as much.

2.2Two branches of statistics

2.3Population and sample

Suppose we want to know any of the following.

If the population has $N$ elements and the sample has $n$, then every sample satisfies $n \le N$. We usually cannot measure the whole population, which is why we study a sample.

2.4Purpose of statistical analysis

• If the purpose is to examine and explore information for its own intrinsic interest only, the study is descriptive.
• If the information is obtained from a sample of a population and the purpose is to draw conclusions about the population, the study is inferential.
• A descriptive study may be performed either on a sample or on a population.
• Only when an inference is made about the population, based on information from the sample, does the study become inferential.

2.5Going deeper: population, sample and inference

1. Find the population from the question. The population is the group the conclusion is about. The sample is the part that was actually measured. If you measured everyone the conclusion is about, there is no sample, and the study is descriptive on a population.
2. Why study a sample at all? Reaching every element is often too costly, too slow or impossible (every house in a state, every car sold in a year).
3. The sample must represent the population. The lecture does not cover how to pick a sample, but remember the idea: a sample that leaves out part of the population can mislead, and inference carries a chance of error (this is why probability comes later).
4. The same numbers can be used in two ways. The Class 12 marks of the 8 students in the admissions table add up to 394 + 437 + 442 + 401 + 451 + 462 + 413 + 375 = 3375, so the average is $3375 \div 8 = 421.875$. Saying "these 8 students averaged 421.875" is descriptive. Saying "admitted students at the university average about 421.9" is inferential.

Words that hint at the type

These words are only hints. The final test is always the purpose: does the statement go beyond the data in hand?

2.6Exam tips and common mistakes

Practice questions (graded-assignment style)

Statistics works on data, so the first job is to understand what data are, where they come from, and how to lay them out so that they can be used.

3.1What is data?

To learn something we need to collect data. Statistics relies on data, the information that is all around us.

3.2Why do we collect data?

We are interested in the characteristics of some group or groups of people, places, things or events. For example:

• To know about the temperatures in a particular month in Chennai, India.
• To know about the marks obtained by students in their Class 12.
• To know how many people like a new song, product or video, collected through comments.

3.3Data collection

3.4Unstructured and structured data

For the information in a database to be useful, we must know the context of the numbers and text it holds. When they are scattered about with no structure, the information is of very little use. Hence we need to organise data.

3.5Variables and cases

• Case (observation): a unit from which data are collected.
• Variable:
  • Intuitively, a variable is something that varies.
  • Formally, it is a characteristic or attribute that varies across all units.
• In our school data set the case is each student, and the variables are name, marks obtained, board and so on.
• Rows represent cases: for each case, the same attributes are recorded.
• Columns represent variables: for each variable, the same type of value is recorded for every case.

A table with $n$ cases and $k$ variables has $n$ rows and $k$ columns, so it holds $n \times k$ data values.

3.6Going deeper: rows, columns and missing values

1. A case is whatever one row stands for. It depends on the data set: a student in the admissions table, a comment in the video data, a day in the potato data. Decide this first and the rest follows.
2. A variable is what is recorded for each case. Even S.No is a column, so it is a variable (an identifier), though we would never average it.
3. A data value is one cell. $n$ cases and $k$ variables give $n \times k$ values. The header row names the variables; it is not a case.
4. A well-organised table passes three checks: every column has a name and holds one kind of thing; every row is one case; every cell holds one value (not "Anjali, F, 484").
5. Missing is not zero. In the potato table, 04-Mar shows quantity 0 and cost NA. No potatoes were bought, so there is no cost to report. Writing 0 would claim the potatoes were free and would pull any average down. NA means "not available".

3.7How data are collected: sources, sampling methods and bias

Section 3.3 says that when the data are not available we must collect them. How we collect them decides whether we can trust what we conclude. This section covers the standard vocabulary.

A. Primary and secondary data

• Primary data are collected first-hand for the question at hand (your own survey, experiment or measurements).
• Secondary data were collected by someone else for another purpose and are reused: published reports, government statistics, existing databases.

B. Census and sample survey

C. Observational study and experiment

• Observational study: the researcher only records what happens (compare the marks of students who sleep 8 hours with those who sleep 5). No treatment is imposed.
• Experiment: the researcher assigns a treatment (for example one group gets a new teaching method, the control group gets the usual one) and compares the outcomes. Assigning units to groups at random makes the groups comparable.
• Only a well-designed experiment can support a cause-and-effect claim. In an observational study a confounding variable (a hidden factor linked to both) can create a false link: ice-cream sales and drowning both rise in summer, but ice cream does not cause drowning.

D. Sampling methods

Stratified versus cluster. Stratified: all groups are used and a few units are taken from each. Cluster: a few groups are used and all units are taken from each.

E. Bias

Bias is a systematic error that pushes the result in one direction. The usual sources:

• Selection bias: part of the population has no chance of being chosen (a phone survey that dials only landlines misses people with mobiles only).
• Voluntary-response bias: people with strong opinions are the ones who reply (online polls, call-in shows).
• Non-response bias: the chosen people who do not answer differ from those who do.
• Measurement (response) bias: a leading question or a faulty instrument distorts the answers.

F. Parameter and statistic

• A parameter is a number that describes the population (the mean of all students, $\mu$; the population size, $N$). It is fixed, usually unknown.
• A statistic is a number computed from a sample (the sample mean $\bar x$; the sample size $n$). It changes from sample to sample.
• We use statistics to estimate parameters. The difference $\bar x - \mu$ is the sampling error, the unavoidable chance difference between a sample and the population. Non-sampling errors (bias, wrong recording, non-response) can occur even in a census.

3.8Exam tips and common mistakes

Practice questions (graded-assignment style)

Once data are in a table, we classify each variable. The kind of variable decides which summaries and graphs make sense, so this classification comes before almost everything else in the course.

4.1Categorical and numerical variables

4.2Measurement units

A measurement unit is the scale that defines the meaning of numerical data, such as weights measured in kilograms, prices in rupees, heights in centimetres.

4.3Cross-sectional and time-series data

• Time series: data recorded over time.
• Cross-sectional: data observed at the same time.

Time-series example from the lecture

The lecture shows a daily record of potatoes, with the quantity (kg), the cost (Rs.) and the selling price (Rs.) for each date in March. The first rows look like this.

Every row is a different date for the same item, so this is a time series. NA means the value is not available.

4.4How to classify a data set

1. Classify each variable as categorical or numerical.
2. For numerical data, find out the unit of measurement.
3. Check whether the data are collected at a point of time (cross-sectional) or over time (time series).

4.5Going deeper: telling the types apart

1. Categorical or numerical? Ask: does an average make sense? Mobile number, PIN code, roll number and S.No are all made of digits, but averaging them means nothing, so they are categorical labels. Marks, price and height can be averaged, so they are numerical.
2. Categorical does not mean text, and numerical does not mean digits. The look of the value does not decide the type; its meaning does.
3. Every numerical variable has one unit. Write it down (marks, kg, Rs., cm). If entries use different units, convert first: 1 lb = 0.4536 kg, 1 m = 100 cm, 1 ft = 30.48 cm.
4. Cross-sectional or time series? Ask what each row is. Different people or items at one time: cross-sectional. The same item on different dates: time series.
5. The shuffle test. Shuffle the rows. If no information is lost, the data are cross-sectional. If the order matters (earlier to later), they are a time series.

Quick classification

4.6Discrete and continuous numerical variables

Numerical variables split once more, and this split decides which graphs and summaries come later in the course.

• Discrete: the possible values can be listed or counted one by one, with gaps between them. Usually the variable is a count: number of siblings, number of defective items, number of goals.
• Continuous: the variable can take any value in an interval, with no gaps. Usually it is a measurement: height, weight, time, temperature, rainfall.

Two cautions. (1) A variable that is continuous in principle is often recorded to a fixed precision (weight in whole kilograms, age in completed years); the underlying quantity is still continuous. If a question says the values are whole numbers only, treat it as it is stated. (2) Do not call a variable numerical just because it is written with digits: ask can it be meaningfully added or averaged? If not, it is a label.

The whole classification tree. Variable $\to$ categorical (nominal, ordinal) or numerical (discrete, continuous); and, for numerical variables, the scale of measurement is interval or ratio.

4.7Exam tips and common mistakes

Practice questions (graded-assignment style)

Data collection uses one of four scales of measurement: nominal, ordinal, interval or ratio. The scale tells us what we are allowed to do with the values. Each scale has all the properties of the one before it, plus one more.

5.1Nominal scale

• The data are labels or names used to identify the characteristic of an observation.
• Examples: name, board, gender, blood group.
• Nominal variables are sometimes numerically coded. For example, we might code men as 1 and women as 2, or men as 3 and women as 1. Both codes are valid.
• There is no ordering in the variable.

5.2Ordinal scale

• The data have the properties of nominal data, and the order or rank of the data is meaningful.
• Example: each customer who visits a restaurant rates the service as excellent, good or poor. The values are labels, as in nominal data. In addition they can be ranked with respect to service quality.

5.3Interval scale

• The data have all the properties of ordinal data, and the interval between values is expressed in terms of a fixed unit of measure.
• Interval data are always numeric. We can find the difference between any two values.
• Ratios of values have no meaning, because the value of zero is arbitrary.

5.4Ratio scale

• The data have all the properties of interval data, and the ratio of two values is meaningful.
• Examples: height, weight, age, marks.

5.5Example: temperature

The same word, "temperature", can be measured on different scales depending on what is recorded.

• If the response to "how hot is the day?" is comfortable or uncomfortable, temperature is a nominal variable.
• If the temperature of a liquid is recorded as cold, warm or hot, the variable is ordinal.
• Temperature in degrees Celsius or Fahrenheit is an interval variable, since there is no absolute zero.

In an air-conditioned room set at 20 °C, with 40 °C outside, it is correct to say the difference in temperature is 20 °C, but it is incorrect to say that outdoors is twice as hot as indoors.

5.6What each scale allows

5.7Going deeper: choosing the scale

Do not guess the scale from how the value looks. Ask four questions in order and stop at the first yes.

• The zero test. Does 0 mean "none of it"? 0 marks means no marks scored (ratio). 0 °C does not mean no heat: the same point is 32 °F, so zero is just a convention (interval).
• The "twice" test. Would "twice as much" survive a change of unit? Rs. 400 is twice Rs. 200 in any currency unit. 40 °C is not twice 20 °C, as the Fahrenheit conversion showed.
• Why ordinal has no differences. Excellent, good and poor are ranked, but nobody knows that "excellent minus good" equals "good minus poor", so subtracting is meaningless.
• The scale depends on how it is recorded. Age as exact years (25) is ratio. Age as a group (young, middle-aged, senior) is ordinal. The temperature example in 5.5 shows the same idea.
• Numbers can hide a nominal scale. Gender coded 1 and 2, or a PIN code, are still nominal. A 1-to-5 satisfaction rating is ordinal: ordered labels, not equal steps.
• Beyond the lecture: temperature in kelvin has a true zero (no heat energy), so ratios are meaningful there. That is why only Celsius and Fahrenheit are the interval example.

5.8Exam tips and common mistakes

Practice questions (graded-assignment style)

Every topic of Week 1 on one sheet, lecture by lecture, for a last-minute revision.

Course overview

Lecture 1.1 · Introduction to Statistics

Lecture 1.2 · Understanding Data

Lecture 1.3 · Classification of Data

Lecture 1.4 · Scales of Measurement

6.1Descriptive or inferential?

Rule: the purpose decides: only summarising the sample → descriptive; drawing a conclusion about the population → inferential (needs probability).

6.2Population and sample

Inference goes from the sample back to the population.

6.3Cross-sectional or time series?

Spot it: rows are different people or items → cross-sectional; rows are different dates of the same item → time series. NA = value not available.

6.4How to classify a variable

6.5The four scales at a glance

6.6What each scale allows

Each scale keeps every property of the one before it, and adds one more.

6.7Formulas and facts to remember

6.8Exam answer templates

For each question type, ask the check question, then use the answer template.

6.9Mixed practice

Practice questions (graded-assignment style)