AP Statistics · Unit 1 · Exploring One-Variable Data and Collecting Data
1.12 Problems with sampling
Identify sources of bias in sampling, explain how undercoverage and nonresponse affect results, and describe ways to reduce sampling problems.
1/5Bias is a systematic error that a larger sample cannot fix
- Bias is a systematic error that a larger sample cannot fix
- Undercoverage leaves some individuals with no chance to appear
- Nonresponse and voluntary response both distort who is counted
- Question wording and honesty can bias the answers too
- Careful design reduces bias, and larger samples reduce variability
- Check your recall
- Practice: 10 questions
Part 1 of 5: Bias is a systematic error that a larger sample cannot fix
Even a well-planned sample can give misleading results. Two different problems need to be kept apart. The first is bias, which pushes results in the same direction every time. Mixing the two up is a common source of errors.
The second is sampling variability, the natural variation in a statistic from one random sample to the next. Variability is not systematic. It shrinks as the sample size grows.
Key terms · tap one for its definition
bias
Bias is a systematic tendency of a sampling method to over- or underrepresent part of the population.
sampling variability
Sampling variability is the natural variation of a statistic from one random sample to another, which decreases as the sample size increases.
Part 2 of 5: Undercoverage leaves some individuals with no chance to appear
One source of bias comes from the list used to choose the sample. Undercoverage occurs when some members of the population have no chance of being selected. This usually happens because the list leaves them out. A survey that uses only a landline phone directory misses households that have only cell phones.
Those missing households may differ from the rest in age, income, or habits. The results then describe the people who could be reached, not the whole population.
Key term
undercoverage
Undercoverage is a source of bias in which some members of the population are left out of the sampling frame and cannot be selected.

Quick check
One question on the part you just read
Easy
Show answer and explanation
Answer: A · Worked explanation
Part 3 of 5: Nonresponse and voluntary response both distort who is counted
Even when the list is complete, not everyone chosen takes part. Nonresponse occurs when a selected individual cannot be contacted or refuses to answer. If those who do not respond differ from those who do, the results are biased. A low response rate makes this more likely.
Voluntary response bias occurs when individuals decide for themselves whether to respond. People with strong opinions are more likely to answer, so results tend to be extreme.
Convenience sampling, which means choosing the individuals who are easiest to reach, causes similar trouble. It leaves out people who are harder to reach.
Worked example
Identifying bias in a survey
- Ask who decided to take part: the students chose for themselves.
- Name the bias: this is voluntary response bias.
- Describe the effect in context: students with strong feelings, very pleased or very upset, are most likely to write.
- Conclude that the comments probably overrepresent extreme opinions about the food.
Answer: The bias is voluntary response bias, and the comments likely overrepresent strongly positive or strongly negative opinions.
Key terms · tap one for its definition
nonresponse
Nonresponse is a source of bias in which a selected individual cannot be reached or declines to provide data.
voluntary response bias
Voluntary response bias is bias that arises because individuals choose whether to respond, so people with strong opinions tend to be overrepresented.
Part 4 of 5: Question wording and honesty can bias the answers too
Bias can also enter after the sample is chosen. Response bias occurs when answers are systematically inaccurate. A leading question pushes people toward one answer. Asking “Don’t you agree that homework is too long?” invites agreement.
Sensitive questions cause the same problem. People may understate behaviors that others might disapprove of. Neutral wording, private answers, and anonymous surveys can reduce this.
Even honest answers can be affected by how a question is asked. The order of the questions and the tone of an interviewer can also change what people say.
Key term
response bias
Response bias is a systematic pattern of inaccurate answers caused by question wording, sensitive topics, or the way a question is asked.

Quick check
One question on the part you just read
Easy
Show answer and explanation
Answer: A · Worked explanation
Part 5 of 5: Careful design reduces bias, and larger samples reduce variability
Now that you can spot each problem, consider how to reduce it. The table matches each problem with a fix.
Problems and ways to reduce them
- Problem
- Undercoverage
- Way to reduce it
- Use a sampling frame that covers the whole population
- Problem
- Nonresponse
- Way to reduce it
- Follow up with those who do not reply
- Problem
- Voluntary response
- Way to reduce it
- Select individuals by random sampling
- Problem
- Convenience sampling
- Way to reduce it
- Replace easy-to-reach choices with random selection
- Problem
- Response bias
- Way to reduce it
- Use neutral wording and private answers
- Problem
- Sampling variability
- Way to reduce it
- Take a larger random sample
Only the last fix uses sample size. Replacing a nonresponder with an easy substitute does not help. The substitute may differ from the person who did not respond. Careful planning is cheaper than trying to repair a flawed sample afterward.
Check your recall[12 marks]
5 written questions. Answer each one in your head or on paper, then open the mark scheme.
- 1.Identify[1 mark]
A website asks visitors to vote on a local issue, and only those who choose to vote are counted. Identify the type of bias.
Show mark schemeHide mark scheme
Model answer
This is voluntary response bias.
- 1 mark for identifying voluntary response bias
- 2.Explain[3 marks]
Explain how undercoverage can lead to results that do not represent the population.
Show mark schemeHide mark scheme
Model answer
Some members of the population are missing from the sampling frame, so they have no chance of being selected. The people left out may differ from those included. The results therefore systematically misrepresent the population.
- 1 mark for stating that some members have no chance of being selected
- 1 mark for stating that those left out may differ from those included
- 1 mark for concluding that the results systematically misrepresent the population
- 3.Describe[2 marks]
Describe two ways to reduce bias caused by nonresponse.
Show mark schemeHide mark scheme
Model answer
Follow up with selected individuals who do not reply, such as by contacting them again. Make the survey easy to complete, so more selected individuals take part.
- 1 mark for describing following up with those who did not respond
- 1 mark for describing a second reasonable method, such as making the survey easy to complete
- 4.Justify[4 marks]
A polling group has a biased method and increases its sample size from 500 to 5,000. Justify whether this removes the bias.
Show mark schemeHide mark scheme
Model answer
It does not remove the bias. Bias is systematic, so it pushes results in the same direction. A larger sample only reduces sampling variability. The same part of the population is still over- or underrepresented, so the method must be changed.
- 1 mark for stating that the bias is not removed
- 1 mark for stating that bias is systematic and pushes results in one direction
- 1 mark for stating that a larger sample reduces only sampling variability
- 1 mark for stating that the method itself must change
- 5.Identify[2 marks]
A survey asks, “Don’t you agree that the cafeteria food is terrible?” Identify the source of bias and describe how to reduce it.
Show mark schemeHide mark scheme
Model answer
This is response bias caused by a leading question. Reword the question so it is neutral.
- 1 mark for identifying response bias from a leading question
- 1 mark for describing neutral wording as the fix

Practice this topic
10 quick-fire multiple-choice questions, easy to hard. Tap an answer to see if you're right and the worked explanation. (The written questions with mark schemes are under “Check your recall” above.)
0 right0 of 10 answered
Includes the 2 quick checks in the note above.
EasyIdentify[1 mark]
Show answer and explanation
Answer: A · Worked explanation
MediumIdentify[1 mark]
Show answer and explanation
Answer: A · Worked explanation
MediumCalculate[1 mark]
Show answer and explanation
Answer: A · Worked explanation
MediumIdentify[1 mark]
Show answer and explanation
Answer: A · Worked explanation
MediumIdentify[1 mark]
Show answer and explanation
Answer: A · Worked explanation
HardJustify[2 marks]
Show answer and explanation
Answer: A · Worked explanation
HardPredict[2 marks]
Show answer and explanation
Answer: A · Worked explanation
HardDetermine[2 marks]
Show answer and explanation
Answer: A · Worked explanation