AP Statistics · Unit 3 · Inference for Categorical Data: Proportions
3.8 Errors in tests
Describe Type I and Type II errors in a hypothesis test and give the consequences of each in context.
1/5A test can reach the wrong conclusion even when done correctly
Part 1 of 5: A test can reach the wrong conclusion even when done correctly
A hypothesis test uses one sample to judge a claim about a population. Samples vary from one to the next, so a sample can sometimes point the wrong way. Even a carefully run test can reach the wrong conclusion.
Every test ends with one of two decisions. We reject the null hypothesis, or we fail to reject it. The null hypothesis is the claim of no change or no difference. In reality, that claim is either true or false, and we never know which.
Crossing the two decisions with the two realities gives four outcomes. Two are correct and two are errors.
Decisions and outcomes in a hypothesis test
- We reject the null hypothesis
- Null hypothesis is true
- Type I error
- Null hypothesis is false
- Correct decision
- We fail to reject the null hypothesis
- Null hypothesis is true
- Correct decision
- Null hypothesis is false
- Type II error
Part 2 of 5: A Type I error rejects a true null hypothesis
Start with the top-left error in the table. A Type I error happens when we reject a null hypothesis that is actually true. The sample suggests a difference, but none really exists. It is a false alarm.
The probability of a Type I error equals the significance level , the cutoff chosen before the test. With , a true null hypothesis is rejected in about 5% of all possible samples. We choose before collecting data, so we control this risk directly.
Key term
Type I error
A Type I error is a false positive: the null hypothesis is true but the test rejects it, with probability equal to .

Quick check
One question on the part you just read
Easy
Show answer and explanation
Answer: C · Worked explanation
Part 3 of 5: A Type II error fails to detect a real effect
The second error runs the other way. A Type II error happens when we fail to reject a null hypothesis that is actually false. A real difference exists, but our sample did not reveal it. It is a missed detection.
The probability of a Type II error is written . We do not choose it directly. It depends on the sample size, on , and on how far the true value lies from the null value.
The chance of correctly rejecting a false null hypothesis is the power of a test. Power equals . If , the power is 0.80.
Key terms · tap one for its definition
Type II error
A Type II error is a false negative: the null hypothesis is false but the test fails to reject it, with probability .
power
The power of a test is the probability of rejecting the null hypothesis when a specific alternative value is true; it equals .
Part 4 of 5: Lowering one error risk usually raises the other
It would be ideal to make both errors rare. With a fixed sample size, the two risks pull against each other.
Lowering makes rejection harder. Fewer true null hypotheses get rejected, so Type I errors become rarer. Fewer false null hypotheses get rejected too, so rises and power falls.
Three changes reduce and raise power:
- A larger sample size lowers and leaves unchanged.
- A larger lowers but raises the Type I risk.
- A true value farther from the null value lowers .

Quick check
One question on the part you just read
Easy
Show answer and explanation
Answer: B · Worked explanation
Part 5 of 5: The cost of each error depends on the context
Now consider what these errors cost in real settings. Neither error is always the worse one. The answer depends on the harm each one causes.
Worked example
Describing both errors in context
- A Type I error rejects a true null hypothesis. Here, the data suggest the cure proportion exceeds 0.70 when it is really 0.70.
- A Type II error fails to reject a false null hypothesis. Here, the data show no convincing improvement, but the true proportion is above 0.70.
- Type I consequence: patients switch to a drug that is no better, and may face extra cost or side effects.
- Type II consequence: patients miss a drug that would have cured more people.
Answer: Type I: concluding the new drug cures more than 70% of patients when it does not. Type II: failing to reach that conclusion when it does.
Priorities can differ between settings. Suppose a factory tests the null hypothesis that a batch of syringes meets a safety standard. A Type II error ships an unsafe batch. A Type I error only discards a good batch. In that setting, a researcher may accept a larger to lower .
Check your recall[12 marks]
6 written questions. Answer each one in your head or on paper, then open the mark scheme.
- 1.Identify[1 mark]
Identify the type of error made when a test rejects a null hypothesis that is actually true.
Show mark schemeHide mark scheme
Model answer
A Type I error.
- 1 mark for identifying a Type I error
- 2.Describe[2 marks]
A bakery tests against , where is the true proportion of loaves with a defective crust. Describe a Type I error in this context.
Show mark schemeHide mark scheme
Model answer
The test concludes that the proportion of defective loaves exceeds 0.10 when the true proportion is actually 0.10.
- 1 mark for stating that the null hypothesis is rejected (convincing evidence that the proportion exceeds 0.10)
- 1 mark for stating that the true proportion is actually 0.10, so the null hypothesis is true
- 3.Describe[2 marks]
For the bakery test with and , describe a Type II error in context.
Show mark schemeHide mark scheme
Model answer
The test finds no convincing evidence that the proportion of defective loaves exceeds 0.10, but the true proportion is actually above 0.10.
- 1 mark for stating that the null hypothesis is not rejected (no convincing evidence of a proportion above 0.10)
- 1 mark for stating that the true proportion of defective loaves is actually above 0.10
- 4.Explain[2 marks]
Explain why lowering the significance level from 0.05 to 0.01, with the same sample size, increases the probability of a Type II error.
Show mark schemeHide mark scheme
Model answer
A smaller makes rejecting the null hypothesis harder. The test therefore also rejects fewer false null hypotheses, so rises.
- 1 mark for stating that a smaller makes rejection harder
- 1 mark for linking this to fewer false null hypotheses being rejected, so increases
- 5.Predict[2 marks]
Predict how increasing the sample size, while keeping fixed, changes and the power of the test.
Show mark schemeHide mark scheme
Model answer
The probability decreases and the power increases, while stays the same.
- 1 mark for predicting that decreases
- 1 mark for predicting that power increases
- 6.Justify[3 marks]
A city tests the null hypothesis that a bridge is safe to carry traffic. Justify which error, Type I or Type II, has the more serious consequence.
Show mark schemeHide mark scheme
Model answer
A Type I error closes a bridge that is actually safe, which costs money and causes inconvenience. A Type II error leaves an unsafe bridge open, which could cause injury or loss of life. The Type II error is more serious because the harm to people outweighs the cost of a closure.
- 1 mark for describing a Type I error as rejecting a true null hypothesis, so a safe bridge is closed unnecessarily
- 1 mark for describing a Type II error as failing to reject a false null hypothesis, so an unsafe bridge stays open
- 1 mark for concluding that the Type II error is more serious because harm to people outweighs the cost of closure

Practice this topic
10 quick-fire multiple-choice questions, easy to hard. Tap an answer to see if you're right and the worked explanation. (The written questions with mark schemes are under “Check your recall” above.)
0 right0 of 10 answered
Includes the 2 quick checks in the note above.
EasyIdentify[1 mark]
Show answer and explanation
Answer: D · Worked explanation
MediumCalculate[2 marks]
Show answer and explanation
Answer: C · Worked explanation
MediumIdentify[2 marks]
Show answer and explanation
Answer: B · Worked explanation
MediumPredict[2 marks]
Show answer and explanation
Answer: B · Worked explanation
MediumIdentify[2 marks]
Show answer and explanation
Answer: D · Worked explanation
HardPredict[2 marks]
Show answer and explanation
Answer: A · Worked explanation
Hard[2 marks]
Show answer and explanation
Answer: C · Worked explanation
HardCompare[2 marks]
Show answer and explanation
Answer: D · Worked explanation