Hypothesis Testing and Decision Errors

Applied Statistics

MTH-361A | Spring 2025 | University of Portland

March 17, 2025

Objectives

Previously… (1/3)

The guiding principle of statistics is statistical thinking.

Statistical Thinking in the Data Science Life Cycle

Statistical Thinking in the Data Science Life Cycle

Previously… (2/3)

Types of Inference

Parameter Estimation Hypothesis Testing
Goal Estimate an unknown population value Assess claims about a population value
Methods Point Estimation: A single value estimate (e.g., sample mean)
Interval Estimation: A range of plausible values (e.g., confidence interval)
State a null and an alternative hypothesis
Compute a test statistic and compare it to a threshold (p-value or critical value)
Key Concept Focuses on precision in estimation (confidence intervals) Focuses on decision-making based on evidence (reject or fail to reject the null hypothesis)

Previously… (3/3)

Confidence Intervals

\[\text{point estimate} \pm z^{\star} \times \text{SE}.\]

Finding \(z^{\star}\) Exactly

Find the \(z^{\star}\) for a 92% confidence level.

Process:

Using R:

cl <- 0.92 # confidence level
lt <- (1-cl)/2 # lower tail probability
qnorm(lt,0,1) # computes the z star
## [1] -1.750686

Introduction to Hypothesis Testing

Hypothesis testing is a statistical method used to make inferences about a population based on a sample. It helps determine if an observed effect is statistically significant.

Key Concepts:

Decision Rule:

Why is Hypothesis Testing Important?

Example 1

Scenario:

A pharmaceutical company tests whether a new drug improves recovery rates compared to a placebo.

Test Results:

Conclusion:

Outcomes of Hypothesis Testing

There are two possible outcomes of the hypothesis test:

Making statistical decisions means that you have to deal with uncertainties.

Decision Errors

Image Source: [Statistical Performance Measures by Neeraj Kumar Vaid](https://neeraj-kumar-vaid.medium.com/statistical-performance-measures-12bad66694b7){target=_blank}

Image Source: Statistical Performance Measures by Neeraj Kumar Vaid

This meme might be over used. If you find some memes similar to this but in “non-pregnancy” context, let me know.

The Significance Level and Decisions Errors

What does this all mean? When the p-value is small, i.e., less than a previously set threshold (\(\alpha\)), we say the results are statistically significant. The value of \(\alpha\) represents how rare an event needs to be in order for the null hypothesis to be rejected. The \(\alpha\) also represents the probability of committing a type I error.

Reality/Decision Reject \(H_0\) Fail to reject \(H_0\)
\(H_0\) is true Type I error
with probability \(\alpha\)
(significance level)
Correct decision
with probability \(1-\alpha\)
(confidence level)
\(H_0\) is false Correct decision
with probability \(1-\beta\)
(power of test)
Type II error
with probability \(\beta\)

Conclusion errors: Type I error (false positive) or Type II error (false negative)

Trade-offs between Type I and Type II errors. (1/2)

Images Source: Type I and Type II errors by Pritha Bhandari

Trade-offs between Type I and Type II Errors. (2/2)

Images Source: [Type I and Type II errors by Pritha Bhandari](https://www.scribbr.com/statistics/type-i-and-type-ii-errors/){target=_blank}

Images Source: Type I and Type II errors by Pritha Bhandari

Example 2

Example 2: Type I error Consequences

A Type I error occurs when the null hypothesis is incorrectly rejected, leading to a wrongful conviction.

This means that an innocent person is found guilty and sentenced, possibly facing imprisonment or even capital punishment. The consequences extend beyond the individual, affecting their family, reputation, and future opportunities. Additionally, the real perpetrator remains free, potentially committing further crimes.

Example 2: Type II error Consequences

A Type II error occurs when the null hypothesis was failed to reject, leading to a wrongful acquittal.

This means that a guilty person is found not guilty and released. As a result, justice is not served for the victims, and the criminal may go on to commit additional offenses, putting society at risk. This error can undermine public trust in the legal system, as it fails to hold the guilty accountable.

Activity: Determine Claims and Error Types

  1. Make sure you have a copy of the M 3/17 Worksheet. This will be handed out physically and it is also digitally available on Moodle.
  2. Work on your worksheet by yourself for 10 minutes. Please read the instructions carefully. Ask questions if anything need clarifications.
  3. Get together with another student.
  4. Discuss your results.
  5. Submit your worksheet on Moodle as a .pdf file.

References

Diez, D. M., Barr, C. D., & Çetinkaya-Rundel, M. (2012). OpenIntro statistics (4th ed.). OpenIntro. https://www.openintro.org/book/os/
Speegle, Darrin and Clair, Bryan. (2021). Probability, statistics, and data: A fresh approach using r. Chapman; Hall/CRC. https://probstatsdata.com/