Objectives

Know how to differentiate between null and alternative sampling distributions
Develop an understanding of decision errors
Activity: Determine Claims and Error Types

These slides are derived from Diez et al. (2012).

Previously… (1/3)

The guiding principle of statistics is statistical thinking.

Statistical Thinking in the Data Science Life Cycle

Previously… (2/3)

	Parameter Estimation	Hypothesis Testing
Goal	Estimate an unknown population value	Assess claims about a population value
Methods	Point Estimation: A single value estimate (e.g., sample mean) Interval Estimation: A range of plausible values (e.g., confidence interval)	State a null and an alternative hypothesis Compute a test statistic and compare it to a threshold (p-value or critical value)
Key Concept	Focuses on precision in estimation (confidence intervals)	Focuses on decision-making based on evidence (reject or fail to reject the null hypothesis)

Previously… (3/3)

Confidence Interval for One Proportion

\[\hat{p} \pm z^{\star} \text{SE}_{\hat{p}}\]

\[ \begin{aligned} \hat{p} & \longrightarrow \text{sample proportion (or the point estimate)} \\ z^{\star} & \longrightarrow \text{critical z-score at a given confidence level} \\ \text{SE}_{\hat{p}} & \longrightarrow \text{standard error of the sampling distribution} \\ \end{aligned} \]

Hypothesis Testing for One Proportion

\[ \begin{aligned} p & \longrightarrow \text{population proportion} \\ \hat{p} & \longrightarrow \text{sample proportion (or the point estimate)} \\ H_0: p = p_0 & \longrightarrow \text{null hypothesis} \\ H_A: p \ne p_0 & \longrightarrow \text{alternative hypothesis (can be } < \text{ or } > \text{)} \\ z & \longrightarrow \text{test statistic} \\ \text{SE}_{p} & \longrightarrow \text{standard error of the null distribution} \\ \end{aligned} \]

An Overview of Hypothesis Testing

Hypothesis testing is a statistical method used to make inferences about a population based on a sample. It helps determine if an observed effect is statistically significant.

Key Concepts:

Null Hypothesis ($H_0$): Assumes no effect or no difference.
Alternative Hypothesis ($H_A$): Represents what we aim to support (effect or difference exists).
Significance Level ($\alpha$): The probability of rejecting $H_0$ when it is true.
Test Statistic: A value calculated from sample data to assess evidence against $H_0$.
P-value: The probability of observing data as extreme as the sample, assuming $H_0$ is true.
Conclusion: Compare p-value with $\alpha$ to decide whether to reject $H_0$ or not.

Decision Rule:

If $\text{p-value} < \alpha$, reject $H_0 \longrightarrow$ Evidence supports $H_A$.
If $\text{p-value} \ge \alpha$, fail to reject $H_0 \longrightarrow$ Not enough evidence for $H_A$.

Why is Hypothesis Testing Important?

Supports decision-making in research
Helps determine if results are due to chance or a real effect

Example 1

Scenario:

A pharmaceutical company tests whether a new drug improves recovery rates compared to a placebo.

Null Hypothesis ($H_0$): The new drug has no effect (the recovery rate is the same as the placebo).
Alternative Hypothesis ($H_A$): The new drug improves recovery rates (higher than the placebo).
Significance Level ($\alpha$): 0.05 (5%).

Test Results:

After conducting a clinical trial, the statistical test produces a p-value of 0.02.

Conclusion:

Since p-value (0.02) < significance level (0.05), we reject $H_0$.
This suggests that there is strong statistical evidence that the new drug improves recovery rates compared to the placebo.

Outcomes of Hypothesis Testing

There are two possible outcomes of the hypothesis test:

Reject $H_0$: If the p-value is less than the significance level, then we reject the null hypothesis. Then, we have enough evidence to support $H_A$.
Fail to Reject $H_0$: If the p-value is greater than or equal to the significance level, then we fail to reject the null hypothesis. This does not mean the the null hypothesis is true.

Making statistical decisions means that you have to deal with uncertainties.

The Significance Level and Decisions Errors

What does this all mean? When the p-value is small, i.e., less than a previously set threshold ($\alpha$), we say the results are statistically significant. The value of $\alpha$ represents how rare an event needs to be in order for the null hypothesis to be rejected. The $\alpha$ also represents the probability of committing a type I error.

Reality/Decision	Reject $H_0$	Fail to reject $H_0$
$H_0$ is true	Type I error with probability $\alpha$ (significance level)	Correct decision with probability $1-\alpha$ (confidence level)
$H_0$ is false	Correct decision with probability $1-\beta$ (power of test)	Type II error with probability $\beta$

Conclusion errors: Type I error (false positive) or Type II error (false negative)

Trade-offs between Type I and Type II errors. (1/2)

Images Source: Type I and Type II errors by Pritha Bhandari

Trade-offs between Type I and Type II Errors. (2/2)

$Images Source: [Type I and Type II errors by Pritha Bhandari](https://www.scribbr.com/statistics/type-i-and-type-ii-errors/){target=_blank}$

Images Source: Type I and Type II errors by Pritha Bhandari

Example 2

Question: In a US court, the defendant is either innocent ($H_0$) or guilty ($H_A$). What does a Type I Error represent in this context? What does a Type II Error represent?

Answer: If the court makes a Type I Error, this means the defendant is innocent ($H_0$ is true) but wrongly convicted. A Type II Error means the court failed to reject $H_0$ (i.e., failed to convict the person) when they were in fact guilty ($H_A$ true).

Example 2: Type I error Consequences

A Type I error occurs when the null hypothesis is incorrectly rejected, leading to a wrongful conviction.

This means that an innocent person is found guilty and sentenced, possibly facing imprisonment or even capital punishment. The consequences extend beyond the individual, affecting their family, reputation, and future opportunities. Additionally, the real perpetrator remains free, potentially committing further crimes.

Example 2: Type II error Consequences

A Type II error occurs when the null hypothesis was failed to reject, leading to a wrongful acquittal.

This means that a guilty person is found not guilty and released. As a result, justice is not served for the victims, and the criminal may go on to commit additional offenses, putting society at risk. This error can undermine public trust in the legal system, as it fails to hold the guilty accountable.

Activity: Determine Claims and Error Types

Make sure you have a copy of the F 3/21 Worksheet. This will be handed out physically and it is also digitally available on Moodle.
Work on your worksheet by yourself for 10 minutes. Please read the instructions carefully. Ask questions if anything need clarifications.
Get together with another student.
Discuss your results.
Submit your worksheet on Moodle as a .pdf file.

Reality/Decision	Reject \(H_0\)	Fail to reject \(H_0\)
\(H_0\) is true	Type I error with probability \(\alpha\) (significance level)	Correct decision with probability \(1-\alpha\) (confidence level)
\(H_0\) is false	Correct decision with probability \(1-\beta\) (power of test)	Type II error with probability \(\beta\)

Decision Errors

Elementary Statistics