Type I and Type II errors
What we've just been describing is known as a Type I error. A Type I error in a hypothesis test occurs when you reject the null hypothesis when it's true. As we've been discussing, the conditional probability of this error is chosen at the beginning of the test to be α, the level of significance.
But another kind of error can occur in a test: you can fail to reject the null hypothesis when it is false (and therefore should be rejected). This is known as a Type II error and the conditional probability of failing to reject the null hypothesis when it is false is denoted β.
At this stage it is worth repeating that you shouldn't interpret the word 'error' to mean that you have done anything wrong! The word is used to indicate that there is a gap between what the test tells you to conclude and what, in reality, is actually happening. All tests involve a level of uncertainty, so you cannot be sure that any conclusions you draw from them are correct. The test, after all, is based on a sample from the population. We are trying to use that sample to provide information about the population. This information is always incomplete, which can potentially lead to either of the above two errors in a hypothesis test.
In a hypothesis test:
A Type I error occurs if the null hypothesis is rejected when it is true (and shouldn't be rejected). The probability of this error is denoted by α.
A Type II error occurs if the null hypothesis is not rejected when it is false (and should have been rejected). The probability of this error is denoted by β.
The table to the right indicates the different situations that can occur in a hypothesis test. Put simply, when you conduct a hypothesis test, either the null hypothesis in the true underlying situation is true or it is false. (This is to do with the way things actually are.) And you will either reject the null hypothesis or you will not. (This is to do with the way you conclude things are from your test.) Depending on how the way things are matches up (or doesn't match up) with the way you conclude the test, you will be in one of the four cells of the table.
We will now turn our attention to the likelihood of the two types of errors, α and β. Earlier in this section we expressed α as a conditional probability formula. We will provide it again here with an analogous formula for β.
α = P(H0 is rejected | H0 is true)
β = P(H0 is not rejected | H0 is false)
If a statistician can know more about how likely the errors are, they will be in a better position to design a hypothesis test to suit the needs of their study.