Introduction

At the end of the previous section, we looked at a hypothesis test for a population proportion. The result of the test gave us cause to look more closely at the sample and the evidence it provided. While we concluded the test by not rejecting the null hypothesis, on closer inspection the evidence against the null hypothesis was still fairly convincing.

But, since hypothesis tests are conducted so that we can make decisions, such close scrutiny of the data may get lost. For example, the test we are referring to was conducted by the Seller Door company to determine whether a new sales routine had increased the rate of sales. The conclusion of the test was that there wasn't enough evidence to suggest that the rate had increased. Perhaps a manager of the company heard this conclusion and decided to scrap the new routine altogether?

In an absolute sense, there's nothing wrong with doing this. But the manager has ignored the finer details of the test, which suggest that the data were extremely unlikely under the assumption that the rate of sales hadn't improved - despite the fact that such an assumption wasn't rejected by said data.

For this reason, there is a slight alternative to the hypothesis tests we have conducted so far in this chapter. In this alternative, we do not simply conclude the test by rejecting the null hypothesis or not rejecting the null hypothesis. We also provide a measure for how expected (or unexpected) the collected data are under the claim that is being tested. To be more specific, this alternative utilizes the following reasoning:

• Whenever we conduct a test, we collect a sample and measure a test statistic for that sample.
• The test statistic can be thought of as a measure of how 'extreme' that sample is under the assumption that the null hypothesis is true.
• Test statistics are standard 'scores', often z-scores or t-scores, in some probability distribution.
• We can report how likely such an extreme test statistic is in the distribution it comes from. This indicates how likely the sample itself is, under the assumption that the null hypothesis is true.
• The less likely the sample, the stronger the evidence against the null hypothesis.