End-to-end Data Analytics for Product Development. Chris Jones

results, not based on intuition alone.

      To begin with, the researchers need to determine which question they want to focus on and then define the hypotheses. A statistical hypothesis is a claim about a population parameter (e.g. about the mean or the standard deviation of a variable of interest).

      Hypotheses should be based on our knowledge of the process, such as how a process has performed in the past or customers' expectations.

      To perform a hypothesis test, we need to define the null hypothesis and the alternative hypothesis.

       Null hypothesis H0: usually states that a population parameter, such as the population mean, equals a specified value or parameters from other populations.E.g. H0: the mean performance of the new product is equal to the industry standard.

       Alternative hypothesis H1: is the opposite of the null hypothesis, so it usually states that the population parameter does not equal a specified value or parameters from other populations.H1: the mean performance of the new product is NOT equal to the industry standard.

      Sometimes the alternative hypothesis is directional or one‐sided; that is, we suspect the population parameter is greater than or less than a given value.

       Null hypothesis H0:E.g. H0: the mean performance of the new product is equal to the industry standard.

       One‐sided alternative hypothesis H1:H1: the mean performance of the new product is GREATER THAN the industry standard.OrH1: the mean performance of the new product is LESS THAN the industry standard.

After defining null and alternative hypotheses, the hypothesis test uses sample data to decide on two possible conclusions:

      1 We can reject the null hypothesis in favor of the alternative hypothesis. If we reject the null hypothesis, we say that our result is statistically significant:E.g. H0 (the mean performance of the new product is equal to the industry standard) is REJECTED.

      2 We can fail to reject the null hypothesis and conclude that we do not have enough evidence to claim that the alternative hypothesis is true. We will say that our results are NOT statistically significant:E.g. H0 (the mean performance of the new product is equal to the industry standard) is NOT REJECTED.

      Because we are using sample data, decisions based on those hypothesis tests could be wrong.

Recall the confidence level (1 − α) that we discussed earlier (Stat Tool1.14). The confidence level is how sure we are that the confidence interval contains the true population parameter value. This confidence level is set by the researcher and is usually equal to 95% (0.95) or 99% (0.99).

      Let's consider the outcomes of a hypothesis test. If the null hypothesis is true and based on our sample data we fail to reject it, we make the correct decision, but if we reject it, we make an error. In hypothesis testing, the probability of rejecting a null hypothesis that instead is true is called significance level and is denoted by α. We always select it before performing the hypothesis test and it is usually equal to 5% (0.05) or 1% (0.01).

      Confidence level and significance level are tools to quantify the uncertainty about our inferential conclusions.

      Stat Tool 1.16 The p‐Value

      After establishing the null and alternative hypotheses and setting the significance level α, how do we decide to reject the null hypothesis?

      When we conduct a hypothesis test, the results include a probability called p‐value.

      We use the p‐value to determine whether we should reject or fail to reject the null hypothesis, by comparing it to the significance level α.

      If the p‐value is less than α, we reject the null hypothesis in favor of the alternative hypothesis (our result is statistically significant):

       E.g. H0 (the mean performance of the new product is equal to the industry standard) is REJECTED.

If the p‐value is greater than or equal to α, we fail to reject the null hypothesis. There is not enough evidence to claim that the alternative hypothesis is true (our results are NOT statistically significant):

       E.g. H0 (the mean performance of the new product is equal to the industry standard) is NOT REJECTED.

       Example 1.5. A researcher wants to investigate whether the performance of a new product differs with respect to the industry standard.

      Suppose they set the significance level α equal to 0.05 (5%). The two hypotheses are:

       H0: The mean performance of the new product is equal to the industry standard.

       H1: The mean performance of the new product is NOT equal to the industry standard.

      What if the p‐value is 0.032? Would they reject or fail to reject the null hypothesis?

      They should reject the null hypothesis H₀ because the p‐value is less than the significance level α and conclude that the new product has a different mean performance with respect to the industry standard. The result of the test is statistically significant at α = 5%.

      And, what if the p‐value is 0.076? Would they reject or fail to reject the null hypothesis?

      They should fail to reject the null hypothesis H₀ because the p‐value is greater than the significance level α and conclude that there is not enough evidence to claim that the new product has a different mean performance with respect to the industry standard. The result of the test is NOT statistically significant at α = 5%.

      The p‐value indicates whether our results are statistically significant. However, just because our results are statistically significant doesn't mean that they are practically significant.

       Example 1.6. A production line manager attempts to reduce production time by modifying the process. They compare the mean production time of the old process with the mean production time of the new process using a hypothesis test.The p‐value is 0.032. Using an alpha of 0.05, would they reject or fail to reject the null hypothesis?Because the p‐value is less than alpha, they reject the null hypothesis that the production times are equal. The difference between the mean production times is statistically significant.Now, consider practical significance. If the difference between the two production times is five seconds, is that really practically significant? Is it worth the cost of implementing the process change?

      Always consider the practical significance of your results and your knowledge of the process before reaching a final conclusion.