As we will see later in this chapter, the statistical analyses for research involving quasi-IVs and IVs are identical; what differs is the language we use to describe the results. When we use quasi-IVs, we use the language of correlation. Thus, if we have sailing team members and nonsailing team members take a cognitive task known as the Mental Rotations Test (MRT; see Vandenberg & Kuse, 1978) and sailing team members score significantly higher than do nonsailing team members on this test, can we state that being a sailing team member caused this higher performance? No. What we can say is that there is a relationship between being a sailing team member and scoring higher on the MRT in comparison to the performance of nonsailing team members.
Reframing a Research Idea
First come research questions; then come research designs; last come statistical analyses. The manner in which your research question is stated guides the research design. Let’s start with a correlational example and transform it into a quasi-experimental and finally an experimental design. This transformation will illustrate that there is usually a way to approximate an experiment based in the real world, even when the specific real-world variables cannot be manipulated.
Revisit and Respond 3.1
Explain how a quasi-IV differs from an IV and the difference in language appropriate to use in a correlational versus an experimental study.
Identify the IV (manipulated variable) and the DV (effect or outcome variable) in the following description:
“The campers were randomly assigned to two different cabins right next to each other, A and B; the camp director wanted to test the effectiveness of different kinds of mosquito netting (MN) and placed MN with white nylon fabric in Cabin A and MN with black nylon fabric in Cabin B. There was a significant difference in the number of bug bites reported by the campers in the two different cabins.”
As discussed earlier, if you ask a question about the sample as a whole without any manipulation that forms groups, you will have a correlational research design. For example, if you ask whether the number of magazine subscriptions in a home is related to reading scores in fourth graders (see Figure 3.2), you will have a correlational design; the statistics will be Pearson’s r.
Figure 3.2 Example of Correlational Research Design
Figure 3.3 Example of Quasi-Experimental Research Design
Source: Adapted from Devlin, A. (2006), Research methods: Planning, conducting, and presenting research. Belmont, CA.: Wadsworth/Thomson., Figure 2.2.
If you ask a question about group differences, and the groups preexist (like subscribing to print magazines or not), you will have a quasi-experimental design. For example, if you ask whether there are higher fourth-grade reading scores in the homes of people who subscribe to print magazines than in the homes of people who do not, that is a quasi-experimental design (the preexisting groups are composed of people with and without print magazine subscriptions; see Figure 3.3). In that particular situation, if your quasi-independent variable is print magazine subscriptions, which has two levels (yes or no) and your dependent variable is the fourth-grade reading scores of the children from those homes (one DV), your statistical analysis will be an independent samples t test. There is still no causality.
If you ask a research question about group differences, and the groups are created through manipulation, you will have an experimental research design. For example, if you create written scenarios (written text describing situations) in which participants are randomly assigned to read a scenario about a home with no (zero) magazine subscriptions/month versus a home with 10 magazine subscriptions/month, and you ask participants their estimate of the reading scores of the fourth graders in the home (chosen from a scale of the possible reading scores), that is an experimental design (see Figure 3.4). In this situation, there is one IV (magazine subscriptions) with two levels (zero vs. 10 subscriptions/month) and one DV (the reading score estimate). Again, your statistical analysis will be an independent samples t test.
The statistical test in the examples of the quasi-experimental and true experimental designs are the same, but the language you use to describe the results will differ. In the case of the quasi-experimental research, you will use the language of correlation—for example, probably that subscribing to print magazines is associated with having higher fourth-grade reading scores than is the case in a home where there are no print magazine subscriptions. With this experimental design, you have the language of causality, for example, that reading about a household with 10 magazine subscriptions/month led to judgments of higher reading scores in fourth graders than did reading about a household with no (zero) magazine subscriptions/month.
Figure 3.4 Example of Experimental Research Design
Source: Adapted from Devlin, A. (2006), Research methods: Planning, conducting, and presenting research. Belmont, CA.: Wadsworth/Thomson., Figure 2.3.
Appendix A (Figure A.1) at the end of the book contains a statistical decision tree that you may find helpful when thinking about how to evaluate your research question statistically.
Type I Versus Type II Error
Research in the social and behavior sciences typically uses inferential statistics, which means that we use samples to make informed guesses about the characteristics of the population from which the sample is drawn. In other words, we don’t know for sure about an answer to a particular outcome because we haven’t asked or assessed every person in the population of interest. We have asked what we hope is a representative sample. But we could be wrong because we are using inferences about our statistical hypotheses. Type I and Type II errors describe the ways in which we could be wrong. In a Type I error, we claim that we have a significant statistical result when that is not the case. Formally, we reject the null hypothesis (of no statistical difference or relationship between groups) when we should not have done so. In a Type II error, we have missed a finding that is there. Formally, we fail to reject the null hypothesis when there is a statistical difference, that is, when we should have done so.
Figure 3.5 Actual State of Affairs
Figure 3.5 represents the four possible outcomes. We can be correct in two ways: We are correct when we reject the null hypothesis when there is a finding; we are correct when we do not reject the null hypothesis because there is no finding. We can also be incorrect in two ways (our Type I and Type II errors). We are incorrect when we reject the null hypothesis (say there are statistical differences or relationships; a false alarm) when there are none (Type I error). We are also incorrect when we do not reject the null hypothesis and we should have, that is, such statistical differences or relationships are there (Type II error; a miss).
Bonferroni adjustment: Adjustment for Type I error by dividing the alpha level (.05) by the number of statistical tests performed to create