In the literature, you will see descriptions of effect sizes as small (.20), medium (.50), and large (.80 and above). Jacob Cohen (1988) is usually the source cited. These three sizes represent different degrees of overlap: 85% (small), 67% (medium), and 53% (large). You can see that these percentages of overlap relate to the idea that there is a lot of overlap when the effect size is small and much less when the effect size is large.
Without doing a power calculation, you can still get some sense of the sample size needed in your topic area (with implications for power) by reading the literature. More participants are needed if the effect size reported in your topic area is small. If no effect size is reported for your area of study, you could make a guess about whether you think it is likely to be small, medium, or large. Without information to the contrary, a conservative estimate (i.e., a small effect size) is probably prudent. Cohen (1988) has published tables that indicate how many participants you will need to detect a difference (i.e., to reject Ho, assuming it is false) for a specific effect size. Power is discussed further in Chapter 9, including the use of an online power calculator.
Revisit and Respond 3.3
Explain why having sufficient power is important for a study.
What are the four factors from Howell (2013) that affect the power of a research design?
What is the most common way to increase power?
In your own words, describe what an effect size is. How is power related to effect size?
Internal Validity
The validity of research refers to the degree to which the research evaluates what we claim it does. When we talk about the internal validity of research, we are talking about the degree to which the research was conducted in a manner that allows us to rule out alternative explanations; in other words, we are talking about the quality of the research process. Researchers are always vigilant to the occurrence of what are called threats to internal validity (additional types of validity related to measures are discussed in Chapter 5). A well-known list of these threats was produced by the researchers Donald Campbell and Julian Stanley in 1963.
Table 3.1
Source: Adapted from Campbell, D. T., Stanley, J. C., & Gage, N. L. (1963). Experimental and quasi-experimental designs for research. Hougton Mifflin.
Threats to internal validity: Factors that undermine the ability of your research to ascertain the influence of an independent variable (IV) on a dependent variable (DV).
Table 3.1 presents these threats to internal validity discussed by Campbell and Stanley (1963).
Let’s expand on each one of these threats and identify some others. In truth, some of these threats are beyond your ability to control, but at least you will be aware of them.
History
History is one of those threats you can’t control. Events in the world that occur during the course of research (e.g., terrorism or illness) may impact respondents’ answers to your surveys or problem-solving skills, as examples. If your sample is large enough, events specific to individuals are likely to be randomly distributed across conditions. In the case of terrorism or some other event that affects the population, you may not be able to tell whether the event interacted with and differentially affected the responses of one group relative to another, especially if the sample size is small.
History: One of Campbell and Stanley’s (1963) threats to internal validity in which something happens between experimental treatments to influence the results.
Maturation
In the case of maturation, you have changes to participants that affect their performance. One aspect of maturation you can control is fatigue. If a long battery of measures is administered to participants (for example, of 60 minutes), they may lose interest, and their performance on the later measures would be different than if there were fewer measures. As a researcher, you can pilot test your questionnaire batteries and ask for feedback about fatigue. Solutions might be to administer the questionnaires in two sittings or to reexamine whether all of those surveys are, in fact, essential.
Maturation: One of Campbell and Stanley’s (1963) threats to internal validity in which capacities of the participants may change as a result of fatigue, illness, age, or hunger that affect the intervention.
Testing
In testing, exposure to one instrument during a pretest or to an assessment that comes between a pretest and a posttest can change people’s responses to the posttest. To determine whether this is the case, some researchers use what is known as the Solomon four-group design (see Chapter 10), which evaluates what would happen with and without the pretest. This approach is expensive in terms of time and resources because there are four groups to run. Another option is to consider whether the pretest is needed.
Testing: One of Campbell and Stanley’s (1963) threats to internal validity involving multiple testing situations in which the first test affects how participants respond to subsequent tests.
Solomon four-group design: Pretest, posttest research design involving four conditions; takes into account the possible effect of sensitization in responding to the pretest measures.
Instrumentation
Instrumentation is a threat that is more easily managed, in principle. With regard to instrumentation, for example, the conditions of projecting images for participants to view, being prepared is the best course of action. Knowing how to use all of the equipment is important (e.g., what to do when you get a “no signal” message for LCD projection or what to have participants do when a survey link doesn’t load). Routinely checking calibration of equipment is advisable.
Another possible threat to internal validity in the category of instrumentation involves your measures. Make sure you include all of your items, and make sure that your participants have looked at all the pages of the questionnaire, if you are administering a paper version. When administering questionnaires online, it is possible to prompt participants to check that they have answered all of the items they intended to answer; such prompts help cut down on missed items.
Another kind of “instrument” is the researcher. If the researcher is giving task instructions, it is important to follow a script to make sure every participant receives the same information. Some researchers record instructions and other material delivered in spoken form to ensure that participants hear the same speaking voice, with the same pace.
Instrumentation: One of Campbell and Stanley’s (1963) threats to internal validity in which changes in equipment and/or observers affect judgments/measurements that are made.
Operational definitions: Describes a variable in terms of the processes used to measure or quantify it.
Statistical regression: One of Campbell and Stanley’s (1963) threats to internal validity when participants are selected on the basis of extreme scores (e.g., high or low intelligence) and their scores move toward the mean on subsequent testing.
Differential selection (biased selection of subjects): One of Campbell and Stanley’s (1963) threats to internal validity in which participants assigned to groups