Replication and Meta-Analysis
Although researchers seek to design studies to rule out alternative hypotheses, they cannot consider every possibility. When a study is performed in different laboratories with different participants, a process referred to as replication, we can have more certainty that the results found reflect the true nature of what we are studying. Thus, scientists seek to find a number of different studies from different laboratories that answer the same research question. For example, various studies from around the world have shown structural brain differences in individuals with schizophrenia including enlarged ventricles in the brain (Faludi & Mirnics, 2011). A number of journals, such as Clinical Psychology Review and Psychological Bulletin, are designed to publish reviews of research in the field.
replication: the process whereby a study is performed in different laboratories with different participants and obtains the same results
Once the literature in a particular area has been reviewed, it is possible to examine statistically the results of all the studies taken together. This technique is referred to as meta-analysis. Meta-analysis is a statistical technique for combining a number of studies to improve the reliability of the results. For example, a large number of studies have examined depression and how it can be treated with cognitive behavioral therapy (Butler, Chapman, Forman, & Beck, 2006). With a meta-analysis, it can be asked, what if we consider all of these studies to be one study. Then we could calculate the common effect size of all of the available studies. A similar meta-analysis was performed to examine the effectiveness of interpersonal psychotherapy for depression (Cuijpers et al., 2011). While the use of meta-analysis is often invaluable, the LENS below points to a potentially complicating factor when undertaking such a review of previously published studies.
meta-analysis: statistical examination of the results of studies taken together and treated as one study
Lens
Treatment and Clinical Perspectives: Failure to Publish the Results of All Clinical Trials Is Hurting Medical Science
Why do so many clinical studies go unpublished?
© iStockphoto.com/triloks
Imagine that you have just set up a new program for treating an eating disorder. How do you know the new treatment you developed works? The gold standard is to perform a randomized control trial involving the treatment. In this case, you would randomly select individuals with a particular eating disorder who would either receive your treatment or an alternative, usually a placebo. Once the study is complete and your treatment works, what do you do next?
If you are like most scientists, you write up the results and publish them. However, if your treatment did not show a significant difference in comparison to the alternative, what do you do then? The ideal response is to publish the results so that the world knows not only what works, but also what does not work. However, publication of negative results does not always happen. Sometimes, scientists move on to more productive projects. This failure to publish negative results has come to be called the file drawer problem. This phenomenon is a significant complication if you do a literature search. Typically, your literature search shows you the treatment studies in which the treatment made changes in the disorder. However, what you do not see are those studies that did not find a significant change, as they remain unpublished in the researcher’s file drawer or computer.
Why would so many studies go unpublished? Researchers, based on their own treatment preferences, might not like the results of a study that didn’t verify certain treatment effectiveness. Large drug companies encourage the publication of studies whose results supported their own interests. Journals choose to publish research articles that found positive results. All of these factors contributed to the so-called file drawer problem.
To help deal with this problem, in 2007 the U.S. government passed an amendment to the Food and Drug Administration Act. This requires clinical trials of drugs, medical devices, or biologics to be registered at the website ClinicalTrials.gov. Further, a basic summary of the results is required to be submitted through the website within 1 year following the completion of the data collection. This was considered to be an ethical obligation to human participants to present results in an understandable fashion.
In order to determine the rate of compliance with the law, Monique Anderson and her colleagues (2015) examined 13,327 studies that ended during the period from 2008 to 2012. Of these, over 77% were drug trials. Of all the trials, only 13.4% reported summary results within 1 year of completing the research. When using a 5-year time frame, 38.9% reported summary results.
A British physician, Ben Goldarce, is part of an international campaign called ALLTrials, which seeks to require researchers worldwide to publish their results. He noted that an antidepressant he had prescribed for his patients showed positive results in the published literature. Whereas the published literature showed this antidepressant to be as effective as any other antidepressant drug, unpublished data involving 3 times as many people as the published data did not find the same results. A similar situation happened for another type of antidepressant drug, referred to as SSRIs (selective serotonin reuptake inhibitors). Prozac is one example of this class of drugs. Although these were introduced in the 1980s, it was not until 2006 that data submitted to the FDA showed a relationship between the drug and suicidal ideation in adolescents. Further, only 51% of the studies submitted to the FDA showed that use of SSRIs led to a positive result, whereas 94% of published studies showed positive results.
Thought Question: What role should the U.S. federal government play in ensuring treatment data from all clinical trials are available to the public?
(Based on “Failure to Publish the Results of All Clinical Trials Is Skewing Medical Science,” The Economist, July 25, 2015)
Concept Check
What kinds of research questions could you explore using each of the following research designs? What are the advantages and disadvantages of each?Single-subject designLongitudinal researchEpidemiological research
What is the overall goal of behavioral genetics research? What are the three primary types of research designs used in behavioral genetics research?
Which is more important, statistical significance or clinical significance? Why?
If a research study has already been conducted and its results have been communicated, why should it be replicated?
Ethics and the Scientific Experiment
Ethics is the study of proper action. Ethics examines relationships between human beings and provides principles regarding how we should treat each other. The ultimate decision in ethical questions resides in judgments of value. Ethical considerations of psychological experimentation have at their heart the idea that people participating in research should not be harmed (see Ray, 2012). Specifically, at the end of an experiment, participants should not be affected in a way that would result in a lower level of human functioning. This includes emotional distress.
ethics: the study of proper action
In most cases, the scientist has a question that he or she wants to ask and that the participant is willing to help answer. In some cases, the participants learn something about themselves from the experience and they are glad to have participated. In brain imaging studies, for example, participants often report that they enjoy seeing their brain activity (e.g., functional magnetic