Do you see any problems with this situation? It is a summary you may have read from your search of the literature with regard to evidence about the treatment of a condition like depression. There are no major flaws here. The process started with a purpose, data were collected consistent with that purpose, and conclusions were drawn that were consistent with the data analyzed.
You saw a reference to statistical analysis, which will be discussed many times in this book, but you are not expected to fully understand it at present. But you surely have seen references to statistical analysis of data and concepts such as “statistical significance.” These clients showed improvement, and the level of their improvement cannot be easily explained by chance (because the data were found to be statistically significant). If your data can be explained by chance, you cannot conclude that they can be taken seriously.
A reference was made to the generalization of data results, another concept that will be discussed many times in the chapters of this book. It was noted that the clients had not been selected at random, a basis for scientific generalization. It is likely that you have seen numerous references to random samples in your review of the literature. That is the superior form of sampling and strengthens the extent to which you can generalize your findings to persons who were not in your study.
Now, let’s examine another situation.
A group of researchers had the purpose of determining if after-school tutoring was effective in the improvement of grades for middle school students. Grades were compared for the period of time before tutoring was offered and measured again after tutoring was completed. The results showed that the average student improved but that statistical significance was not achieved (i.e., these results can too easily be explained by chance). These researchers concluded that tutoring is an effective way to improve grades for middle school students.
Do you see a problem with this situation? You need an appreciation of the theme of chance to grasp the lesson provided here. These researchers treated the data seriously in their conclusions even though the data failed to be statistically significant. The proper conclusion would be that the study failed to find an improvement in grades. What they discovered was chance. If chance can explain your data, you cannot take them seriously with regard to your study conclusions.
What do you think of the following situation?
Some fellow students conduct a study of students in one social work research class and find that the majority of them are identified with the Democratic Party. They conclude that there are more people in this university who identify as Democratic than Republican.
The problem here is easy to identify. You cannot generalize from a small sample of people in one class in a university to the entire university. The issue of generalization is discussed with regard to the nature of the sample that you employ. A small class of students in one specific place is not a random sample. And you would need a much larger sample than this to generalize to the entire university.
Now examine this situation.
Paul is a social work research student who starts the process of conducting research by deciding that he would like to conduct interviews of his fellow students. He has not yet decided on the purpose of his study.
What is the problem here? Well, Paul has put the cart before the horse. He has decided on the methods he will use to conduct his study before he decides on the purpose of the study. The purpose should come first. Why? Different purposes suggest different types of methods to be used. You need to know the purpose before you can make decisions on your methods. Interviews serve some purposes better than others.
Finally, let’s examine this situation.
Paula is an individual and family counselor who has been trained in New Beginnings Therapy (hypothetical). Paula has used this treatment for the past 3 months when the objective was to reduce stress. She believes that it has been effective with some clients but has not collected evidence on the outcomes with her clients. She examined the literature on evidence regarding the effectiveness of this approach to counseling. She found 1 study that revealed that New Beginnings Therapy was effective in the reduction of stress, and she found 23 studies that showed that it is not effective in this regard. Paula concluded that this therapy is effective because she found a study that showed that it is.
Is there a problem here? Is it okay to use only 1 study that shows a certain outcome and ignore 23 studies that show an opposite outcome? No, not if you are being scientific. Science is comprehensive. You use all the evidence that you can find, not just the ones that support your predetermined conclusion. Perhaps Paula could report on all the evidence and conclude that she will conduct her own study given the fact that there is more evidence against this approach to therapy than evidence that supports it. Perhaps she will find it to be effective with the types of clients she serves. The lack of sound evidence should encourage Paula to measure client outcome with her individual clients or to search for other methods of treatment.
The Nature of Science
According to the Science Council, science is defined as “the pursuit and application of knowledge and understanding of the natural and social world following a systematic methodology based on evidence” (http://sciencecouncil.org/about-us/our-definition-of-science). According to Wikipedia, science is “a systematic enterprise that builds and organizes knowledge in the form of testable explanations and propositions about the universe” (https://en.wikipedia.org/wiki/Science). It is not about hunches, or untested opinions, or something you were told.
Science is based on a special method of inquiry that includes clarity of the research question, a systematic method of investigation, the objective collection and analysis of data, and the drawing of conclusions that are logical based on the results of your data analysis. It is not the cherry-picking of facts to support an opinion, nor is it an incomplete inquiry with many critical avenues untouched by the inquiry. Instead, it is open and comprehensive.
Scientific research is a means of gaining relevant knowledge through the use of the scientific methods. The scientific method is orderly and strives for the achievement of objectivity. An inquiry that is designed to prove a point is not an example of research that fits the spirit of scientific inquiry. A study that is incomplete or illogical also fails to meet the standards of the scientific method.
Sometimes our observations from day to day are a good guide for action. Sometimes, however, our observations are in error. A typical error is a failure to see the evidence that contradicts an opinion that we have embraced. The scientific method is designed to reduce human error in observation. It moves in a logical sequence of steps. For example, we must have a clear idea of what we want to find out before we select a method for our inquiry. We should be clear about the nature of our target population before we select people for our study. Before we can draw conclusions about our research question, we must collect and analyze data. If we are abiding by the spirit of scientific inquiry, we will use study methods that provide us with the opportunity to find that our previous expectations about reality are not supported, as well as the opportunity to find support for our expectations.
Research methods can be used to test certain assumptions you have acquired in your work or questions that you have developed. When your perceptions are supported by your research efforts, you can have more confidence in their accuracy, and perhaps, you will learn something in this process that will refine your understanding. When your expectations are not supported by your research results, you can be stimulated to further growth by your efforts to rethink the problem under study. As you continue in this growth and the validation of practice principles and methods, you will be in a better position to meet the challenges of accountability in social work, which have been increasing in recent decades.
The spirit of scientific inquiry suggests certain behaviors. First, you would not state the purpose of your study as proving that your expectation is correct. You do not study the effectiveness of your tutoring program