We now show that using regular single-level regression techniques to address multilevel issues is fraught with problems. The main problem is that your results are likely to be filled with errors that originate from various violations of the regression assumptions. In many cases, we get poorly estimated results that will be statistically significant, leading us to reject our null hypotheses when we really shouldn’t be rejecting them. In other words, finding associations where none exist, erroneous conclusions, and possibly leading to ineffective policies.
Statistical reasons for multilevel modeling
There are many statistical reasons to choose multilevel modeling techniques over standard linear OLS regression. You may have accepted that you just need to learn this and don’t really care about all the technical reasons why, but we would argue that you should at least grasp the basic reasons why OLS is deficient in estimating models with nested data. You may have already tried some of the ‘workarounds’, which we discuss below, in OLS to model nested data. These ‘workarounds’ have been, and continue to be, used by many researchers and it is not difficult to find examples of them in the literature. They are still technically flawed, however, and we explain below why it is problematic to choose OLS, despite these ‘workarounds’, when trying to deal with nested data structures.
Assumptions of OLS
Multilevel modeling is an extension of OLS. The thing that makes multilevel modeling special is that it addresses specific violations of the ‘assumptions’ of OLS. Remember the assumptions? These are the conditions under which OLS regression works properly. All statistics have a set of assumptions under which they perform as they were intended.
One assumption is that the relationship between the independent and dependent variable is linear. Another is that the sample used in the analysis is representative of its population. Yet another is that the independent variables are measured without error. We more or less follow these assumptions in our day-to-day usage of OLS – and we should check that we are meeting some of these assumptions by doing regression diagnostics.
Dependence among observations
There are assumptions that relate to ‘independence of observations’ that we might think about less often. But it is this particular violation of the assumptions that multilevel modeling techniques are best suited to address. The assumption of independence means that cases in our data should be independent of one another, but if we have people clustered into groups, their group membership will likely make them similar to each other. Once people (Level 1) start having similar characteristics based on a group membership (Level 2), then the assumption of independence is violated. If you violate it, you get incorrect estimations of the standard errors. This isn’t just a niggly pedantic point. If you violate the assumptions, you are more likely to wrongly achieve statistical significance and make conclusions that are simply incorrect.
Perhaps an overlooked common-sense fact is that even if you don’t really care about group-level factors in your analysis (i.e. they aren’t part of your hypotheses), ignoring them does not make the problem go away. This is easy to demonstrate.
Suppose we are interested in how gender and parental occupational status influence academic achievement. Table 1.1 presents results from an OLS regression of reading scores on gender and parental occupational status. Reading scores are standardized within the sample to a mean of 0 and a standard deviation of 1. Gender is a dummy variable with 1 denoting males. Parental occupational status is a 64-category ordinal scale (the lowest value presenting low occupational status) that is treated as an interval-level variable here. The data come from the Australian sample from the 2006 Programme for International Student Assessment (PISA) organized by the Organisation for Economic Co-operation and Development (OECD, 2009). Data were obtained when all children were 15 and 16 years of age from schools in all eight states and territories of Australia.
Table 1.1N
b – unstandardized regression coefficients; s.e. – standard errors
* p < 0.05, ** p < 0.01, *** p < 0.001
Table 1.1 indicates that males (compared to females) have lower reading scores by 0.381 and that each unit increase in parental occupational status is associated with increases in reading scores of 0.020. These results are both statistically significant and have small standard errors. Our model has no group-level indicators, such as class or school. Just because we haven’t included group-level indicators, it does not mean that our problems of dependence among observations and thus correlated errors do not exist.
First, we need to predict our residuals from the regression equation whose coefficients we have just identified. Remember that the residuals are the difference between our predicted reading score based on these characteristics and the actual reading score we see in the data. Next, we can test if the assumption is violated if we run an analysis of variance (ANOVA) of these residuals by a grouping factor. Our grouping factor here is the region of Australia – the state or territory. If the residuals are independent of the regions, that is great – that means our errors are randomly distributed. This is not, however, the case in our data as the ANOVA returns a result of F = 55.8, df = 7, p < 0.001.
It might be helpful to think of it this way: we have several thousand students within the eight different regions in Australia. The ANOVA tells us that our individual-level results violate that assumption of uncorrelated errors because we find that the ANOVA gives statistically significant results. Table 1.2 shows the mean reading scores by region.
Table 1.2
We could assume that a ‘fix’ to this would be to add dummy variables to the model that represent the different regions. We add dummy variables to a model so that we can include nominal-level variables in our estimation. As the regions are nominal, we can then add them as a set of dummies with an omitted reference category. Table 1.3 shows how many students are in each region in Australia, while Table 1.4 displays the regression results for the model including the region dummy variables.
Table 1.3
As you can see in Table 1.3, students from this sample are dispersed among the eight regions of Australia. From reviewing the literature, we may have reason to believe that regional effects are determinants of academic achievement in Australia, as they are in many other countries around the world. For example, educational policies and resources are governed at the state level in Australia, and those regions containing the largest cities may offer the best resources for students (Australian Government, no date).
In Table 1.4 the gender and parental occupational status variables are the same as in Table 1.1. The region variable is entered as seven dummy variables, with the Australian Capital Territory as the omitted category.
Table 1.4N
b