II. Simple Regression in SPSS

Let’s go back to chapman_1.sav that we had seen earlier.

 

Here is SPSS output for the simple regression analysis.

The following part just tells us what $X$ we are using (i.e., age)

Here we see the $r, R^2$ and adjusted $R^2$ for the data as well as the SEE.

$r = .411$

$r^2 = R^2= .169$

The correlation r is the correlation between $X$ and $Y$. However, it is also the correlation between $Y$ and $\hat{Y}$. Therefore r is telling us about how close the predicted values are to the observed values of $Y$. Similarly $r^2$ or $R^2$ is the square of that value, r. Clearly, we want r to be big and also we want $R^2$ to be big.

When we have several $X$, then $R^2$ will be the correlation between the $Y$ and predicted values that are based on all of the $X$ – so it is a measure that captures more than the set of individual $r_{xy}s$ for the $X$ in our model. $R^2$ is called the coefficient of determination.

We can interpret $R^2$ as the proportion of variance in $Y$ that is explained (= SSR/SST). In other words, $R^2$ is about how well all of the model with the independent variables is working to predict the dependent variable, therefore, we would like $R^2$ to be close to 1. Here, with $R^2 =.31$ we have explained about a third of the variation in $Y$ scores with this $X$.

We will learn more about the following table later, but for now, we use it to find the MSE (see the red circle below).

MSE = 3542.486

SEE = 59.435

The MSE (see the ANOVA table) is the “Mean Squared Error” - actually this is the name for the variance of the $e_i$ values around the line. The closer the values are to the line, the smaller MSE will be. It is good for MSE to be small.

We can compare MSE to the variance of $Y$. Here the variance of $Y$ is 4230.31, so our MSE of 3542.486 is a decent amount lower than ${S_Y}^2$.

The SEE or standard error of estimate is just the SD of the $e_is$, or $\sqrt{MSE}$. Therefore, here SEE = 59.435 and can be compared to $S_Y$, which is 65.041. This is telling us the same info as MSE, but SEE is in the score metric (not the squared metric). You can guess – SEE should also be small.

The following table contains the slope $(b_1)$ and Y-intercept $(b_0)$ as well as t-test of their values.

Based on the above table, an estimated regression model is written. We can call it a regression equation or regression line: $\hat{Y} = b_0 +b_1X$, where $\hat{Y}$ is the expected value for the dependent variable.

As shown in the above scatterplot, SPSS computed it as

Okay, let’s interpret Y-intercept $(b_0)$ from the above table. Here is a generic regression equation: $\hat{Y} = b_0 + b_1X$.

• $b_0$ is called the Y-intercept.
• $b_0$ is the expected value of $Y$ when $X = 0$.
• This value is only meaningful when $X$ can have a realistic value of zero.

Expected cholesterol level $= b_0 +b_1 \ast age$

Plug the estimated value that the Coefficients table has in the equation (i.e., 187.383):

Expected cholesterol level = 187.383 + 2.296*age

• $b_0$ tells us that the expected cholesterol for patients at age 0 is 187.383.
• First, what do you think of age 0? The average mean cholesterol level at age 0 is actually 70 (American Committee of Pediatric Biochemistry, 2019).
• Second, the approximate value 187 on the Y-intercept is telling us that the patients’ cholesterol level would be 187 on average when age is 0. Therefore, for this case, treat this Y-intercept as one point on the regression line, but not one that is very relevant.

Now, let’s interpret regression coefficient or slope $(b_1)$ from the above table. Here is a generic regression equation: $\hat{Y} = b_0 + b_1X$.

• $b_1$ is called the regression coefficient.
• $b_1$ is the expected change/difference in $Y$ for a one-unit increase in $X$.
• Direction: Look at sign in front of $b_1$.
• Strength: The higher the value of $b_1$, the more $Y$ responds to changes in $X$.

Expected cholesterol level $= b_0 +b_1 \ast age$

Plug the estimated value that the Coefficients table has in the equation (i.e., 2.296):

Expected cholesterol level = 187.383 + 2.296*age

• As the age of a patient increases by 1 (say, from 21 to 22), their cholesterol level increases by 2.296.
• For each one-unit increase in age, patients’ cholesterol level increases by 2.296.

III. Standardized Slopes and Regression Model Tests

Earlier, we estimated the regression model. Here we get the values of $b_0$ and $b_1$ plus other indices we saw earlier (MSE, SEE), and also standardized slopes.

1. The F-Test for the Overall Model

We test the overall relationship using the F test. We learn how to test whether the model as a whole (the set of $Xs$ taken together) explains variation in $Y$ (the omnibus test = Overall test). If the overall relationship is significant, then, continue with the description of the effects of individual $Xs$.

For the test of significance of the overall relationship - a test of $H_0: \rho^2 = 0$ or technically of

$H_0 = {\rho^2}_{Y\hat{Y}} = 0$

that is, a test of whether the outcome and predicted values are related. While in practice we test the overall model quality, before testing individual slopes.

When we do the F test we want it to be large. We make the test as F = MSR/MSE. Also, F = SSR/MSE when we have one $X$. We want SSR or MSR to be large, and MSE to be small. Both of those lead to a large F test.

So how do we do the F test? Let’s use our chapman_1.sav data and compute

F = MSR/MSE = SSR/MSE = 142399.403/3532.486 = 40.311

Then use the F table with the right df. In our original example, we have F(1,198). The F table in most statistics books shows that the critical value for the .05 test is 2.705. Our F is above the critical value, therefore, we reject $H_0$. Also the printed $p =.000 < .05$ which gives the same decision.

Now, how do we interpret the F test? For our champan_1.sav data we reject $H_0$, which we said was either:

$H_0: {\rho^2}_{Y\hat{Y}} = 0$ or $H_0: \beta_1 = 0$

therefore,

$H_0: {\rho^2}_{choles.\hat{choles.}} = 0$ or $H_0: \beta_{age} = 0$

The value F = 40.311 would occur less than 5% of the time if either of these $H_0$ was true, and that is too unusual. So we decide probably these $H_0$ is not reasonable. In other words, our $X$ probably predicts $Y$ very well, so we decide $X$ has a nonzero slope $(\beta_1 \neq 0)$ thus predicts $Y$, or the predicted $Y$ values relate to the actual $Ys: \rho^2 \neq 0$.

2. The T-Test for the Individual Slope Tests

We have the test of whether a slope is zero. For the test of significance of the individual slope - a test of $H_0: \beta_1 = 0$ is conducted. So far we have only one $X$, so the test of $\beta_1 = 0$ (the t-test) will give the same decision as to the overall F test.

We want to test whether the relationships that we see in the sample exist in the population. Researchers can develop the null hypothesis (i.e., $H_0$), which means that we believe that there is no relationship between $X$ and $Y$. By virtue of you decide to conduct the study, you will always have a research hypothesis, which means that we believe that there is a relationship between the variables in question.

Before we examine any tests, we will stop to consider how to decide if a slope is important. The importance here is independent of statistical significance – this relates to practical importance and to what is known about the outcome in the literature (not in your sample). Also, we draw on conventional “rules of thumb.”

You may do this when comparing two different models (e.g., for two different $X$ variables), and in multiple regression, we will eventually want to choose among the predictors in one model to identify the most important $X$ (i.e., compare the magnitudes of $X$s).

To do this we use the standardized regression equation. The slope $b_1$ depends on the scale of the $X$ variable as well as that of $Y$. Therefore, if we want to compare slopes, we need to be sure they represent variables on the same scale. Unless two $X$s have the same scale, we cannot compare their slopes.

Therefore, we need to equate the scales of the $X$s. Note that, with the same logic, we would not able to compare the magnitudes of weight and height variables in raw scales because they use different scales, but once we standardize them, we can compare the magnitudes of these on $Y$ because standardized variables can be compared each other.

In sum, once we see whether or not the regression coefficients (i.e., unstandardized coefficient) are statistically significant, then we would like to see standardized coefficients because we want to compare the magnitudes of the significant variables on the $Y$ outcome.

Here, the standardized coefficients are called beta-weights. The beta-weights are about how well independent variables are pulling their weight to predict the dependent variables.

Beta-weight is the average amount by which the dependent variable (DV) increases when the independent variable (IV) increases one standard deviation, controlling for all other IVs (held constant).

Think of multiple regression that has multiple $X$s. We want to compare the magnitudes of$X$s and to choose which one is the most important on $Y$. We use beta-weight for it because it is scale-free.

We interpret beta-weight as follows: When $X$ (IV) changes (either increases or decreases) by one SD, the y-hat (DV) changes by beta-weight SD, holding the other variables in the model constant.

We say the standardized slopes (${b\ast}_1$ ) is “the predicted number of standard deviations of change in $Y$, for one-standard-deviation unit increase in $X.$” So if ${b\ast}_1 = .411$ we predict that $Y$ will increase by .411 of a SD, when $X$ increases one SD.

In contrast, regression coefficients are expressed in the units of measurement of Y-hat (DV), and the units may not be comparable (e.g., age and blood pressure predict cholesterol but how can you compare the strengths between age and blood pressure because they have different units?). To make the scales comparable we compute a new score:

$Z(X_i) = \frac{(X_i = \overline{Y})}{S_X}$

where we subtract from each $X_i$ the mean and divide by the SD of $X$. Recall that Z scores always have mean 0 and variance 1.

If we do this for all the $Xs$ (and $Ys$) the slopes computed based on those new Z scores will be comparable.

For single-predictor models this gives us a very simple equation:

$Z(Y_i) = {b\ast}_1 Z(X_i) + {e\ast}_i$

where ${b\ast}_1$ is the so-called “beta weight”. Note the intercept is 0 because both $Z(X)$ and $Z(Y)$ have means of 0.

We might also write the formula for the standardized regression line. As before we need to put a hat over the outcome to show it represents a predicted value.

$\hat{Z(Y_i)} = {b\ast}_1 Z(X_i)$

where $\hat{Z(Y_i)}$ = The predicted standardized score on $Y_i$ for case $i$.

${b\ast}_1$ = the “beta weight” or “standardized coefficient”, the number of standard deviations of change predicted in $Y$ for one standard-deviation increase in $X$.

$Z(X_i)$ = The standardized score on $X_i$ for case $i$.

One last thing is true of beta weights in one-predictor regressions. That is, the standardized regression slope is equal to the correlation of $X$ with $Y$. Therefore, ${b\ast}_1 = r_{XY}$  for the equation $Z(Y_i) = {b\ast}_1 Z(X_i) + {e\ast}_i$.

A standardized coefficient is not equal to the correlation between $X$ and $Y$ when we get to the multiple regression context. However, for bivariate (one-predictor) regression this means we can use rules of thumb for correlations to help interpret the sizes of the beta weights.

Jacob Cohen provided a set of values that have been used to represent sizes of correlations, mean differences, and a lot of other statistics. These came from evaluations of the power of tests – but they have been widely used in social sciences in other situations.

For correlations, beta weights with only one $X$, and $r^2$ the values are

	r or b*	r2
small	.10	.01
medium	.30	.09
large	.50	.25

 

[Exercise 2]

$R^2$ is about how well ALL of the IVs are working together to predict the DV. With an $R^2$ of .40, this means that 40% of the variation in DV (40% of the DV’s predictability) is explained by all the variables in the model. To understand what other UNKNOWN factors might help predict the DV, the researcher would consider what other variables to add to the model.

With an $R^2$ of .40, what percent (%) of the variation in DV (60% of the DV’s predictability) is UNEXPLAINED by all the variables in the model?

With an $R^2$ of .40, can you evaluate if the model predicts the DV well, according to Cohen’s rules of thumb?