Tutorial 10: Regression Modelling III

The concepts of probabilities and odds are fundamental in statistics, and while they are related, they describe different aspects of likelihood:

Probabilities, a re-cap:

Probabilities quantify the likelihood of an event happening, measured on a scale from 0 to 1, where 0 indicates impossibility and 1 indicates certainty. The probability of an event is calculated as the number of favourable outcomes divided by the total number of possible outcomes. For example, the probability of rolling a 3 on a standard six-sided die is 1/6, as there is one favourable outcome (rolling a 3) and six possible outcomes in total.

Odds

Odds, on the other hand, compare the likelihood of an event happening to the likelihood of it not happening. They are expressed as a ratio of two numbers: the number of favourable outcomes to the number of unfavourable outcomes. Using the dice example, the odds of rolling a 3 are 1 to 5, because there is one favorable outcome (rolling a 3) and five unfavorable outcomes (rolling a 1, 2, 4, 5, or 6).

There are some key differences, these are: 

• Representation: Probabilities are represented as a fraction of the total number of outcomes, while odds are represented as a ratio of favourable outcomes to unfavourable outcomes. 

• Range: Probabilities range between 0 and 1, whereas odds range from 0 to infinity. A probability of 0.5 corresponds to even odds (1:1), but as probabilities approach 1, the odds increase without bounds. 

• Interpretation: Probabilities provide direct information about the likelihood of an event within the universe of all possible outcomes. Odds provide a comparison of the event happening versus not happening.

Logistic regression is a way to explore the relationship between a series of factors (like age, income, and education) and a particular outcome that has two possibilities (like voting for candidate A or not voting for candidate A). The tricky part is that this outcome is like an on-off switch (0 or 1), which doesn't fit well with regular straight-line (linear) predictions.

To handle this, logistic regression takes the on-off switch and stretches it out on a scale from 0 to 1 using logs. This stretching turns our simple on-off outcome into something that can be mapped with a curve, allowing us to use a linear approach. In essence, it transforms the prediction problem into figuring out the odds of the switch being on (1) or off (0), based on the factors we're looking at. By using logs, logistic regression can make predictions and show how different factors increase or decrease the likelihood of the outcome happening.

The output of a Logistic Regression looks very similar to that of a Linear Regression. There are, however, some significant differences. 

In logistic regression, the coefficients represent the log odds, which is a way of quantifying the relationship between each predictor variable and the probability of the outcome occurring. Log odds can be a bit tricky to interpret directly, but they essentially tell us how the odds of the outcome change with a one-unit increase in the predictor variable. When you exponentiate a coefficient, you get the odds ratio, which is easier to understand – it shows the factor by which the odds of the outcome increase (if the odds ratio is greater than 1) or decrease (if it's less than 1) for a one-unit increase in the predictor. Most programmes will also provide you with the OR which is easier to understand and interpret. 

Because logistic regressions don't have an equivalent to the R-squared statistic from linear regression (which measures how well the model explains the variability of the data), we often use a Pseudo R-squared measure instead. Pseudo R-squared values provide a way to gauge the model's explanatory power, but they don't have a direct interpretation like the R-squared in linear regression. There are several types of Pseudo R-squared, with Nagelkerke’s R-squared being one popular option. These measures give us a rough idea of how well our model fits the data, but they should be interpreted with caution and understood as not directly comparable to the R-squared from linear regression.

Interpreting Pseudo R Squared

There are several different Pseudo R Squared below are two commonly used ones. Pseudo R-Square typically ranges from range from 0 to just under 1. However, its values are generally much lower than what you would expect from a linear regression R-squared. A value of 0 indicates that the model has no explanatory power, and values closer to 1 indicate a better fit.

McFadden's R-Squared

• Interpretation: A rough guideline for McFadden's R-Squared is that values around 0.2 to 0.4 represent a good fit. However, these are very rough benchmarks, and interpretation should consider the specific context of the model and the data.

Nagelkerke’s R-Squared

• Interpretation: Similar to McFadden's R-Squared, higher values indicate a model that better fits the data. However, because Nagelkerke’s R-Squared is scaled to the full 0 to 1 range, its values might appear more intuitive, resembling the R-squared values from linear regression. Still, values will generally be lower than those typically seen in linear regression models.

Key Points for Both

• Context is Critical: The interpretation of both McFadden's and Nagelkerke’s R-Squared should always consider the context of the study and the complexity of the model. There are no hard-and-fast rules for what constitutes a "good" value.

• Comparative Use: These measures are most useful for comparing the fit of different models for the same data set rather than as absolute indicators of model quality. A model with a higher pseudo R-squared value is generally considered to have a better fit compared to a model with a lower value.

• Not Directly Comparable to Linear R-Squared: Despite the intuitive appeal of Nagelkerke’s R-Squared, it's important to remember that these measures do not have the same interpretation as the R-squared from linear regression. They do not represent the proportion of variance explained by the model in the same way.

In summary, McFadden's and Nagelkerke’s R-Squared values provide useful, albeit rough, measures of model fit in logistic regression, with Nagelkerke’s adjustment offering a scale that is perhaps easier to interpret for those familiar with linear regression. However, their interpretation should be contextual and cautious, particularly when comparing across different types of models.

Above you see the typical output of a Binomial Logistic Regression. You can of course also create additional visualisations, Marginal Means plots and tables etc. 

The R-Squared for example indicates that this might be an okay fit, for the data McFadden (0.21). 

Interpreting the Output

The Intercept

The intercept (−0.83) represents the log odds of being in the "Extreme" category when all predictors are at their reference levels (zero or baseline category). The odds ratio of 0.43 suggests lower odds of being "Extreme" at the baseline levels of predictors.

Predictor Coefficients

Each predictor’s estimate represents the change in the log odds of being "Extreme" for a one-unit increase in that predictor, holding all other predictors constant.

Impact_Globalisation: A coefficient of −0.25 indicates that as the impact of globalisation increases by one unit, the log odds of being "Extreme" decrease, with an odds ratio of 0.78 suggesting a decrease in odds.

Age: For each one-year increase in age, the log odds of being "Extreme" decrease by 0.03, with the odds ratio of 0.97 indicating a slight decrease in odds.

Gender (Female as the reference group): Being male (compared to female) increases the log odds of being "Extreme" by 1.07, translating to an odds ratio of 2.91, suggesting that males have higher odds of being "Extreme" than females.

Political_LeaningRight_Left: A coefficient of −0.35 means that moving from right to left in political leaning decreases the log odds of being "Extreme", with an odds ratio of 0.71 indicating reduced odds.

Significance (p-values)

• p-values below 0.05 (e.g., Age, Gender: Male – Female, Political_LeaningRight_Left, and several categories under "Share_nothing_Society") indicate statistically significant effects.

• Larger p-values suggest the evidence is weaker for a predictor's effect on the outcome (e.g. the various strain variables).

Odds Ratios

• Values greater than 1 (e.g., Gender: Male – Female) indicate increased odds of the outcome ("Extreme") with each unit increase in the predictor or moving to the specified category.

• Values less than 1 (e.g., Impact_Globalisation, Age, Political_LeaningRight_Left) indicate decreased odds of the outcome with each unit increase in the predictor.

Strains and Social Factors

• Different "Strain" factors and "SocialMedia_Use" show varying influences, with most not significantly affecting the odds of being "Extreme", as indicated by their p-values and odds ratios close to 1.

• Share_nothing_Society: Strong opinions (either disagreeing or agreeing strongly with sharing nothing in society) significantly decrease the odds of being "Extreme", highlighting how certain social attitudes are associated with the outcome.

Has Degree Yes/No

Having a degree (Yes – No) with a coefficient of −0.39 and an odds ratio of 0.68 suggests that having a degree slightly decreases the odds of being "Extreme", though this effect is not statistically significant (p = 0.105).

This table offers a comprehensive view of how various factors influence the likelihood of being categorized as "Extreme", combining individual, social, and political predictors to provide insights into the factors associated with extreme responses.

Below we can see a visual representation of Age, Gender and Political Leaning. 

Multinomial logistic regression is a statistical method used when you want to predict outcomes that fall into three or more categories, which are not ordered. For example, if you're studying voters' preferences for different political parties (e.g., Party A, Party B, Party C), multinomial logistic regression can help you understand how different factors (like age, income, or education) influence someone's likelihood of preferring one party over the others. 

Unlike binary logistic regression, which deals with outcomes that have two possible states (yes/no, win/lose), multinomial logistic regression handles multiple categories by comparing the log odds of being in one category to the log odds of being in a baseline category, for each predictor variable. This allows you to model and predict more complex relationships where the outcome isn't just a simple yes or no but includes several distinct options.

They are generally interpreted the same way as a binomial logistic Regression. The big difference is that it will provide you a breakdown for each level compared with the other levels. 

Here you can see is give us the output of each level in the table compared t the reference level which is Strongly Agree. 

Again you can visualise the output using Marginal Means plots and tables. 

In ordinal logistic regression, which is used for predicting an ordinal dependent variable (a variable with categories that have a natural order, but no consistent interval between them), model thresholds (or cut points) play a crucial role. These thresholds separate the categories of the dependent variable based on the predicted probabilities.

Understanding Model Thresholds

Model thresholds are values derived from the model that define the boundaries between the ordinal categories. For an ordinal outcome with k categories, there will k-1 thresholds. These thresholds are estimated during the model fitting process and are used to determine the predicted category for each observation based on its predicted probabilities.

Interpreting Model Thresholds

• Location of Thresholds: Each threshold marks the point along the continuum of the linear predictor (the combination of predictors weighted by their coefficients) at which the probability of being in a given category or below shifts to being more likely than being in a higher category. The first threshold marks the boundary between the first and second categories, the second threshold between the second and third categories, and so on.

• Predicted Category Assignment: To assign a predicted category to an observation, the model uses these thresholds along with the observation's values on the predictor variables. If an observation's linear predictor score falls below the first threshold, it's predicted to be in the first category. If it falls between the first and second thresholds, it's predicted to be in the second category, and so forth.

• Significance of Thresholds: The exact values of the thresholds tell you about the distribution of the predicted probabilities across the categories. Larger gaps between thresholds can indicate areas where the categories are more distinctly separated by the predictor variables.

Model thresholds in ordinal logistic regression offer a mechanistic insight into how the model discriminates between the ordered categories based on the predictor variables. Understanding and interpreting these thresholds help in grasping the underlying dynamics of the model predictions and the influence of predictor variables on the ordinal outcome.