Machine Learning

Machine learning models may provide magical predictions,...

Black Box Models

...but being able to explain how many machine learning models produce the predictions is not an easy task.

The Importance of Explanability

Literature on Explanability

General trends I've noticed:

• Many recent papers

• Often machine learning and computer science perspectives

• Lots of European authors

  • General Data Protection Regulation (GDPR) implemented in 2018
  • Goodman and Flaxman (2016): "It is reasonable to suppose that any adequate explanation would, at a minimum, provide an account of how input features relate to predictions, allowing one to answer questions such as: Is the model more or less likely to recommend a loan if the applicant is a minority?"

Key resources for this talk:

• Interpretable Machine Learning by Christoph Molnar
• Corresponding R package: iml
• Computational statistics working group at Ludwig-Maximilians-University in Munich led by Bernd Bischl (mlr and mlr3 R packages)

The Plan...

Setting the Stage

• Definitions and Philosophical Aspects

Methods

• Model Agnostic

• Random Forest Specific

• Neural Network Specific

Concluding Thoughts

• Additional Methods and Resources

• A Cautionary Conclusion

Explainability versus Interpretability

There are not agreed upon definitions...

Explainability versus Interpretability

My definitions (based on a conversation with Nick Street (University of Iowa))...

Explainability = the ability to use the model in an indirect manner to understand the relationships in the data captured by the mode

• LIME: model agnostic method that uses a surrogate model

Figure from LIME paper (Ribeiro 2016)

Should we explain black-box models?

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead by Cynthia Rudin:

• Debunks the “accuracy-interpretability trade-off” myth

• "Explanations must be wrong. They cannot have perfect fidelity with respect to the original model. If the explanation was completely faithful to what the original model computes, the explanation would equal the original model..."

• "...it is possible that the explanation leaves out so much information that it makes no sense."

• Rudin has worked on developing machine learning models with direct interpretability

Overview of Model Agnostic Methods

From Interpretable Machine Learning (Molnar)

Washington D.C. Bike Rentals

Example data in Interpretable Machine Learning - can be accessed here

bike <- load("data/bike.RData")
data(bike)
# Fit a random forest
bike_mod = randomForest::randomForest(x = bike %>% dplyr::select(-cnt), y = bike$cnt)

Partial Dependence Plots Friedman 2001

Purpose: Visualize marginal relationship between one (or two) predictors and model predictions

Estimated partial dependence function:

$$\hat{f}_{x_{int}}(x_{int})=\frac{1}{n}\sum_{i=1}^n\hat{f}(x_{int},x^{(i)}_{other})$$

• $\hat{f}=$ machine learning model fit using predictor variables
• $x_{int}=$ value of the predictor of interest
• $x^{(i)}_{other}=$ vector of training data values of other predictors in the model for observation $i$

Partial Dependence Plots in iml

# Create a "predictor" object that 
# holds the model and the data
bike_pred = 
  Predictor$new(
    model = bike_mod,
    data = bike)
# Compute the partial dependence 
# function for temp and windspeed
pdp = 
  FeatureEffect$new(
    predictor = bike_pred, 
    feature = c("hum", "temp"), 
    method = "pdp") 
# Create the partial dependence plot
pdp$plot() +
  viridis::scale_fill_viridis(
    option = "D") + 
  labs(x = "Humidity", 
       y = "Temperature", 
       fill = "Prediction")

Interactive PDPs Krause, Perer, and Ng 2016

Individual Conditional Expectation Goldstein et. al. 2013

Purpose: Similar to partial dependence plots, but consider each observation separately instead of taking an average.

Estimated individual conditional expectation function:

$$\hat{f}^{(i)}_{x_{int}}(x_{int})=\hat{f}(x_{int},x^{(i)}_{other})$$

• $\hat{f}=$ machine learning model fit using predictor variables
• $x_{int}=$ value of the predictor of interest
• $x^{(i)}_{other}=$ vector of training data values of other predictors in the model for observation $i$

ICE Plots in iml

# Compute the ICE function
ice = 
  FeatureEffect$new(
  predictor = bike_pred,
  feature = "temp",
  method = "ice")
# Create the plot
plot(ice)

Centered ICE Plots

"Sometimes it can be hard to tell whether the ICE curves differ between individuals because they start at different predictions. A simple solution is to center the curves at a certain point in the feature and display only the difference in the prediction to this point." - Molnar

# Center the ICE function for 
# temperature at the 
# minimum temperature and 
# include the pdp
ice_centered = 
  FeatureEffect$new(
  predictor = bike_pred,
  feature = "temp",
  center.at = min(bike$temp),
  method = "pdp+ice")
# Create the plot
plot(ice_centered)

Addressing Correlation and Interactions

ALE and Interaction Plots in iml

Code for the plots on the previous slide

# Compute the ALEs
ale = FeatureEffect$new(
  predictor = bike_pred,
  feature = c("hum", "temp"), 
  method = "ale", 
  grid.size = 40)
# Plot the ALEs
plot(ale) +
  scale_x_continuous(
    "Relative Humidity") + 
  scale_y_continuous(
    "Temperature")+
  viridis::scale_fill_viridis(
    option = "D") + 
  labs(fill = "ALE")

# Compute the interaction metrics
int = 
  Interaction$new(
    predictor = bike_pred,
    grid.size = 100, 
    feature = "season") 
# Plot the interaction metrics
plot(int) +
  scale_x_continuous(
    "2-way interaction strength")

Parallel Coordinate Plots

Provide a nice overview of the predictions across

PCP plot with bike data (made with ggpcp)

Parallel Coordinate Plots in ggpcp

Code for the plot on the previous slide

# Determine order of features 
bike_ft_ordered = bike_vi %>% arrange(desc(IncNodePurity)) %>% pull(var)
# Create the pcp
bike %>%
  mutate(rf_pred = predict(bike_mod)) %>% 
  ggplot(aes(color = rf_pred)) + 
  ggpcp::geom_pcp(aes(vars = dplyr::vars(all_of(bike_ft_ordered))), alpha = 0.4) +
  viridis::scale_color_viridis(option = "D") + 
  labs(x = "Featured ordered by feature importance (left to right)",
       y = "Standardized Feature Value",
       color = "Random Forest Prediction") + 
  theme(legend.position = "bottom") + 
  guides(color = guide_colourbar(barwidth = 15))

Interactive Parallel Coordinate Plots Beckett (2018)

R package [Rfviz] for interactive parallel coordinate plots with random forest models, but it could be extended to other machine learning models.

Permutation Feature Importance

Background

• Idea originally proposed by Breiman (2001) for random forests

• Adapted to be used with all models by Fisher, Rudin, and Dominici (2018)

  • Call this measure the model class reliance

Concept

• Measure feature importance by seeing how much the prediction error is affected when a feature is permuted

  • important feature: one that affects the prediction error when changed

  • non-important feature: one that does not affect the prediction error when changed

Permutation Feature Importance in iml

Permutation feature importance of bike data random forest

# Create the predictor 
# (seemingly FeatureImp 
# requires y)
bike_pred = 
  Predictor$new(
    model = bike_mod, 
    data = bike, 
    y = bike$cnt)
# Compute the feature
# importance values
bike_imp = 
  FeatureImp$new(
    predictor = bike_pred, 
    loss = 'mae')
# Plot the feature
# importance values
plot(bike_imp)

Permutation FI with p-values Altmann et. al. (2010)

• Permutation based feature importance method that returns p-values

• Example from the paper comparing Gini importance values to their permutation feature importance method with p-values

More Feature Importance Casalicchio, Molnar, and Bischl (2019)

Global Surrogate Models

Using a Tree as the Global Surrogate

Using a classification tree as the global surrogates for the random forest model fit to the sine data

Local Surrogate Model: LIME Ribeiro, Singh, Guestrin (2016)

LIME = Local Interpretable Model-Agnostic Explanations

• Consider one prediction of interest
• Use a surrogate model to explain the black-box model in a "local" region about a point of interest

LIME in R

Shapley Values Štrumbelj and Kononenko (2014)

Idea: Use game theory to determine contributions of predictor variables to one prediction of interest

Game Theory Connection: Shapley values are "a method for assigning payouts to players depending on their contribution to the total payout."

Shapley Values in iml

Interpretation: "The value of the $j$th feature contributed $\phi_j$ to the prediction of this particular instance compared to the average prediction for the dataset."

# Select obs of interest and perpare data
x_int = bike[names(bike) != 'cnt'][285,]
# Compute prediction values
avg_pred = mean(predict(bike_mod))
actual_pred = predict(bike_mod, newdata = bike[names(bike) != 'cnt'][285,])
diff_pred = actual_pred - avg_pred
# Compute shapley values
predictor = 
  Predictor$new(model = bike_mod, data = bike[names(bike) != 'cnt'])
shapley = 
  Shapley$new(predictor = predictor,
              x.interest = x_int)
# Create the plot
plot(shapley) +  
  scale_y_continuous("Feature value contribution") +
  ggtitle(sprintf("Actual prediction: %.0f\nAverage prediction: %.0f\nDifference: %.0f", actual_pred, avg_pred, diff_pred))

Quick intro to random forests

Idea: Aggregation of many trees (bootstrap data and randomly select predictors for each tree)

Feature Importance Plot

Mean decrease in impurity (gini importance): measures the average improvement in node purity for a predictor variable

# Extract the importance values
bike_rfimp <- bike_mod$importance
# Put the feature importance in a df
bike_vi <-
  data.frame(var = rownames(bike_rfimp),
             bike_rfimp) %>%
  arrange(IncNodePurity)
# Create a feature importance plot
bike_vi %>%
  mutate(var = factor(x = var, levels = bike_vi$var)) %>%
  ggplot(aes(x = var, 
             y = IncNodePurity)) + 
  geom_col() + 
  coord_flip() + 
  labs(x = "Feature")

Visualizing Sets of Trees Simon Urbanek (2008)

Cut points from all trees for two predictor variables

Visualizing Sets of Trees Simon Urbanek (2008)

Trace plots of all trees in a random forest

ggRandomForests Ehrlinger (2015)

R package for visually exploring random forests fit using randomForests or randomForest

library(ggRandomForests)

plot(gg_error(bike_mod)) + theme_gray()

plot(gg_vimp(bike_mod)) + theme_gray()

rfviz Beckett (2018)

Previously mentioned...R package for interacting with parallel coordinate plots for random forests

# Prepare data
rfprep <- rfviz::rf_prep(x = bike[names(bike) != "cnt"], y = bike$cnt)
# View plots
rfviz::rf_viz(rfprep, input = TRUE, imp = TRUE, cmd = TRUE, hl_color = 'black')

Forest Floor Visualizations Welling et. al. (2016)

Quick intro to random forests

Idea: Combination of many non-linear regression models

Image source

Feature Visualization overview article by Olah, Mordvintsev, and Schubert (2017)

Idea: Determine values of predictor variables that maximize activation functions at a specific "location" in the neural network

Formula Version: For a node in the network: $\ \ \ \underset{x}{\arg\max} \ f_j(x, w_j)$

• $x$ = values of predictor variables
• $w_j$ = estimated weights at node $j$
• $f_j$ = activation function used at node $j$

Image from Olah, Mordvintsev, and Schubert (2017)

Saliency Maps Simonyan, (2014)

Grand Tours Li, Zhao, and Scheidegger (2020)

Idea: Make use of the Grand Tour to visualize behaviors of neural networks

Image from Simonyan, Vedaldi, and Zisserman (2014)

More Methods

Overviews

Additional Resources for Overviews for Explainable Machine Learning

Gilpin et. al. (2019)

• Explaining Explanations: An Overview of Interpretability of Machine Learning

Mohseni, Zarei, and Ragan (2019)

• A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems

Ming (2017)

• A Survey on Visualization for Explainable Classifiers

Guidotti et. al. (2018)

• A Survey Of Methods For Explaining Black Box Models

Some thoughts on EML