Explainability of Graph Neural Networks
王翔
University of Science and Technology of China
2023.03.02

Why Graph & Graph Neural Networks (GNNs)?
Background
Protein Structure
• Graph data are everywhere.
Social Network Knowledge Graph Transaction Network
• Graph neural networks (GNNs) are popular.
• Powerful representation learning:
• incorporate graph structure with node/edge features in an end-to-end fashion;
• Impressive performance:
• graph classification, node classification, link prediction, graph matching …
……

Why Explainability?
Motivation
• GNNs work as a black box.
Input
Output
• Call For Explainability!
• Given a GNN model, how can we interpret to users the model outcome?
• ”What knowledge should/does the model use to make decisions?”
• Knowledge: insights for a particular audience into a specific problem.
“The black box is an algorithm that tasks data & turns it into
something. The issue that black boxes often find patterns without
being able to explain their methodology.”
It cannot be fully trusted, especially in applications on safety, security!

Why Explainability?
Type I: Post-hoc Explainability
GNN Model 𝒇 Output Prediction "
𝒚
Input Graph 𝑮
Core of GNNs
Post-hoc Explainability
Which fraction of the input graph is
most inﬂuential to the model’s
decision?
Explanatory Subgraph 𝑮𝒔
Input Graph 𝑮
Output Prediction "
𝒚
Use an additional explainer method to exhibit what a model learns as a black box
• Emulate the decision-making process of the target model

Post-hoc Explainability
Our Work 1: Explaining Graph Neural Networks
Post-hoc Explainability
Which fraction of the input graph is
most inﬂuential to the model’s
decision?
Explanatory Subgraph 𝑮𝒔
Input Graph 𝑮
Output Prediction "
𝒚
Screen Graphs of an Image Subgraphs with Top Gradient & Attention Scores
Wang et al. Reinforced Causal Explainer for Graph Neural Networks. TPAMI’2022

Post-hoc Explainability
Our Work 1: Drawbacks of Existing GNN Explainers
Drawbacks of Gradient- & Attention-based Explainers
Our Goal
• Causation:
• We would like to identify the subgraph that may plausibly be the causal determinants of the model outcome
• e.g., (standing, on, surfboard).
• Conciseness:
• We need concise explanations to avoid redundancy, considering the dependencies of interpretability across edges.
• Spurious correlation:
• Due to the confounding associations (human-related objects),
some edges are wrongly highlighted;
• e.g., (shorts, on, man), (man, has, hand).
• Redundancy:
• As the edge dependencies within the subgraph are
ignored, edges might have no unique information;
• e.g., (man, on, ocean) vs (man, ridding, waves).
Wang et al. Reinforced Causal Explainer for Graph Neural Networks. TPAMI’2022

Post-hoc Explainability
Our Work 1: Causal Screening
• Key idea
• Screening à a sequential decision process
• Start from an empty set as the explanatory subgraph
• Incrementally add one edge to the subgraph
• One edge at each step.
• Causal Attribution à do-intervention
• Treatment: feed one edge with the previously selected edges into the model
• Control: feed the previous selected edges into the model
• Difference: the causal effect caused by the edge
Wang et al. Reinforced Causal Explainer for Graph Neural Networks. TPAMI’2022

Post-hoc Explainability
Our Work 1: Reinforced Causal Screening
Our Goal
• Learning to Perform Causal Screening:
• Train a reinforcement learning agent, which learns to do intervene & explain individual predictions
• Action à Do intervention
• Reward à Causal Effect of Action
Wang et al. Reinforced Causal Explainer for Graph Neural Networks. TPAMI’2022

Post-hoc Explainability
Our Work 1: Evaluation of Explanations
Wang et al. Reinforced Causal Explainer for Graph Neural Networks. TPAMI’2022

Post-hoc Explainability
Our Work 2: Out-of-distribution (OOD) Issue
A Causal Look At Out-of-Distribution
• Feature Removal:
• Given a subgraph of interest 𝑮𝐬
• Remove the complement 𝑮#
𝒔
• Quantify the mutual information between the subgraph & the
prediction on the full graph;
• One unobserved variable O is the confounder of 𝑮𝐬
& 𝒀
• Open backdoor path 𝑮𝐬
← 𝑫 → 𝒀
• Introduce spurious correlations
Wu et al. Deconfounding to Explanation Evaluation in Graph Neural Networks.

Post-hoc Explainability
Our Work 2: Front-door Adjustment
Deconfounding
• Feature Removal & In-filling:
• Given a subgraph of interest
• Remove the complement à make the subgraph off the data manifold
• Imagine possible complements instead à make the “subgraph + complements” on the data manifold
• Quantify the mutual information between the “in-filled surrogate” & the target prediction;
Wu et al. Deconfounding to Explanation Evaluation in Graph Neural Networks.

Post-hoc Explainability
Our Work 2: Deconfounded Subgraph Evaluator (DSE)
Wu et al. Deconfounding to Explanation Evaluation in Graph Neural Networks.
• Post-hoc explanations may not be faithful to the original GNNs
• Post-hoc explanations often do not make sense, or do not provide enough details to
understand what a black-box model is doing
• “Stop explaining black box machine learning models for high stakes decisions and use
interpretable models instead” —— [Rudin, Nature Machine Intelligence’19]

Why Explainability?
Type II: Intrinsic Interpretability
GNN Model 𝒇 Output Prediction "
𝒚
Input Graph 𝑮
Core of GNNs
Intrinsic Interpretability
Prediction "
𝒚 with Rationale Subgraph 𝑮𝒔
Input Graph 𝑮
Incorporate a rationalization module into the model design to make the predictions transparent
• Intrinsically reason about causes and effect observable within a model

Intrinsic Interpretability
Our Work 3: Rationale Discovery
Wu et al. Discovering Invariant Rationales For Graph Neural Networks. ICLR’2022
• Deep learning models like GNNs generally
• Fail to exhibit interpretability
• Fail to generalize out of distribution
Solution:
• Find causal feature 𝐶!

Intrinsic Interpretability
Our Work 3: Invariant Learning for Rationale Discovery
Wu et al. Discovering Invariant Rationales For Graph Neural Networks. ICLR’2022
In general, only the pairs of input 𝐺 and label 𝑌 are observed during training, while
neither causal feature 𝑪 nor shortcut feature 𝑺 is available.

Intrinsic Interpretability
Our Work 3: Invariant Learning for Rationale Discovery
Wu et al. Discovering Invariant Rationales For Graph Neural Networks. ICLR’2022

Intrinsic Interpretability
Our Work 3: Invariance Condition
Wu et al. Discovering Invariant Rationales For Graph Neural Networks. ICLR’2022

Intrinsic Interpretability
Our Work 3: Discovering Invariant Rationale (DIR)
Wu et al. Discovering Invariant Rationales For Graph Neural Networks. ICLR’2022

Intrinsic Interpretability
Our Work 3: Discovering Invariant Rationale (DIR)
Wu et al. Discovering Invariant Rationales For Graph Neural Networks. ICLR’2022

Intrinsic Interpretability
Our Work 3: Empirical Results of DIR
Wu et al. Discovering Invariant Rationales For Graph Neural Networks. ICLR’2022

Intrinsic Interpretability
Our Work 4: Graph Contrastive Learning
Li et al. Let Invariant Rationale Discovery inspire Graph Contrastive Learning. ICML 2022.
Graph Augmentation
Contrastive Learning
Invariance Look
If augmentations are
too aggressive?
Instance discrimination
may fail …

Intrinsic Interpretability
Our Work 4: Let IRD inspire Graph Pretraining
Li et al. Let Invariant Rationale Discovery inspire Graph Contrastive Learning. ICML 2022.
Graph Augmentation
Contrastive Learning
Invariance Look
If augmentations are
too aggressive?
Instance discrimination
may fail …
Rationale Discovery
Invariance Look
Sufficiency & Independence Principles
Rationale captures
discriminative info.
𝒚 is the label of “instance
discrimination”

Intrinsic Interpretability
Our Work 4: Let IRD inspire Graph Pretraining
Li et al. Let Invariant Rationale Discovery inspire Graph Contrastive Learning. ICML 2022.

Intrinsic Interpretability
Our Work 2: Let IRD inspire Graph Pretraining
Li et al. Let Invariant Rationale Discovery inspire Graph Contrastive Learning. ICML 2022.

Summary
Explainability of GNNS
• Post-hoc explainability
• Using an additional explainer method to explain a black-box model post hoc
• Explanations could be unfaithful to the decision-making process of model
• Intrinsic Interpretability
• Incorporating a rationalization module into the model design, so as to transform a
black-box to a white-box.
• Causal theory is one promising solution!
• Interpretability & Generalization

Summary
Explainability of GNNS
Ø Towards Multi-grained Explainability for Graph Neural Networks (NeurIPS’2021)
• https://github.com/Wuyxin/ReFine
Ø Reinforced Causal Explainer for Graph Neural Networks (TPAMI’2022)
• https://github.com/xiangwang1223/reinforced_causal_explainer
Ø Discovering invariant rationales for graph neural networks (ICLR’2022)
• https://github.com/Wuyxin/DIR-GNN
Ø Let Invariant Rationale Discovery inspire Graph Contrastive Learning (ICML’2022)
• https://github.com/lsh0520/RGCL
Ø Causal Attention for Interpretable and Generalizable Graph Classiﬁcation (KDD’2022)
• https://github.com/yongduosui/CAL
Ø Invariant Grounding for Video Question Answering (CVPR’2022, oral & best paper final list)
• https://github.com/yl3800/IGV
Ø Equivariant and Invariant Grounding for Video Question Answering (MM’2022)
• https://github.com/yl3800/EIGV

Thanks!