Approximation Ratios of Graph Neural Networks for Combinatorial Problems [Sato+ NeurIPS 2019] 2019/10/23 Kenshin Abe 1 

TL;DR ✴ What’s this paper? ‣ Establish stronger GNN model with knowledge from distributed local algorithm ‣ Analyze GNN’s approximation ratios for combinatorial problems ‣ Prove preprocessing strengthens representation power ✴ Why this paper? ‣ GNN intuitively corresponds with distributed local algorithm ‣ Theoretically interesting result on preprocessing 2 

Related Work Few studies on theoretical analysis of GNN ✴ [Xu+ ICLR 2019] ‣ Representation power in terms of graph isomorphism ‣ Compare with WL isomorphism test (heuristic algorithm) ✴ This paper ‣ Representation power to solve other combinatorial problems ‣ Compare with distributed local algorithm 3 

Distributed Local Algorithm 4 

Distributed Local Algorithm ✴ Graph’s each node has a processor with infinite computational resources ✴ In each step, each processor synchronously 1. Send messages to neighbors 2. Receive messages from neighbors 3. Update its state ✴ Decides the output within a constant number of steps ✴ Usually assume graph degree is bounded by A Δ 5 

Developed models ✴ Deterministic algorithm for identical processors is known to be weak ✴ Many developed models ‣ Port numbering, unique identifiers, randomness, etc. 6 

Port Numbering ✴ Number each port of a node as ‣ : node sending message to port of node ‣ : port of node to port of node ✴ Consistent port numbering: ‣ Computed in linear time v 1,2,⋯, deg(v) ptail (v, i) i v pn (v, i) ptail (v, i) i v ∀(v, i) . p(p(v, i)) = (v, i) 7 Inconsistent port nubmering Consistent port nubmering https://arxiv.org/pdf/1205.2051.pdf 

Models Classes [Hella+ 2015] ✴ Set: ✴ Multiset: ✴ Broadcast: send same messages to all neighbors set( ⃗ a ) = set( ⃗ b ) ⇒ f( ⃗ a ) = f( ⃗ b ) multiset( ⃗ a ) = multiset( ⃗ b ) ⇒ f( ⃗ a ) = f( ⃗ b ) 8 Incoming Messages Outgoing Messages SB(1) Set Broadcast MB(1) Multiset Broadcast VVc(1) Vector Vector 

Correspondence between Distributed Local Algorithm and GNN 9 

SB-GNN ✴ ✴ Multiple values don’t count ✴ GraphSAGE-pool ‣ zl+1 v = fl(zl v , SET(zl u |u ∈ N(v))) zl+1 v = max({σ(Wlzl u + bl)|u ∈ N(v)}) 10 

MB-GNN ✴ ✴ Permutation invariant ✴ GraphSAGE-mean ‣ ✴ GAT, GIN, etc. zl+1 v = fl(zl v , MULTISET(zl u |u ∈ N(v))) zl+1 v = CONCAT(zl v , 1 |N(v)| ∑ u∈N(v) Wlzl u ) 11 

Proposed: VVc-GNN ✴ ✴ Handle messages from different nodes differently ✴ CPNGNN (Proposed!!) ‣ ‣ Most powerful among VVc-GNNs (if node feature is finite) zl+1 v = fl(zl v , zl ptail (v,1) , pn (v,1), zl ptail (v,2) , pn (v,2), ⋯, zl ptail (v,Δ) , pn (v, Δ)) zl+1 v = σ(WlCONCAT(zl v , zl ptail (v,1) , pn (v,1), zl ptail (v,2) , pn (v,2), ⋯, zl ptail (v,Δ) , pn (v, Δ))) 12 

Hierarchy of GNNs ✴ Representation power correspondence ‣ SB-GNN = SB(1) ‣ MB-GNN = MB(1) ‣ VVc-GNN = VVc(1) ✴ SB-GNN MB-GNN VVc-GNN (Natural by definition) ‣ Strict order? ≤ ≤ 13 

VVc-GNN strictly stronger than MB-GNN? ✴ Finding single leaf problem ‣ Select only a single leaf from a star graph ‣ MB-GNN cannot solve • Leaf embeddings are always same ‣ VVc-GNN can solve ✴ SB-GNN MB-GNN VVc-GNN < < 14 https://en.wikipedia.org/wiki/Star_(graph_theory) 

Approximation ratio of VVc-GNN on Combinatorial Problems 15 

Approximation ratio of VVc-GNN Give degree information as node input feature ✴ Minimum Dominating Set ‣ ✴ Minimum Vertex Cover ‣ 2 ✴ Maximum Matching ‣ No approximation for any Δ + 1 α α > 0 16 

Give more information as node feature ✴ Only degree information might be too poor ✴ Weak 2-coloring ‣ Any graph has a weak 2-coloring ‣ Calculated in linear time by breadth-first search 17 

Approximation ratio of VVc-GNN Give degree and weak 2-coloring information as node input feature ✴ Minimum Dominating Set ‣ → ✴ Minimum Vertex Cover ‣ 2 ✴ Maximum Matching ‣ No approximation for any → Δ + 1 Δ + 1 2 α α > 0 Δ + 1 2 18 

Summary 19 

Summary ✴ Establish stronger GNN model with knowledge from distributed local algorithm ✴ Analyze GNN’s approximation ratios for combinatorial problems ✴ Prove preprocessing strengthens representation power 20 

My Thoughts ✴ Impressed with how well distributed local algorithm represents GNN ✓Number of layer is constant - Degree upper bound assumption ✴ Proposed network seems too artificial (concatenating integer to feature vectors) - Does it actually learn well? 21 

References 1. R. Sato, M. Yamada, and H. Kashima. Approximation Ratios of Graph Neural Networks for Combinatorial Problems. In NeurIPS 2019. 2. K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How Powerful are Graph Neural Networks? In ICLR 2019. 3. Hella et al. Weak Models of Distributed Computing, with Connections to Modal Logic. 22