TuongHH GALE

Transcript

1. ICML 2025 Ho Huu Tuong Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: tuonghh@catholic.ac.kr
2. 1 Contents ➢ Introduction ➢ Proposed Approach ➢ Experiment results ➢ Conclusion
3. 1. Introduction Why node equivalence matters Graphs contain nodes that play identical structural roles or have identical/similar attributes. • Automorphic equivalence: nodes can be exchanged through a graph automorphism while preserving all edges. • Attribute equivalence: nodes have identical feature vectors. • Desired representation property: • equivalent nodes → nearby embeddings; • non-equivalent nodes → separated embeddings. For example, permuting nodes 1,2,3 or swapping 6,7 preserves the graph, hence those nodes are structurally equivalent; swapping nodes 4 and 5 does not. 2
4. 1. Introduction Graph Automorphism 3
5. 4 1. Introduction Graph Automorphism Network Datasets Graph-Level Datasets
6. 5 1. Introduction The gap in graph contrastive learning Standard graph contrastive learning creates two augmented graph views, 𝐺1 and 𝐺2 , then: • aligns node 𝑖 with itself across views; • treats every other node as a negative; • therefore, implicitly assumes every node forms a singleton equivalence class. This can incorrectly repel two different, but structurally or semantically equivalent, nodes. 𝑛 exp sim 𝐳𝑖1 , 𝐳𝑖2 /𝜏 1 ℒcon = − ෍ log 𝑛 𝑛 σ𝑗=1 exp sim 𝐳𝑖1 , 𝐳𝑗2 /𝜏 𝑖=1 Where: • Positive: 𝐳𝑖1 , 𝐳𝑖2 • Negatives: 𝐳𝑖1 , 𝐳𝑗2 , 𝑗 ≠ 𝑖 • 𝜏: temperature parameter.
7. 6 1. Introduction Problem formulation Given a graph 𝐺 = 𝑉, 𝐸 , 𝑉 = 𝑛, 𝐸 = 𝑚, and optional node-feature matrix 𝐗 ∈ ℝ𝑛×𝑑 , learn an unsupervised embedding for every node: 𝐳𝑣 ∈ ℝ𝑞 , 𝑣 ∈ 𝑉. The proposed framework, GALE, constructs equivalence classes from graph structure and node attributes, then regularizes a graph encoder so embeddings respect those classes.
8. 7 1. Introduction Equivalence relations and partitions A relation ∼ over a set 𝑆 is an equivalence relation if it satisfies: • Reflexivity: 𝑥 ∼ 𝑥 • Symmetry: 𝑥 ∼ 𝑦 ⇒ 𝑦 ∼ 𝑥 • Transitivity: 𝑥 ∼ 𝑦 ∧ 𝑦 ∼ 𝑧 ⇒ 𝑥 ∼ 𝑧 The equivalence class of 𝑥 is 𝑥 = 𝑦 ∈ 𝑆: 𝑥 ∼ 𝑦 . Equivalence classes form a partition: they are non-empty, mutually disjoint, and their union covers all nodes.
9. 8 Contents ➢ Introduction ➢ Proposed Approach ➢ Experiment results ➢ Conclusion
10. 9 2. Methodology Overview of GALE GALE has two branches: 1. Graph encoder: a GNN produces node embeddings 𝐳𝑣 . 2. Equivalence constructor: derives node partitions from: • graph topology (Group I), • node attributes (Group II). The two partitions are fused, then used by an (GrAph self-supervised Learning framework with Equivalence) equivalence loss to regularize the encoder.
11. 10 2. Methodology Structural equivalence: graph automorphisms A graph automorphism is a permutation 𝜋 of nodes that preserves edges: 𝜋 ∈ Aut 𝐺 ⟺ 𝐸 = 𝐸𝜋. Two nodes 𝑢, 𝑣 are automorphically equivalent if at least one automorphism maps 𝑢 to 𝑣: 𝑢 ∼auto 𝑣 ⟺ ∃𝜋 ∈ Aut 𝐺 , 𝑢𝜋 = 𝑣. The resulting equivalence classes are graph-automorphism orbits.
12. 11 2. Methodology Attribute equivalence For nodes 𝑢, 𝑣 ∈ 𝑉 with attribute vectors 𝐱 𝑢 , 𝐱 𝑣 : 𝑢 ∼attr 𝑣 ⟺ 𝐱 𝑢 = 𝐱 𝑣 . This creates an attribute partition 𝑃attr = 𝐷𝑖 , where every 𝐷𝑖 contains all nodes with the same features. Interpretation: exact feature equality provides semantic equivalence, complementing structural role equivalence.
13. 12 2. Methodology Fusion: require both structure and attributes GALE fuses structural and attribute partitions by intersection: 𝑃fuse = 𝐶𝑖 ∩ 𝐷𝑗 𝐶𝑖 ∈ 𝑃auto , 𝐷𝑗 ∈ 𝑃attr , 𝐶𝑖 ∩ 𝐷𝑗 ≠ ∅ . A fused group contains nodes that are equivalent under both relations. 𝑢 ∼fuse 𝑣 ⟺ 𝑢 ∼auto 𝑣 ∧ 𝑢 ∼attr 𝑣 . The paper proves that this fused relation is reflexive, symmetric, and transitive. Why intersection rather than union? Intersection makes the positive set stricter: a pair must agree structurally and semantically.
14. 13 2. Methodology Equivalence loss Let 𝑃fuse = 𝐹𝑖 be the fused partition and 𝐳𝑢 node 𝑢’s embedding. Intra-class attraction For every pair in the same fused class: ℒintra = ෍ ෍ − 𝐷 𝐳𝑢 , 𝐳𝑣 . 𝑖 𝑢,𝑣∈𝐹𝑖 Inter-class repulsion For nodes from different fused classes: ℒinter = ෍ ෍ 𝐷 𝐳𝑠 , 𝐳𝑡 . 𝑖≠𝑗 𝑠∈𝐹𝑖 𝑡∈𝐹𝑗 Total equivalence loss ℒequiv = ℒintra + ℒinter . Inner-product similarity: 𝐷 𝐳𝑖 , 𝐳𝑗 = 𝐳𝑖⊤ 𝐳𝑗 .
15. 14 2. Methodology Approximate automorphic equivalence Exact graph automorphism detection is described as computationally intractable. GALE therefore approximates automorphic equivalence with PageRank. PageRank is the stationary distribution: 𝐫 = 𝛼𝐏 ⊤ 𝐫 + 1 − 𝛼 𝐯, where: • 𝐏 is the transition matrix; • 𝛼 ∈ 0,1 is the teleportation parameter; • 𝐯 is usually uniform. The key implication used by GALE: 𝑢 ∼auto 𝑣 ⟹ 𝑟𝑢 = 𝑟𝑣 . Thus, nodes with similar PageRank scores are grouped as approximate structural equivalents.
16. 15 2. Methodology Approximate attribute equivalence Exact attribute equality can split practically similar nodes into separate groups. GALE relaxes it with an 𝜖-ball condition: 𝑢 ∼𝜖attr 𝑣 ⟺ 𝐱 𝑢 − 𝐱 𝑣 2 ≤ 𝜖. • 𝜖 = 0: exact feature equivalence; • 𝜖 > 0: feature-similarity groups. Note that: for 𝜖 > 0, this relation is not necessarily a strict equivalence relation because transitivity may fail.
17. 16 2. Methodology Computational complexity Component Complexity PageRank structural approximation 𝑂 𝑚 Attribute grouping 𝑂 𝑛2 Partition fusion 𝑂 𝑛 in practice Equivalence-loss calculation 𝑂 𝑛2 worst case Overall 𝑂 𝑛2 + 𝑚 Practical implication: PageRank makes structural equivalence tractable, but all-pairs attribute grouping and inter-class loss remain quadratic bottlenecks.
18. 2. Methodology 17 Relationship to GNNs and graph transformers Message-passing neural networks MPNN neighborhood aggregation naturally maps automorphically equivalent nodes to similar embeddings. But increasing depth can make all nodes similar, over-smoothing, without protecting inter-class separation. Graph transformers Graph transformers use positional encodings plus global attention. They do not inherently preserve graph automorphisms; their treatment of equivalent nodes depends on whether the positional encoding respects those symmetries.
19. 18 Contents ➢ Introduction ➢ Proposed Approach ➢ Experiment results ➢ Conclusion
20. 3. Experiments and Results 19 Experimental setup Node classification • 8 datasets: Cora, CiteSeer, PubMed, Wiki-CS, Amazon-Computers, Amazon-Photo, Coauthor-CS, Coauthor-Physics. • Train/validation/test: 10%/10%/80%. • Mean test accuracy over 50 runs. • Linear classifier trained on learned embeddings. Graph classification • 8 TU datasets: NCI1, PROTEINS, DD, MUTAG, COLLAB, RDT-B, RDT-M5K, IMDB-B. • Linear SVM evaluation. • Five runs of 10-fold cross-validation.
21. 3. Experiments and Results Graph classification • GALE leads 7/8 datasets; largest gains: PROTEINS +5.15%, IMDB-B +4.06%. • GALE-APR nearly matches GALE, validating PageRank approximation. 20
22. 3. Experiments and Results Node classification • GALE/GALE-APR lead all 8 datasets among unsupervised methods. • Largest gain: PubMed 85.06%, +3.10% over prior best. 21
23. 3. Experiments and Results Equivalence ablation • Removing either structural or feature equivalence hurts every dataset. • Feature equivalence is particularly important on CiteSeer: −9.86% without it. 22
24. 3. Experiments and Results Fusion ablation • Intersection fusion beats union on all datasets. • Requiring both structural and feature agreement yields cleaner equivalence groups. 23
25. 24 Contents ➢ Introduction ➢ Proposed Approach ➢ Experiment results ➢ Conclusion
26. 4. Conclusions 25 1. GALE reframes self-supervised graph representation learning as learning under equivalence constraints. 2. It fuses automorphic and attribute equivalence using partition intersection. 3. It attracts equivalent-node embeddings and separates different equivalence classes. 4. PageRank provides an efficient approximation to exact automorphic equivalence. 5. The method outperforms the reported self-supervised baselines across many node- and graphclassification benchmarks.
27. 4. Conclusions Limitations 1. Static graphs only. • In dynamic graphs, changing topology or attributes can change equivalence classes, requiring incremental partition updates. 2. Homogeneous graphs only. • Heterogeneous graphs with multiple node and edge types require type-aware automorphism and equivalence definitions. Future work ➢ Dynamic GALE: maintain equivalence partitions as nodes, edges, and features evolve. ➢ Heterogeneous GALE: define equivalence under node/edge types and relation-specific symmetries. ➢ Scalable grouping: replace 𝑂 𝑛2 attribute and loss computations with sampled, sparse, or approximate objectives. ➢ Transformer integration: build positional encodings explicitly invariant to automorphisms. ➢ Beyond classification: test on link prediction, graph generation, anomaly detection, molecular modeling, and role discovery. 26