ICML2019読み会: Hyperbolic Disk Embeddings for Directed Acyclic Graphs

ICLR/ICML2019 読み会 (https://connpass.com/event/138672/) での発表資料です。

Transcript

1. Hyperbolic Disk Embeddings
 for Directed Acyclic Graphs ICML2019 ಡΈձ Ryota Suzuki ©2019 LAPRAS Inc.
2. 2 Researcher / Algorithm Engineer ɾNEC → LAPRAS (2018.10)
 ɾػցֶशɾ৴߸ॲཧɺ෺ཧ(෺ੑཧ࿦) ɾtwitter: nunuki_ ɾLAPRAS: https://lapras.com/public/1HETMLB ɾHolacracy roles: ɹɹAlgorithm Prototyping / ࿦จࣥච / ੒ՌמऔΓϚϯ / σʔλ෼ੳ / ɹɹWebApp ࣮૷ / ϓϩμΫτಛڐઓུ / ϓϩμΫτ๏ྩ९क / etc. Ryota Suzuki (ླ໦ ྄ଠ)
3. 3 ※ Ҏ߱ɺ΄΅ӳޠͰ͢
4. *OUSPEVDUJPO
5. 5 Embedding Methods Vector representation of discrete entities • Natural languages, graphs, … • Data structure is encoded as geometrical properties • Used as first layer of neural networks [Mikolov, NIPS 2013]
6. 6 Embedding Methods ʙ     Word2Vec
 [Mikolov, NIPS2013] Various Applications Poincaré Embedding
 [Nickel, NIPS 2017] Order Embedding
 [Vendrov, ICLR2016] Probabilistic
 Order Embedding
 [Lai, EACL2017] Hyperbolic
 Entailment Cones
 [Ganea, ICML2018] Box Embedding
 [Vilnis ACL 2018] Lorentz Embedding
 [Nickel, ICML 2018] Smoothing
 Box Embedding
 [Li, ICLR 2019] Hyperbolic
 Disk Embeddings
 [Suzuki, ICML 2019] &NCFEEJOH )ZQFSCPMJD 0SEFS 1SPCBCJMJTUJD
7. 7 Embedding Directed Acyclic Graphs (DAGs) • Semantics, citation networks, genealogical networks • Asymmetric / transitive relation of nodes
 → Partially ordered set (poset) DAG Embedding • Embedding nodes into continuous poset
 so that transitive relation is preserved. DAG Embedding Models
8. 8 Existing Methods Order Embeddings
 [Vendrov, ICLR 2016] Hyperbolic Entailment Cones
 [Ganea, ICML 2018]
9. 9 2D Euclidean Disk Embedding = Eular Diagram • Disk Embedding: General framework for embedding DAGs Our Contributions • Novel Hyperbolic Disk Embedding • Experiments • Theorems: Existing methods are special cases of DE
 (+ extra restrictions)
10. %JTL
    &NCFEEJOHT
11. 11 Disk Embedding Disk Embedding: DAG relation → Inclusive relation of disks Disk = Center + Radius X y x dX (x, y) + rx − ry ≤ 0 Dx = (x, rx )
12. 12 Embedding DAGs Ancestors = Upper cone Descendants = Lower cone Transitive relation of DAG induces “cones” in the embedding space.
13. 13 Key Idea y x Relation of DAG nodes
14. 13 Key Idea = Inclusion of lower cones y x Relation of DAG nodes
15. 13 Key Idea = Inclusion of projected disks X = Inclusion of lower cones y x Relation of DAG nodes
16. 13 Key Idea = Inclusion of projected disks X = Inclusion of lower cones y x Relation of DAG nodes X y x dX (x, y) + rx − ry ≤ 0 = Disk Embedding
17. %JTL
    &NCFEEJOH
    BT
    (FOFSBM
    'SBNFXPSL
18. 15 Disk Embedding as General Framework Disk Embedding is easily extensible to general metric spaces Further generalizations • Metric spaces → Quasi-metric spaces • (Closed) Disks → Formal disks Existing methods can be understood as special cases of DEs X y x
19. 16 Quasi-metric Spaces ∥x∥W = 1 w1 w2 w3 ∥x∥W := max i {w⊤ i x} Proposition 2 ∥x∥1 = 1 ∥x∥∞ = 1 ∥x∥2 = 1 L1-distance Euclidean distance Uniform distance Symmetric quasi-metric (metric) Asymmetric quasi-metric Unit disk Polyhedral quasimetric
20. 17 Formal Disks Abstraction of closed / open disks y x y x Closed disk Open disk Dx ⊆ Dy D∘ x ⊆ D∘ y (x, r) (x, rx ) ⊑ (y, ry ) d(x, y) + rx − ry ≤ 0 Formal Disk Radii can be negative → Reversibility of Disk Embeddings
21. 18 Existing Methods Order Embeddings
 [Vendrov, ICLR 2016] Hyperbolic Entailment Cones
 [Ganea, ICML 2018] Lower Cones
22. 19 Equivalence of models Order Embeddings [Vendrov 2016] (a) ϕ Euclidean Disk Embedding (w/ polyhedral quasimetric) xk ≥ yk for k = 1,⋯, n (x′, r) = ( Px, a − 1 n n ∑ k=1 xk) dW (x′, y′) + rx − ry ≤ 0
23. 20 Equivalence of models (a) (b) ϕ Spherical Disk Embedding d (x′, y′) + rx − ry ≤ 0 Hyperbolic Entailment Cones
 [Ganea, ICML 2018] (x′, r) = x ∥x∥ , arcsin ( 1 + ∥x∥2 2∥x∥ sin θ0) ψ(x) ≥ Ξ(x, y)
24. 21 Limitations of Existing Methods Hierarchy Non-Hierarchy Tree-like Complex DAG Root No root
25. 22 Limitations of Existing Methods Order Embeddings
 [Vendrov, ICLR 2016] Hyperbolic Entailment Cones
 [Ganea, ICML 2018] “Origin” Existing methods implicitly assumes the existence of the origin
 ≒ Assuming rooted hierarchies
26. 23 Advantage of Disk Embedding Extends existing methods • Negative radius: reversibility & translational symmetry • Avoid gradient vanishing on loss functions Applicable for various (quasi-)metric spaces. Positive sample Negative sample Case of Negative Radius Loss Functions Ours Existing
 methods
27. )ZQFSCPMJD
    %JTL
    &NCFEEJOH
28. 25 Base Space Vector Embedding Disk Embedding Euclidean space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017) Hyperbolic
 Disk Embedding
29. 26 Hyperbolic Disk Embedding 2 1 3 4 5 6 2 1 3 4 5 6 7 More neighboring disks in hyperbolic space All disks are of the same radius Euclidean Hyperbolic
30. &YQFSJNFOUT
31. 28 Experiments Hierarchy Non-Hierarchy Tree-like Reversed Tree-like Complex DAG This Paper
32. 29 Experimental Results Hierarchy Non-
 Hierarchy Our Disk Embedding models achieved significant improvement especially in non-hierarchical data
33. $PODMVTJPO
    
    'VUVSF
    8PSLT
34. 31 • Disk Embedding: General framework for embedding DAGs Conclusion • Novel Hyperbolic Disk Embedding • Experiments • Theorems: Existing methods are special cases of DE
 (+ extra restrictions)
35. 32 Future Works: Experiments Hierarchy Non-Hierarchy Tree-like Reversed Tree-like Complex DAG Future Work
36. 33 Future Works: Theories ʙ     Word2Vec
 [Mikolov, NIPS2013] Various Applications Poincaré Embedding
 [Nickel, NIPS 2017] Order Embedding
 [Vendrov, ICLR2016] Probabilistic
 Order Embedding
 [Lai, EACL2017] Hyperbolic
 Entailment Cones
 [Ganea, ICML2018] Box Embedding
 [Vilnis ACL 2018] Lorentz Embedding
 [Nickel, ICML 2018] Smoothing
 Box Embedding
 [Li, ICLR 2019] Hyperbolic
 Disk Embeddings
 [Suzuki, ICML 2019] &NCFEEJOH )ZQFSCPMJD 0SEFS 1SPCBCJMJTUJD .PSF
    HFOFSBMJ[BUJPOT
 GPS
    QSPCBCJMJTUJD
    NPEFMT
37. 5IBOL
    ZPV
    GPS
    MJTUFOJOH