ICML2019読み会: Hyperbolic Disk Embeddings for Directed Acyclic Graphs

Transcript

1. Hyperbolic Disk Embeddings
 for Directed Acyclic Graphs ICML2019 ಡΈձ Ryota

   Suzuki ©2019 LAPRAS Inc.

2. 2 Researcher / Algorithm Engineer ɾNEC → LAPRAS (2018.10)
 ɾػցֶशɾ৴߸ॲཧɺ෺ཧ(෺ੑཧ࿦)

   ɾtwitter: nunuki_ ɾLAPRAS: https://lapras.com/public/1HETMLB ɾHolacracy roles: ɹɹAlgorithm Prototyping / ࿦จࣥච / ੒ՌמऔΓϚϯ / σʔλ෼ੳ / ɹɹWebApp ࣮૷ / ϓϩμΫτಛڐઓུ / ϓϩμΫτ๏ྩ९क / etc. Ryota Suzuki (ླ໦ ྄ଠ)

3. 3 ※ Ҏ߱ɺ΄΅ӳޠͰ͢

4. *OUSPEVDUJPO

5. 5 Embedding Methods Vector representation of discrete entities • Natural

   languages, graphs, … • Data structure is encoded as geometrical properties • Used as first layer of neural networks [Mikolov, NIPS 2013]

6. 6 Embedding Methods ʙ     Word2Vec
 [Mikolov,

   NIPS2013] Various Applications Poincaré Embedding
 [Nickel, NIPS 2017] Order Embedding
 [Vendrov, ICLR2016] Probabilistic
 Order Embedding
 [Lai, EACL2017] Hyperbolic
 Entailment Cones
 [Ganea, ICML2018] Box Embedding
 [Vilnis ACL 2018] Lorentz Embedding
 [Nickel, ICML 2018] Smoothing
 Box Embedding
 [Li, ICLR 2019] Hyperbolic
 Disk Embeddings
 [Suzuki, ICML 2019] &NCFEEJOH )ZQFSCPMJD 0SEFS 1SPCBCJMJTUJD

7. 7 Embedding Directed Acyclic Graphs (DAGs) • Semantics, citation networks,

   genealogical networks • Asymmetric / transitive relation of nodes
 → Partially ordered set (poset) DAG Embedding • Embedding nodes into continuous poset
 so that transitive relation is preserved. DAG Embedding Models

8. 8 Existing Methods Order Embeddings
 [Vendrov, ICLR 2016] Hyperbolic Entailment

   Cones
 [Ganea, ICML 2018]

9. 9 2D Euclidean Disk Embedding = Eular Diagram • Disk

   Embedding: General framework for embedding DAGs Our Contributions • Novel Hyperbolic Disk Embedding • Experiments • Theorems: Existing methods are special cases of DE
 (+ extra restrictions)

10. %JTL
    &NCFEEJOHT

11. 11 Disk Embedding Disk Embedding: DAG relation → Inclusive relation

    of disks Disk = Center + Radius X y x dX (x, y) + rx − ry ≤ 0 Dx = (x, rx )

12. 12 Embedding DAGs Ancestors = Upper cone Descendants = Lower

    cone Transitive relation of DAG induces “cones” in the embedding space.

13. 13 Key Idea y x Relation of DAG nodes

14. 13 Key Idea = Inclusion of lower cones y x

    Relation of DAG nodes

15. 13 Key Idea = Inclusion of projected disks X =

    Inclusion of lower cones y x Relation of DAG nodes

16. 13 Key Idea = Inclusion of projected disks X =

    Inclusion of lower cones y x Relation of DAG nodes X y x dX (x, y) + rx − ry ≤ 0 = Disk Embedding

17. %JTL
    &NCFEEJOH
    BT
    (FOFSBM
    'SBNFXPSL

18. 15 Disk Embedding as General Framework Disk Embedding is easily

    extensible to general metric spaces Further generalizations • Metric spaces → Quasi-metric spaces • (Closed) Disks → Formal disks Existing methods can be understood as special cases of DEs X y x

19. 16 Quasi-metric Spaces ∥x∥W = 1 w1 w2 w3 ∥x∥W

    := max i {w⊤ i x} Proposition 2 ∥x∥1 = 1 ∥x∥∞ = 1 ∥x∥2 = 1 L1-distance Euclidean distance Uniform distance Symmetric quasi-metric (metric) Asymmetric quasi-metric Unit disk Polyhedral quasimetric

20. 17 Formal Disks Abstraction of closed / open disks y

    x y x Closed disk Open disk Dx ⊆ Dy D∘ x ⊆ D∘ y (x, r) (x, rx ) ⊑ (y, ry ) d(x, y) + rx − ry ≤ 0 Formal Disk Radii can be negative → Reversibility of Disk Embeddings

21. 18 Existing Methods Order Embeddings
 [Vendrov, ICLR 2016] Hyperbolic Entailment

    Cones
 [Ganea, ICML 2018] Lower Cones

22. 19 Equivalence of models Order Embeddings [Vendrov 2016] (a) ϕ

    Euclidean Disk Embedding (w/ polyhedral quasimetric) xk ≥ yk for k = 1,⋯, n (x′, r) = ( Px, a − 1 n n ∑ k=1 xk) dW (x′, y′) + rx − ry ≤ 0

23. 20 Equivalence of models (a) (b) ϕ Spherical Disk Embedding

    d (x′, y′) + rx − ry ≤ 0 Hyperbolic Entailment Cones
 [Ganea, ICML 2018] (x′, r) = x ∥x∥ , arcsin ( 1 + ∥x∥2 2∥x∥ sin θ0) ψ(x) ≥ Ξ(x, y)

24. 21 Limitations of Existing Methods Hierarchy Non-Hierarchy Tree-like Complex DAG

    Root No root

25. 22 Limitations of Existing Methods Order Embeddings
 [Vendrov, ICLR 2016]

    Hyperbolic Entailment Cones
 [Ganea, ICML 2018] “Origin” Existing methods implicitly assumes the existence of the origin
 ≒ Assuming rooted hierarchies

26. 23 Advantage of Disk Embedding Extends existing methods • Negative

    radius: reversibility & translational symmetry • Avoid gradient vanishing on loss functions Applicable for various (quasi-)metric spaces. Positive sample Negative sample Case of Negative Radius Loss Functions Ours Existing
 methods

27. )ZQFSCPMJD
    %JTL
    &NCFEEJOH

28. 25 Base Space Vector Embedding Disk Embedding Euclidean space Word2Vec

    etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017) Hyperbolic
 Disk Embedding

29. 26 Hyperbolic Disk Embedding 2 1 3 4 5 6

    2 1 3 4 5 6 7 More neighboring disks in hyperbolic space All disks are of the same radius Euclidean Hyperbolic

30. &YQFSJNFOUT

31. 28 Experiments Hierarchy Non-Hierarchy Tree-like Reversed Tree-like Complex DAG This

    Paper

32. 29 Experimental Results Hierarchy Non-
 Hierarchy Our Disk Embedding models

    achieved significant improvement especially in non-hierarchical data

33. $PODMVTJPO
    
    'VUVSF
    8PSLT

34. 31 • Disk Embedding: General framework for embedding DAGs Conclusion

    • Novel Hyperbolic Disk Embedding • Experiments • Theorems: Existing methods are special cases of DE
 (+ extra restrictions)

35. 32 Future Works: Experiments Hierarchy Non-Hierarchy Tree-like Reversed Tree-like Complex

    DAG Future Work

36. 33 Future Works: Theories ʙ     Word2Vec


    [Mikolov, NIPS2013] Various Applications Poincaré Embedding
 [Nickel, NIPS 2017] Order Embedding
 [Vendrov, ICLR2016] Probabilistic
 Order Embedding
 [Lai, EACL2017] Hyperbolic
 Entailment Cones
 [Ganea, ICML2018] Box Embedding
 [Vilnis ACL 2018] Lorentz Embedding
 [Nickel, ICML 2018] Smoothing
 Box Embedding
 [Li, ICLR 2019] Hyperbolic
 Disk Embeddings
 [Suzuki, ICML 2019] &NCFEEJOH )ZQFSCPMJD 0SEFS 1SPCBCJMJTUJD .PSF
    HFOFSBMJ[BUJPOT
 GPS
    QSPCBCJMJTUJD
    NPEFMT

37. 5IBOL
    ZPV
    GPS
    MJTUFOJOH