ICML2019読み会: Hyperbolic Disk Embeddings for Directed Acyclic Graphs

ICML2019読み会: Hyperbolic Disk Embeddings for Directed Acyclic Graphs

ICLR/ICML2019 読み会 (https://connpass.com/event/138672/) での発表資料です。

2c4e06a13a944723032415c88e96cbe5?s=128

ぬぬき

July 21, 2019
Tweet

Transcript

  1. Hyperbolic Disk Embeddings
 for Directed Acyclic Graphs ICML2019 ಡΈձ Ryota

    Suzuki ©2019 LAPRAS Inc.
  2. 2 Researcher / Algorithm Engineer ɾNEC → LAPRAS (2018.10)
 ɾػցֶशɾ৴߸ॲཧɺ෺ཧ(෺ੑཧ࿦)

    ɾtwitter: nunuki_ ɾLAPRAS: https://lapras.com/public/1HETMLB ɾHolacracy roles: ɹɹAlgorithm Prototyping / ࿦จࣥච / ੒ՌמऔΓϚϯ / σʔλ෼ੳ / ɹɹWebApp ࣮૷ / ϓϩμΫτಛڐઓུ / ϓϩμΫτ๏ྩ९क / etc. Ryota Suzuki (ླ໦ ྄ଠ)
  3. 3 ※ Ҏ߱ɺ΄΅ӳޠͰ͢

  4. *OUSPEVDUJPO

  5. 5 Embedding Methods Vector representation of discrete entities • Natural

    languages, graphs, … • Data structure is encoded as geometrical properties • Used as first layer of neural networks [Mikolov, NIPS 2013]
  6. 6 Embedding Methods ʙ     Word2Vec
 [Mikolov,

    NIPS2013] Various Applications Poincaré Embedding
 [Nickel, NIPS 2017] Order Embedding
 [Vendrov, ICLR2016] Probabilistic
 Order Embedding
 [Lai, EACL2017] Hyperbolic
 Entailment Cones
 [Ganea, ICML2018] Box Embedding
 [Vilnis ACL 2018] Lorentz Embedding
 [Nickel, ICML 2018] Smoothing
 Box Embedding
 [Li, ICLR 2019] Hyperbolic
 Disk Embeddings
 [Suzuki, ICML 2019] &NCFEEJOH )ZQFSCPMJD 0SEFS 1SPCBCJMJTUJD
  7. 7 Embedding Directed Acyclic Graphs (DAGs) • Semantics, citation networks,

    genealogical networks • Asymmetric / transitive relation of nodes
 → Partially ordered set (poset) DAG Embedding • Embedding nodes into continuous poset
 so that transitive relation is preserved. DAG Embedding Models
  8. 8 Existing Methods Order Embeddings
 [Vendrov, ICLR 2016] Hyperbolic Entailment

    Cones
 [Ganea, ICML 2018]
  9. 9 2D Euclidean Disk Embedding = Eular Diagram • Disk

    Embedding: General framework for embedding DAGs Our Contributions • Novel Hyperbolic Disk Embedding • Experiments • Theorems: Existing methods are special cases of DE
 (+ extra restrictions)
  10. %JTL&NCFEEJOHT

  11. 11 Disk Embedding Disk Embedding: DAG relation → Inclusive relation

    of disks Disk = Center + Radius X y x dX (x, y) + rx − ry ≤ 0 Dx = (x, rx )
  12. 12 Embedding DAGs Ancestors = Upper cone Descendants = Lower

    cone Transitive relation of DAG induces “cones” in the embedding space.
  13. 13 Key Idea y x Relation of DAG nodes

  14. 13 Key Idea = Inclusion of lower cones y x

    Relation of DAG nodes
  15. 13 Key Idea = Inclusion of projected disks X =

    Inclusion of lower cones y x Relation of DAG nodes
  16. 13 Key Idea = Inclusion of projected disks X =

    Inclusion of lower cones y x Relation of DAG nodes X y x dX (x, y) + rx − ry ≤ 0 = Disk Embedding
  17. %JTL&NCFEEJOHBT(FOFSBM'SBNFXPSL

  18. 15 Disk Embedding as General Framework Disk Embedding is easily

    extensible to general metric spaces Further generalizations • Metric spaces → Quasi-metric spaces • (Closed) Disks → Formal disks Existing methods can be understood as special cases of DEs X y x
  19. 16 Quasi-metric Spaces ∥x∥W = 1 w1 w2 w3 ∥x∥W

    := max i {w⊤ i x} Proposition 2 ∥x∥1 = 1 ∥x∥∞ = 1 ∥x∥2 = 1 L1-distance Euclidean distance Uniform distance Symmetric quasi-metric (metric) Asymmetric quasi-metric Unit disk Polyhedral quasimetric
  20. 17 Formal Disks Abstraction of closed / open disks y

    x y x Closed disk Open disk Dx ⊆ Dy D∘ x ⊆ D∘ y (x, r) (x, rx ) ⊑ (y, ry ) d(x, y) + rx − ry ≤ 0 Formal Disk Radii can be negative → Reversibility of Disk Embeddings
  21. 18 Existing Methods Order Embeddings
 [Vendrov, ICLR 2016] Hyperbolic Entailment

    Cones
 [Ganea, ICML 2018] Lower Cones
  22. 19 Equivalence of models Order Embeddings [Vendrov 2016] (a) ϕ

    Euclidean Disk Embedding (w/ polyhedral quasimetric) xk ≥ yk for k = 1,⋯, n (x′, r) = ( Px, a − 1 n n ∑ k=1 xk) dW (x′, y′) + rx − ry ≤ 0
  23. 20 Equivalence of models (a) (b) ϕ Spherical Disk Embedding

    d (x′, y′) + rx − ry ≤ 0 Hyperbolic Entailment Cones
 [Ganea, ICML 2018] (x′, r) = x ∥x∥ , arcsin ( 1 + ∥x∥2 2∥x∥ sin θ0) ψ(x) ≥ Ξ(x, y)
  24. 21 Limitations of Existing Methods Hierarchy Non-Hierarchy Tree-like Complex DAG

    Root No root
  25. 22 Limitations of Existing Methods Order Embeddings
 [Vendrov, ICLR 2016]

    Hyperbolic Entailment Cones
 [Ganea, ICML 2018] “Origin” Existing methods implicitly assumes the existence of the origin
 ≒ Assuming rooted hierarchies
  26. 23 Advantage of Disk Embedding Extends existing methods • Negative

    radius: reversibility & translational symmetry • Avoid gradient vanishing on loss functions Applicable for various (quasi-)metric spaces. Positive sample Negative sample Case of Negative Radius Loss Functions Ours Existing
 methods
  27. )ZQFSCPMJD%JTL&NCFEEJOH

  28. 25 Base Space Vector Embedding Disk Embedding Euclidean space Word2Vec

    etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017) Hyperbolic
 Disk Embedding
  29. 26 Hyperbolic Disk Embedding 2 1 3 4 5 6

    2 1 3 4 5 6 7 More neighboring disks in hyperbolic space All disks are of the same radius Euclidean Hyperbolic
  30. &YQFSJNFOUT

  31. 28 Experiments Hierarchy Non-Hierarchy Tree-like Reversed Tree-like Complex DAG This

    Paper
  32. 29 Experimental Results Hierarchy Non-
 Hierarchy Our Disk Embedding models

    achieved significant improvement especially in non-hierarchical data
  33. $PODMVTJPO'VUVSF8PSLT

  34. 31 • Disk Embedding: General framework for embedding DAGs Conclusion

    • Novel Hyperbolic Disk Embedding • Experiments • Theorems: Existing methods are special cases of DE
 (+ extra restrictions)
  35. 32 Future Works: Experiments Hierarchy Non-Hierarchy Tree-like Reversed Tree-like Complex

    DAG Future Work
  36. 33 Future Works: Theories ʙ     Word2Vec


    [Mikolov, NIPS2013] Various Applications Poincaré Embedding
 [Nickel, NIPS 2017] Order Embedding
 [Vendrov, ICLR2016] Probabilistic
 Order Embedding
 [Lai, EACL2017] Hyperbolic
 Entailment Cones
 [Ganea, ICML2018] Box Embedding
 [Vilnis ACL 2018] Lorentz Embedding
 [Nickel, ICML 2018] Smoothing
 Box Embedding
 [Li, ICLR 2019] Hyperbolic
 Disk Embeddings
 [Suzuki, ICML 2019] &NCFEEJOH )ZQFSCPMJD 0SEFS 1SPCBCJMJTUJD .PSFHFOFSBMJ[BUJPOT
 GPSQSPCBCJMJTUJDNPEFMT
  37. 5IBOLZPVGPSMJTUFOJOH