Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ICML2019読み会: Hyperbolic Disk Embeddings for Dir...

ICML2019読み会: Hyperbolic Disk Embeddings for Directed Acyclic Graphs

ICLR/ICML2019 読み会 (https://connpass.com/event/138672/) での発表資料です。

Ryota Suzuki / ぬぬき

July 21, 2019
Tweet

More Decks by Ryota Suzuki / ぬぬき

Other Decks in Technology

Transcript

  1. 2 Researcher / Algorithm Engineer ɾNEC → LAPRAS (2018.10)
 ɾػցֶशɾ৴߸ॲཧɺ෺ཧ(෺ੑཧ࿦)

    ɾtwitter: nunuki_ ɾLAPRAS: https://lapras.com/public/1HETMLB ɾHolacracy roles: ɹɹAlgorithm Prototyping / ࿦จࣥච / ੒ՌמऔΓϚϯ / σʔλ෼ੳ / ɹɹWebApp ࣮૷ / ϓϩμΫτಛڐઓུ / ϓϩμΫτ๏ྩ९क / etc. Ryota Suzuki (ླ໦ ྄ଠ)
  2. 5 Embedding Methods Vector representation of discrete entities • Natural

    languages, graphs, … • Data structure is encoded as geometrical properties • Used as first layer of neural networks [Mikolov, NIPS 2013]
  3. 6 Embedding Methods ʙ     Word2Vec
 [Mikolov,

    NIPS2013] Various Applications Poincaré Embedding
 [Nickel, NIPS 2017] Order Embedding
 [Vendrov, ICLR2016] Probabilistic
 Order Embedding
 [Lai, EACL2017] Hyperbolic
 Entailment Cones
 [Ganea, ICML2018] Box Embedding
 [Vilnis ACL 2018] Lorentz Embedding
 [Nickel, ICML 2018] Smoothing
 Box Embedding
 [Li, ICLR 2019] Hyperbolic
 Disk Embeddings
 [Suzuki, ICML 2019] &NCFEEJOH )ZQFSCPMJD 0SEFS 1SPCBCJMJTUJD
  4. 7 Embedding Directed Acyclic Graphs (DAGs) • Semantics, citation networks,

    genealogical networks • Asymmetric / transitive relation of nodes
 → Partially ordered set (poset) DAG Embedding • Embedding nodes into continuous poset
 so that transitive relation is preserved. DAG Embedding Models
  5. 9 2D Euclidean Disk Embedding = Eular Diagram • Disk

    Embedding: General framework for embedding DAGs Our Contributions • Novel Hyperbolic Disk Embedding • Experiments • Theorems: Existing methods are special cases of DE
 (+ extra restrictions)
  6. 11 Disk Embedding Disk Embedding: DAG relation → Inclusive relation

    of disks Disk = Center + Radius X y x dX (x, y) + rx − ry ≤ 0 Dx = (x, rx )
  7. 12 Embedding DAGs Ancestors = Upper cone Descendants = Lower

    cone Transitive relation of DAG induces “cones” in the embedding space.
  8. 13 Key Idea = Inclusion of projected disks X =

    Inclusion of lower cones y x Relation of DAG nodes
  9. 13 Key Idea = Inclusion of projected disks X =

    Inclusion of lower cones y x Relation of DAG nodes X y x dX (x, y) + rx − ry ≤ 0 = Disk Embedding
  10. 15 Disk Embedding as General Framework Disk Embedding is easily

    extensible to general metric spaces Further generalizations • Metric spaces → Quasi-metric spaces • (Closed) Disks → Formal disks Existing methods can be understood as special cases of DEs X y x
  11. 16 Quasi-metric Spaces ∥x∥W = 1 w1 w2 w3 ∥x∥W

    := max i {w⊤ i x} Proposition 2 ∥x∥1 = 1 ∥x∥∞ = 1 ∥x∥2 = 1 L1-distance Euclidean distance Uniform distance Symmetric quasi-metric (metric) Asymmetric quasi-metric Unit disk Polyhedral quasimetric
  12. 17 Formal Disks Abstraction of closed / open disks y

    x y x Closed disk Open disk Dx ⊆ Dy D∘ x ⊆ D∘ y (x, r) (x, rx ) ⊑ (y, ry ) d(x, y) + rx − ry ≤ 0 Formal Disk Radii can be negative → Reversibility of Disk Embeddings
  13. 19 Equivalence of models Order Embeddings [Vendrov 2016] (a) ϕ

    Euclidean Disk Embedding (w/ polyhedral quasimetric) xk ≥ yk for k = 1,⋯, n (x′, r) = ( Px, a − 1 n n ∑ k=1 xk) dW (x′, y′) + rx − ry ≤ 0
  14. 20 Equivalence of models (a) (b) ϕ Spherical Disk Embedding

    d (x′, y′) + rx − ry ≤ 0 Hyperbolic Entailment Cones
 [Ganea, ICML 2018] (x′, r) = x ∥x∥ , arcsin ( 1 + ∥x∥2 2∥x∥ sin θ0) ψ(x) ≥ Ξ(x, y)
  15. 22 Limitations of Existing Methods Order Embeddings
 [Vendrov, ICLR 2016]

    Hyperbolic Entailment Cones
 [Ganea, ICML 2018] “Origin” Existing methods implicitly assumes the existence of the origin
 ≒ Assuming rooted hierarchies
  16. 23 Advantage of Disk Embedding Extends existing methods • Negative

    radius: reversibility & translational symmetry • Avoid gradient vanishing on loss functions Applicable for various (quasi-)metric spaces. Positive sample Negative sample Case of Negative Radius Loss Functions Ours Existing
 methods
  17. 25 Base Space Vector Embedding Disk Embedding Euclidean space Word2Vec

    etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017) Hyperbolic
 Disk Embedding
  18. 26 Hyperbolic Disk Embedding 2 1 3 4 5 6

    2 1 3 4 5 6 7 More neighboring disks in hyperbolic space All disks are of the same radius Euclidean Hyperbolic
  19. 29 Experimental Results Hierarchy Non-
 Hierarchy Our Disk Embedding models

    achieved significant improvement especially in non-hierarchical data
  20. 31 • Disk Embedding: General framework for embedding DAGs Conclusion

    • Novel Hyperbolic Disk Embedding • Experiments • Theorems: Existing methods are special cases of DE
 (+ extra restrictions)
  21. 33 Future Works: Theories ʙ     Word2Vec


    [Mikolov, NIPS2013] Various Applications Poincaré Embedding
 [Nickel, NIPS 2017] Order Embedding
 [Vendrov, ICLR2016] Probabilistic
 Order Embedding
 [Lai, EACL2017] Hyperbolic
 Entailment Cones
 [Ganea, ICML2018] Box Embedding
 [Vilnis ACL 2018] Lorentz Embedding
 [Nickel, ICML 2018] Smoothing
 Box Embedding
 [Li, ICLR 2019] Hyperbolic
 Disk Embeddings
 [Suzuki, ICML 2019] &NCFEEJOH )ZQFSCPMJD 0SEFS 1SPCBCJMJTUJD .PSFHFOFSBMJ[BUJPOT
 GPSQSPCBCJMJTUJDNPEFMT