Save 37% off PRO during our Black Friday Sale! »

Hyperbolic Disk Embeddings for Directed Acyclic Graphs

2c4e06a13a944723032415c88e96cbe5?s=47 ぬぬき
February 14, 2019

Hyperbolic Disk Embeddings for Directed Acyclic Graphs

Presentation in "scouty Knowledge Graph and Embedding Night"
https://scouty.connpass.com/event/114430/

2c4e06a13a944723032415c88e96cbe5?s=128

ぬぬき

February 14, 2019
Tweet

Transcript

  1. scouty Knowledge Graph and Embedding Night Hyperbolic Disk Embeddings for

    
 Directed Acyclic Graphs 2019.2.14 Ryota Suzuki scouty Inc.
  2. ࣗݾ঺հ ❖ Ryota Suzuki (͵͵͖)
 Twitter: nunuki_ GitHub: nunukim /&$

    म࢜՝ఔ ෺ੑ෺ཧ ػցֶश Τ͵δχΞ 2018.11 தԝݚڀॴ ৴߸ॲཧ / ػցֶश
  3. Hyperbolic Disk Embedding for Directed Acyclic Graphs https://arxiv.org/pdf/1902.04335.pdf

  4. Contents ❖ Introduction ❖ Disk Embedding Models ❖ Learning Method

    ❖ Experiments
  5. Introduction

  6. Embedding methods ❖ Embedding symbolic entity
 in continuous space (e.g.

    Euclidean space) ❖ Used as a first layer of NN ❖ Obtains relation of entities
 as geometrical structure ❖ This paper: Embedding directed acyclic graphs (DAGs) vwoman − vman + vking = vqueen
  7. Directed Acyclic Graphs (DAGs) ❖ Directed graphs without cycles. ❖

    Directed reachability between nodes forms a partial ordering. B A Root nodes Leaf nodes Intermediate nodes
  8. Partially ordered set (poset) ❖ Transitive, reflexive, anti-symmetric binary relation.

    ❖ Transitivity: x ⪯ y ∧ y ⪯ z ⇒ x ⪯ z Lower cone Upper cone C⪯ (x) = {y|y ⪯ x} C⪰ (x) = {y|y ⪰ x} y ⪯ x C⪯ (x) ⊆ C⪯ (y) ⇔ Partial ordering = Inclusive relation of lower cones x Proposition 1 x y
  9. Embedding DAGs in poset ❖ Embedding graph structure so that

    transitive relation (reachability) is preserved. Ancestors = Upper cone Descendants = Lower cone
  10. Existing Methods Order Embeddings
 (Vendrov; ICLR 2016) Hyperbolic Entailment Cones


    (Ganea; ICML 2018) Lower cones
  11. Hierarchies and complex DAGs Complex DAG Tree-like hierarchy Exponentially increasing

    descendants, few ancestors Exponentially increasing descendants AND ancestors • Organization chart • Semantic hierarchy of words • Citation network • Family genealogy • Dependency network of libraries • Manufacturing system … • Translational
 symmetry • Reversibility
  12. Disadvantages of existing methods (Vendrov; ICLR 2016) (Ganea; ICML 2018)

    Assuming the special “origin” (≒ the root of tree). ⇒ Implicitly assuming tree-like structures. The “origin"
  13. Contribution of the paper ❖ Proposed Disk Embeddings. ❖ A

    general framework for DAG embeddings. ❖ Hyperbolic DE. ❖ Proved existing methods = DE with extra assumptions. ❖ Order Embeddings (Vendrov 2016) = Euclidean DE ❖ Hyperbolic Cones (Ganea 2018) = Spherical DE ❖ Evaluation of effectiveness by experiment.
  14. Disk Embeddings

  15. Quasi-metric spaces ❖ Symmetric (metric) Quasi-metrics on ∥x∥1 = 1

    ∥x∥∞ = 1 ∥x∥2 = 1 ∥x∥W = 1 w1 w2 w3 L1-distance Euclidean distance Uniform distance Polyhedral quasi-metric ℝn ∥x∥W := max i {w⊤ i x} Proposition 2 ❖ Asymmetric Unit disk
  16. Formal disks ❖ (X,d): quasi-metric space (X: set, d: quasi-metric

    on X) x r x r Closed disk D(x, r) ❖ Inclusive Relation of closed (open) disks D(x, r) ⊆ D(y, s) ⟺ D∘(x, r) ⊆ D∘(y, s) ⟺ d(x, y) ≤ s − r y s r x y s r x (x, r) ⊑ (y, s) ⇔ Def Formal disk: abstraction of open and closed disks Open disk D∘(x, r) Partial ordering ∈ B+(X) = X × ℝ+
  17. Generalized Formal Disks ❖ Partial ordering of formal disks
 ❖

    Radii need not to be positive
 ⇒ Generalized formal disks ❖ Transitive symmetry ❖ Reversibility (y, s) ⊑ (x, r) ⇔ d(x, y) ≤ r − s Fig. 1 Lower cones of formal disks Radius Center Negative radius
  18. Disk Embedding Models

  19. Disk Embeddings ❖ Given quasi-metric space (X,d) ❖ Embedding DAGs:

    ❖ Nodes: formal disk ❖ Edges: Partial ordering of formal disks (xi , ri ) (xi , ri ) ⊑ (xj , rj ) United Kingdom Ireland (island) Ireland (state) Nothern Ireland England Scotland Wales Fig. 2 2D Euclidean DE (r > 0) = 2D Euler diagrams of circles
  20. = Inclusion of projected disks
 = Disk Embedding Equivalence of

    Existing Methods x y DAG Embedding
 = Partial ordering
 = Inclusion of lower cones Motivation
  21. Order Embeddings (Vendrov 2016) (a) (b) Reversed product order on

    n ⋀ i=1 xi ≥ yi dW (u, v) ≤ r − s Formal disks on hyperplane Hn−1 (u, r) = ( Px, a − 1 n n ∑ i=1 xi) ϕord Order Embeddings Euclidean Disk Embedding ℝn Polyhedral quasi-metric
  22. Hyperbolic Cones (Ganea 2018) Reversed product order on ψ(x) ≥

    Ξ(x, y) d (u, v) ≤ r − s Formal disks on (n-1)-sphere Sn−1 (u, r) = x ∥x∥ , arcsin ( 1 + ∥x∥2 2∥x∥ sin θ0) ϕhyp Hyperbolic Entailment Cones Spherical Disk Embedding ℝn (a) (b)
  23. Vector and Disk Embeddings Base Space Vector Embedding Disk Embedding

    Euclidean space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018)
  24. Vector and Disk Embeddings Base Space Vector Embedding Disk Embedding

    Euclidean space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017)
  25. Hyperbolic geometry ❖ Non-Euclidean geometry ❖ Constant negative curvature ❖

    Interior angles of triangle < 180° ❖ Several equivalent models ❖ Poincaré ball model ❖ Hyperboloid (Lorentz) model ❖ Can embed any weighted tree
 w/o losing geometrical properties Poincaré disk model http://nunuki.hatenablog.com/entry/2018/12/15/055136
  26. Poincaré Embedding ❖ Embedding entities into
 hyperbolic space such that


    distances are preserved. ❖ Only few (~5) dimensional
 PE outperforms 200D
 Euclidean methods
  27. Hyperbolic Disk Embedding Base Space Vector Embedding Disk Embedding Euclidean

    space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017) Hyperbolic
 Disk Embedding
  28. Hyperbolic Disk Embedding ❖ Disk Embedding in hyperbolic geometry 2

    1 3 4 5 6 2 1 3 4 5 6 7 Euclidean Hyperbolic More neighboring disks in hyperbolic space All disks are of
 the same radius
  29. Learning Method

  30. Loss functions ❖ Margin Loss Eij = d(xi , xj

    ) − ri + rj h+ (x) = max(x, 0) Positive samples Negative samples ❖ Inclusion of formal disks (x, r) ⊑ (y, s) ⟺ d(x, y) ≤ s − r Eij
  31. Loss functions of existing methods This paper Vanishing gradient
 (Vendor

    2016, Ganda 2018)
  32. Riemannian SGD ❖ SGD on Riemannian manifold. ❖ Gradient →

    Riemannian gradient ❖ Delta update → Update via exponential map Ordinary (Euclidean) SGD Θt+1 = Θt − η∇L Riemannian SGD Θt+1 = expΘt (−η∇X L) Exponential map Riemannian gradient
  33. RSGD on formal disks ❖ Collection of formal disks ❖

    Gradient on B(X) ❖ Exponential map on B(X) B(X) = X × ℝ Riemannian manifold ∇B(X) L = (∇X L, ∂L ∂r ) expB(X) (x,r) (u, t) = (expX x (u), r + t) Euclidian space
  34. Frequently used geometries ❖ Euclidean geometry ❖ Ordinary SGD ❖

    Spherical geometry ❖ Hyperbolic geometry
  35. Experiments

  36. Experimental Data ❖ WordNet ❖ Large lexical database ❖ Contains

    “is-a” relation of nouns ❖ Tree-like structure ❖ Reversed WordNet ❖ Reversed relations of WordNet ❖ Non tree-like structure
  37. Baseline methods ❖ Poincare Embedding (Nickel 2017) ❖ Order Embedding

    (Vendrov 2016) ❖ Equivalent to Euclidean Disk Embedding ❖ Hyperbolic Entailment Cones (Ganea 2018) ❖ Equivalent to Spherical Disk Embedding
  38. Results Tree-like data: improvement on loss function Reversed tree-like data:

    maintains reversibility (allows negative radius)
  39. Conclusions ❖ Proposed Disk Embeddings. ❖ A general framework for

    DAG embeddings. ❖ Hyperbolic DE. ❖ Proved existing methods = DE with extra assumptions. ❖ Order Embeddings (Vendrov 2016) = Euclidean DE ❖ Hyperbolic Cones (Ganea 2018) = Spherical DE ❖ Evaluation of effectiveness by experiment.
  40. Future work This paper Tree-like Reversed tree-like Complex DAGs
 e.g.

    citation network
  41. Thank you for listening!

  42. “ ”