Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Hyperbolic Disk Embeddings for Directed Acyclic...

Hyperbolic Disk Embeddings for Directed Acyclic Graphs

Presentation in "scouty Knowledge Graph and Embedding Night"
https://scouty.connpass.com/event/114430/

Ryota Suzuki / ぬぬき

February 14, 2019
Tweet

More Decks by Ryota Suzuki / ぬぬき

Other Decks in Technology

Transcript

  1. scouty Knowledge Graph and Embedding Night Hyperbolic Disk Embeddings for

    
 Directed Acyclic Graphs 2019.2.14 Ryota Suzuki scouty Inc.
  2. ࣗݾ঺հ ❖ Ryota Suzuki (͵͵͖)
 Twitter: nunuki_ GitHub: nunukim /&$

    म࢜՝ఔ ෺ੑ෺ཧ ػցֶश Τ͵δχΞ 2018.11 தԝݚڀॴ ৴߸ॲཧ / ػցֶश
  3. Embedding methods ❖ Embedding symbolic entity
 in continuous space (e.g.

    Euclidean space) ❖ Used as a first layer of NN ❖ Obtains relation of entities
 as geometrical structure ❖ This paper: Embedding directed acyclic graphs (DAGs) vwoman − vman + vking = vqueen
  4. Directed Acyclic Graphs (DAGs) ❖ Directed graphs without cycles. ❖

    Directed reachability between nodes forms a partial ordering. B A Root nodes Leaf nodes Intermediate nodes
  5. Partially ordered set (poset) ❖ Transitive, reflexive, anti-symmetric binary relation.

    ❖ Transitivity: x ⪯ y ∧ y ⪯ z ⇒ x ⪯ z Lower cone Upper cone C⪯ (x) = {y|y ⪯ x} C⪰ (x) = {y|y ⪰ x} y ⪯ x C⪯ (x) ⊆ C⪯ (y) ⇔ Partial ordering = Inclusive relation of lower cones x Proposition 1 x y
  6. Embedding DAGs in poset ❖ Embedding graph structure so that

    transitive relation (reachability) is preserved. Ancestors = Upper cone Descendants = Lower cone
  7. Hierarchies and complex DAGs Complex DAG Tree-like hierarchy Exponentially increasing

    descendants, few ancestors Exponentially increasing descendants AND ancestors • Organization chart • Semantic hierarchy of words • Citation network • Family genealogy • Dependency network of libraries • Manufacturing system … • Translational
 symmetry • Reversibility
  8. Disadvantages of existing methods (Vendrov; ICLR 2016) (Ganea; ICML 2018)

    Assuming the special “origin” (≒ the root of tree). ⇒ Implicitly assuming tree-like structures. The “origin"
  9. Contribution of the paper ❖ Proposed Disk Embeddings. ❖ A

    general framework for DAG embeddings. ❖ Hyperbolic DE. ❖ Proved existing methods = DE with extra assumptions. ❖ Order Embeddings (Vendrov 2016) = Euclidean DE ❖ Hyperbolic Cones (Ganea 2018) = Spherical DE ❖ Evaluation of effectiveness by experiment.
  10. Quasi-metric spaces ❖ Symmetric (metric) Quasi-metrics on ∥x∥1 = 1

    ∥x∥∞ = 1 ∥x∥2 = 1 ∥x∥W = 1 w1 w2 w3 L1-distance Euclidean distance Uniform distance Polyhedral quasi-metric ℝn ∥x∥W := max i {w⊤ i x} Proposition 2 ❖ Asymmetric Unit disk
  11. Formal disks ❖ (X,d): quasi-metric space (X: set, d: quasi-metric

    on X) x r x r Closed disk D(x, r) ❖ Inclusive Relation of closed (open) disks D(x, r) ⊆ D(y, s) ⟺ D∘(x, r) ⊆ D∘(y, s) ⟺ d(x, y) ≤ s − r y s r x y s r x (x, r) ⊑ (y, s) ⇔ Def Formal disk: abstraction of open and closed disks Open disk D∘(x, r) Partial ordering ∈ B+(X) = X × ℝ+
  12. Generalized Formal Disks ❖ Partial ordering of formal disks
 ❖

    Radii need not to be positive
 ⇒ Generalized formal disks ❖ Transitive symmetry ❖ Reversibility (y, s) ⊑ (x, r) ⇔ d(x, y) ≤ r − s Fig. 1 Lower cones of formal disks Radius Center Negative radius
  13. Disk Embeddings ❖ Given quasi-metric space (X,d) ❖ Embedding DAGs:

    ❖ Nodes: formal disk ❖ Edges: Partial ordering of formal disks (xi , ri ) (xi , ri ) ⊑ (xj , rj ) United Kingdom Ireland (island) Ireland (state) Nothern Ireland England Scotland Wales Fig. 2 2D Euclidean DE (r > 0) = 2D Euler diagrams of circles
  14. = Inclusion of projected disks
 = Disk Embedding Equivalence of

    Existing Methods x y DAG Embedding
 = Partial ordering
 = Inclusion of lower cones Motivation
  15. Order Embeddings (Vendrov 2016) (a) (b) Reversed product order on

    n ⋀ i=1 xi ≥ yi dW (u, v) ≤ r − s Formal disks on hyperplane Hn−1 (u, r) = ( Px, a − 1 n n ∑ i=1 xi) ϕord Order Embeddings Euclidean Disk Embedding ℝn Polyhedral quasi-metric
  16. Hyperbolic Cones (Ganea 2018) Reversed product order on ψ(x) ≥

    Ξ(x, y) d (u, v) ≤ r − s Formal disks on (n-1)-sphere Sn−1 (u, r) = x ∥x∥ , arcsin ( 1 + ∥x∥2 2∥x∥ sin θ0) ϕhyp Hyperbolic Entailment Cones Spherical Disk Embedding ℝn (a) (b)
  17. Vector and Disk Embeddings Base Space Vector Embedding Disk Embedding

    Euclidean space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018)
  18. Vector and Disk Embeddings Base Space Vector Embedding Disk Embedding

    Euclidean space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017)
  19. Hyperbolic geometry ❖ Non-Euclidean geometry ❖ Constant negative curvature ❖

    Interior angles of triangle < 180° ❖ Several equivalent models ❖ Poincaré ball model ❖ Hyperboloid (Lorentz) model ❖ Can embed any weighted tree
 w/o losing geometrical properties Poincaré disk model http://nunuki.hatenablog.com/entry/2018/12/15/055136
  20. Poincaré Embedding ❖ Embedding entities into
 hyperbolic space such that


    distances are preserved. ❖ Only few (~5) dimensional
 PE outperforms 200D
 Euclidean methods
  21. Hyperbolic Disk Embedding Base Space Vector Embedding Disk Embedding Euclidean

    space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017) Hyperbolic
 Disk Embedding
  22. Hyperbolic Disk Embedding ❖ Disk Embedding in hyperbolic geometry 2

    1 3 4 5 6 2 1 3 4 5 6 7 Euclidean Hyperbolic More neighboring disks in hyperbolic space All disks are of
 the same radius
  23. Loss functions ❖ Margin Loss Eij = d(xi , xj

    ) − ri + rj h+ (x) = max(x, 0) Positive samples Negative samples ❖ Inclusion of formal disks (x, r) ⊑ (y, s) ⟺ d(x, y) ≤ s − r Eij
  24. Riemannian SGD ❖ SGD on Riemannian manifold. ❖ Gradient →

    Riemannian gradient ❖ Delta update → Update via exponential map Ordinary (Euclidean) SGD Θt+1 = Θt − η∇L Riemannian SGD Θt+1 = expΘt (−η∇X L) Exponential map Riemannian gradient
  25. RSGD on formal disks ❖ Collection of formal disks ❖

    Gradient on B(X) ❖ Exponential map on B(X) B(X) = X × ℝ Riemannian manifold ∇B(X) L = (∇X L, ∂L ∂r ) expB(X) (x,r) (u, t) = (expX x (u), r + t) Euclidian space
  26. Experimental Data ❖ WordNet ❖ Large lexical database ❖ Contains

    “is-a” relation of nouns ❖ Tree-like structure ❖ Reversed WordNet ❖ Reversed relations of WordNet ❖ Non tree-like structure
  27. Baseline methods ❖ Poincare Embedding (Nickel 2017) ❖ Order Embedding

    (Vendrov 2016) ❖ Equivalent to Euclidean Disk Embedding ❖ Hyperbolic Entailment Cones (Ganea 2018) ❖ Equivalent to Spherical Disk Embedding
  28. Conclusions ❖ Proposed Disk Embeddings. ❖ A general framework for

    DAG embeddings. ❖ Hyperbolic DE. ❖ Proved existing methods = DE with extra assumptions. ❖ Order Embeddings (Vendrov 2016) = Euclidean DE ❖ Hyperbolic Cones (Ganea 2018) = Spherical DE ❖ Evaluation of effectiveness by experiment.