5.5k

# Hyperbolic Disk Embeddings for Directed Acyclic Graphs

Presentation in "scouty Knowledge Graph and Embedding Night"
https://scouty.connpass.com/event/114430/

## ぬぬき

February 14, 2019

## Transcript

1. ### scouty Knowledge Graph and Embedding Night Hyperbolic Disk Embeddings for

Directed Acyclic Graphs 2019.2.14 Ryota Suzuki scouty Inc.
2. ### ࣗݾ঺հ ❖ Ryota Suzuki (͵͵͖)  Twitter: nunuki_ GitHub: nunukim /&\$

म࢜՝ఔ ෺ੑ෺ཧ ػցֶश Τ͵δχΞ 2018.11 தԝݚڀॴ ৴߸ॲཧ / ػցֶश

4. ### Contents ❖ Introduction ❖ Disk Embedding Models ❖ Learning Method

❖ Experiments

6. ### Embedding methods ❖ Embedding symbolic entity  in continuous space (e.g.

Euclidean space) ❖ Used as a ﬁrst layer of NN ❖ Obtains relation of entities  as geometrical structure ❖ This paper: Embedding directed acyclic graphs (DAGs) vwoman − vman + vking = vqueen
7. ### Directed Acyclic Graphs (DAGs) ❖ Directed graphs without cycles. ❖

Directed reachability between nodes forms a partial ordering. B A Root nodes Leaf nodes Intermediate nodes
8. ### Partially ordered set (poset) ❖ Transitive, reﬂexive, anti-symmetric binary relation.

❖ Transitivity: x ⪯ y ∧ y ⪯ z ⇒ x ⪯ z Lower cone Upper cone C⪯ (x) = {y|y ⪯ x} C⪰ (x) = {y|y ⪰ x} y ⪯ x C⪯ (x) ⊆ C⪯ (y) ⇔ Partial ordering = Inclusive relation of lower cones x Proposition 1 x y
9. ### Embedding DAGs in poset ❖ Embedding graph structure so that

transitive relation (reachability) is preserved. Ancestors = Upper cone Descendants = Lower cone
10. ### Existing Methods Order Embeddings  (Vendrov; ICLR 2016) Hyperbolic Entailment Cones

(Ganea; ICML 2018) Lower cones
11. ### Hierarchies and complex DAGs Complex DAG Tree-like hierarchy Exponentially increasing

descendants, few ancestors Exponentially increasing descendants AND ancestors • Organization chart • Semantic hierarchy of words • Citation network • Family genealogy • Dependency network of libraries • Manufacturing system … • Translational  symmetry • Reversibility
12. ### Disadvantages of existing methods (Vendrov; ICLR 2016) (Ganea; ICML 2018)

Assuming the special “origin” (≒ the root of tree). ⇒ Implicitly assuming tree-like structures. The “origin"
13. ### Contribution of the paper ❖ Proposed Disk Embeddings. ❖ A

general framework for DAG embeddings. ❖ Hyperbolic DE. ❖ Proved existing methods = DE with extra assumptions. ❖ Order Embeddings (Vendrov 2016) = Euclidean DE ❖ Hyperbolic Cones (Ganea 2018) = Spherical DE ❖ Evaluation of effectiveness by experiment.

15. ### Quasi-metric spaces ❖ Symmetric (metric) Quasi-metrics on ∥x∥1 = 1

∥x∥∞ = 1 ∥x∥2 = 1 ∥x∥W = 1 w1 w2 w3 L1-distance Euclidean distance Uniform distance Polyhedral quasi-metric ℝn ∥x∥W := max i {w⊤ i x} Proposition 2 ❖ Asymmetric Unit disk
16. ### Formal disks ❖ (X,d): quasi-metric space (X: set, d: quasi-metric

on X) x r x r Closed disk D(x, r) ❖ Inclusive Relation of closed (open) disks D(x, r) ⊆ D(y, s) ⟺ D∘(x, r) ⊆ D∘(y, s) ⟺ d(x, y) ≤ s − r y s r x y s r x (x, r) ⊑ (y, s) ⇔ Def Formal disk: abstraction of open and closed disks Open disk D∘(x, r) Partial ordering ∈ B+(X) = X × ℝ+
17. ### Generalized Formal Disks ❖ Partial ordering of formal disks  ❖

Radii need not to be positive  ⇒ Generalized formal disks ❖ Transitive symmetry ❖ Reversibility (y, s) ⊑ (x, r) ⇔ d(x, y) ≤ r − s Fig. 1 Lower cones of formal disks Radius Center Negative radius

19. ### Disk Embeddings ❖ Given quasi-metric space (X,d) ❖ Embedding DAGs:

❖ Nodes: formal disk ❖ Edges: Partial ordering of formal disks (xi , ri ) (xi , ri ) ⊑ (xj , rj ) United Kingdom Ireland (island) Ireland (state) Nothern Ireland England Scotland Wales Fig. 2 2D Euclidean DE (r > 0) = 2D Euler diagrams of circles
20. ### = Inclusion of projected disks  = Disk Embedding Equivalence of

Existing Methods x y DAG Embedding  = Partial ordering  = Inclusion of lower cones Motivation
21. ### Order Embeddings (Vendrov 2016) (a) (b) Reversed product order on

n ⋀ i=1 xi ≥ yi dW (u, v) ≤ r − s Formal disks on hyperplane Hn−1 (u, r) = ( Px, a − 1 n n ∑ i=1 xi) ϕord Order Embeddings Euclidean Disk Embedding ℝn Polyhedral quasi-metric
22. ### Hyperbolic Cones (Ganea 2018) Reversed product order on ψ(x) ≥

Ξ(x, y) d (u, v) ≤ r − s Formal disks on (n-1)-sphere Sn−1 (u, r) = x ∥x∥ , arcsin ( 1 + ∥x∥2 2∥x∥ sin θ0) ϕhyp Hyperbolic Entailment Cones Spherical Disk Embedding ℝn (a) (b)
23. ### Vector and Disk Embeddings Base Space Vector Embedding Disk Embedding

Euclidean space Word2Vec etc.  (Minkolov 2013) Order Embeddings  (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones  (Ganea 2018)
24. ### Vector and Disk Embeddings Base Space Vector Embedding Disk Embedding

Euclidean space Word2Vec etc.  (Minkolov 2013) Order Embeddings  (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones  (Ganea 2018) Hyperbolic space Poincaré Embedding  (Nickel 2017)
25. ### Hyperbolic geometry ❖ Non-Euclidean geometry ❖ Constant negative curvature ❖

Interior angles of triangle < 180° ❖ Several equivalent models ❖ Poincaré ball model ❖ Hyperboloid (Lorentz) model ❖ Can embed any weighted tree  w/o losing geometrical properties Poincaré disk model http://nunuki.hatenablog.com/entry/2018/12/15/055136
26. ### Poincaré Embedding ❖ Embedding entities into  hyperbolic space such that

distances are preserved. ❖ Only few (~5) dimensional  PE outperforms 200D  Euclidean methods
27. ### Hyperbolic Disk Embedding Base Space Vector Embedding Disk Embedding Euclidean

space Word2Vec etc.  (Minkolov 2013) Order Embeddings  (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones  (Ganea 2018) Hyperbolic space Poincaré Embedding  (Nickel 2017) Hyperbolic  Disk Embedding
28. ### Hyperbolic Disk Embedding ❖ Disk Embedding in hyperbolic geometry 2

1 3 4 5 6 2 1 3 4 5 6 7 Euclidean Hyperbolic More neighboring disks in hyperbolic space All disks are of  the same radius

30. ### Loss functions ❖ Margin Loss Eij = d(xi , xj

) − ri + rj h+ (x) = max(x, 0) Positive samples Negative samples ❖ Inclusion of formal disks (x, r) ⊑ (y, s) ⟺ d(x, y) ≤ s − r Eij
31. ### Loss functions of existing methods This paper Vanishing gradient  (Vendor

2016, Ganda 2018)
32. ### Riemannian SGD ❖ SGD on Riemannian manifold. ❖ Gradient →

Riemannian gradient ❖ Delta update → Update via exponential map Ordinary (Euclidean) SGD Θt+1 = Θt − η∇L Riemannian SGD Θt+1 = expΘt (−η∇X L) Exponential map Riemannian gradient
33. ### RSGD on formal disks ❖ Collection of formal disks ❖

Gradient on B(X) ❖ Exponential map on B(X) B(X) = X × ℝ Riemannian manifold ∇B(X) L = (∇X L, ∂L ∂r ) expB(X) (x,r) (u, t) = (expX x (u), r + t) Euclidian space
34. ### Frequently used geometries ❖ Euclidean geometry ❖ Ordinary SGD ❖

Spherical geometry ❖ Hyperbolic geometry

36. ### Experimental Data ❖ WordNet ❖ Large lexical database ❖ Contains

“is-a” relation of nouns ❖ Tree-like structure ❖ Reversed WordNet ❖ Reversed relations of WordNet ❖ Non tree-like structure
37. ### Baseline methods ❖ Poincare Embedding (Nickel 2017) ❖ Order Embedding

(Vendrov 2016) ❖ Equivalent to Euclidean Disk Embedding ❖ Hyperbolic Entailment Cones (Ganea 2018) ❖ Equivalent to Spherical Disk Embedding