Hyperbolic Disk Embeddings for Directed Acyclic Graphs

Presentation in "scouty Knowledge Graph and Embedding Night"
https://scouty.connpass.com/event/114430/

Transcript

1. scouty Knowledge Graph and Embedding Night Hyperbolic Disk Embeddings for 
 Directed Acyclic Graphs 2019.2.14 Ryota Suzuki scouty Inc.
2. ࣗݾ঺հ ❖ Ryota Suzuki (͵͵͖)
 Twitter: nunuki_ GitHub: nunukim /&$ म࢜՝ఔ ෺ੑ෺ཧ ػցֶश Τ͵δχΞ 2018.11 தԝݚڀॴ ৴߸ॲཧ / ػցֶश
3. Hyperbolic Disk Embedding for Directed Acyclic Graphs https://arxiv.org/pdf/1902.04335.pdf
4. Contents ❖ Introduction ❖ Disk Embedding Models ❖ Learning Method ❖ Experiments
5. Introduction
6. Embedding methods ❖ Embedding symbolic entity
 in continuous space (e.g. Euclidean space) ❖ Used as a first layer of NN ❖ Obtains relation of entities
 as geometrical structure ❖ This paper: Embedding directed acyclic graphs (DAGs) vwoman − vman + vking = vqueen
7. Directed Acyclic Graphs (DAGs) ❖ Directed graphs without cycles. ❖ Directed reachability between nodes forms a partial ordering. B A Root nodes Leaf nodes Intermediate nodes
8. Partially ordered set (poset) ❖ Transitive, reflexive, anti-symmetric binary relation. ❖ Transitivity: x ⪯ y ∧ y ⪯ z ⇒ x ⪯ z Lower cone Upper cone C⪯ (x) = {y|y ⪯ x} C⪰ (x) = {y|y ⪰ x} y ⪯ x C⪯ (x) ⊆ C⪯ (y) ⇔ Partial ordering = Inclusive relation of lower cones x Proposition 1 x y
9. Embedding DAGs in poset ❖ Embedding graph structure so that transitive relation (reachability) is preserved. Ancestors = Upper cone Descendants = Lower cone
10. Existing Methods Order Embeddings
 (Vendrov; ICLR 2016) Hyperbolic Entailment Cones
 (Ganea; ICML 2018) Lower cones
11. Hierarchies and complex DAGs Complex DAG Tree-like hierarchy Exponentially increasing descendants, few ancestors Exponentially increasing descendants AND ancestors • Organization chart • Semantic hierarchy of words • Citation network • Family genealogy • Dependency network of libraries • Manufacturing system … • Translational
 symmetry • Reversibility
12. Disadvantages of existing methods (Vendrov; ICLR 2016) (Ganea; ICML 2018) Assuming the special “origin” (≒ the root of tree). ⇒ Implicitly assuming tree-like structures. The “origin"
13. Contribution of the paper ❖ Proposed Disk Embeddings. ❖ A general framework for DAG embeddings. ❖ Hyperbolic DE. ❖ Proved existing methods = DE with extra assumptions. ❖ Order Embeddings (Vendrov 2016) = Euclidean DE ❖ Hyperbolic Cones (Ganea 2018) = Spherical DE ❖ Evaluation of effectiveness by experiment.
14. Disk Embeddings
15. Quasi-metric spaces ❖ Symmetric (metric) Quasi-metrics on ∥x∥1 = 1 ∥x∥∞ = 1 ∥x∥2 = 1 ∥x∥W = 1 w1 w2 w3 L1-distance Euclidean distance Uniform distance Polyhedral quasi-metric ℝn ∥x∥W := max i {w⊤ i x} Proposition 2 ❖ Asymmetric Unit disk
16. Formal disks ❖ (X,d): quasi-metric space (X: set, d: quasi-metric on X) x r x r Closed disk D(x, r) ❖ Inclusive Relation of closed (open) disks D(x, r) ⊆ D(y, s) ⟺ D∘(x, r) ⊆ D∘(y, s) ⟺ d(x, y) ≤ s − r y s r x y s r x (x, r) ⊑ (y, s) ⇔ Def Formal disk: abstraction of open and closed disks Open disk D∘(x, r) Partial ordering ∈ B+(X) = X × ℝ+
17. Generalized Formal Disks ❖ Partial ordering of formal disks
 ❖ Radii need not to be positive
 ⇒ Generalized formal disks ❖ Transitive symmetry ❖ Reversibility (y, s) ⊑ (x, r) ⇔ d(x, y) ≤ r − s Fig. 1 Lower cones of formal disks Radius Center Negative radius
18. Disk Embedding Models
19. Disk Embeddings ❖ Given quasi-metric space (X,d) ❖ Embedding DAGs: ❖ Nodes: formal disk ❖ Edges: Partial ordering of formal disks (xi , ri ) (xi , ri ) ⊑ (xj , rj ) United Kingdom Ireland (island) Ireland (state) Nothern Ireland England Scotland Wales Fig. 2 2D Euclidean DE (r > 0) = 2D Euler diagrams of circles
20. = Inclusion of projected disks
 = Disk Embedding Equivalence of Existing Methods x y DAG Embedding
 = Partial ordering
 = Inclusion of lower cones Motivation
21. Order Embeddings (Vendrov 2016) (a) (b) Reversed product order on n ⋀ i=1 xi ≥ yi dW (u, v) ≤ r − s Formal disks on hyperplane Hn−1 (u, r) = ( Px, a − 1 n n ∑ i=1 xi) ϕord Order Embeddings Euclidean Disk Embedding ℝn Polyhedral quasi-metric
22. Hyperbolic Cones (Ganea 2018) Reversed product order on ψ(x) ≥ Ξ(x, y) d (u, v) ≤ r − s Formal disks on (n-1)-sphere Sn−1 (u, r) = x ∥x∥ , arcsin ( 1 + ∥x∥2 2∥x∥ sin θ0) ϕhyp Hyperbolic Entailment Cones Spherical Disk Embedding ℝn (a) (b)
23. Vector and Disk Embeddings Base Space Vector Embedding Disk Embedding Euclidean space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018)
24. Vector and Disk Embeddings Base Space Vector Embedding Disk Embedding Euclidean space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017)
25. Hyperbolic geometry ❖ Non-Euclidean geometry ❖ Constant negative curvature ❖ Interior angles of triangle < 180° ❖ Several equivalent models ❖ Poincaré ball model ❖ Hyperboloid (Lorentz) model ❖ Can embed any weighted tree
 w/o losing geometrical properties Poincaré disk model http://nunuki.hatenablog.com/entry/2018/12/15/055136
26. Poincaré Embedding ❖ Embedding entities into
 hyperbolic space such that
 distances are preserved. ❖ Only few (~5) dimensional
 PE outperforms 200D
 Euclidean methods
27. Hyperbolic Disk Embedding Base Space Vector Embedding Disk Embedding Euclidean space Word2Vec etc.
 (Minkolov 2013) Order Embeddings
 (Vendrov 2016) Sphere Several approaches Hyperbolic Entailment Cones
 (Ganea 2018) Hyperbolic space Poincaré Embedding
 (Nickel 2017) Hyperbolic
 Disk Embedding
28. Hyperbolic Disk Embedding ❖ Disk Embedding in hyperbolic geometry 2 1 3 4 5 6 2 1 3 4 5 6 7 Euclidean Hyperbolic More neighboring disks in hyperbolic space All disks are of
 the same radius
29. Learning Method
30. Loss functions ❖ Margin Loss Eij = d(xi , xj ) − ri + rj h+ (x) = max(x, 0) Positive samples Negative samples ❖ Inclusion of formal disks (x, r) ⊑ (y, s) ⟺ d(x, y) ≤ s − r Eij
31. Loss functions of existing methods This paper Vanishing gradient
 (Vendor 2016, Ganda 2018)
32. Riemannian SGD ❖ SGD on Riemannian manifold. ❖ Gradient → Riemannian gradient ❖ Delta update → Update via exponential map Ordinary (Euclidean) SGD Θt+1 = Θt − η∇L Riemannian SGD Θt+1 = expΘt (−η∇X L) Exponential map Riemannian gradient
33. RSGD on formal disks ❖ Collection of formal disks ❖ Gradient on B(X) ❖ Exponential map on B(X) B(X) = X × ℝ Riemannian manifold ∇B(X) L = (∇X L, ∂L ∂r ) expB(X) (x,r) (u, t) = (expX x (u), r + t) Euclidian space
34. Frequently used geometries ❖ Euclidean geometry ❖ Ordinary SGD ❖ Spherical geometry ❖ Hyperbolic geometry
35. Experiments
36. Experimental Data ❖ WordNet ❖ Large lexical database ❖ Contains “is-a” relation of nouns ❖ Tree-like structure ❖ Reversed WordNet ❖ Reversed relations of WordNet ❖ Non tree-like structure
37. Baseline methods ❖ Poincare Embedding (Nickel 2017) ❖ Order Embedding (Vendrov 2016) ❖ Equivalent to Euclidean Disk Embedding ❖ Hyperbolic Entailment Cones (Ganea 2018) ❖ Equivalent to Spherical Disk Embedding
38. Results Tree-like data: improvement on loss function Reversed tree-like data: maintains reversibility (allows negative radius)
39. Conclusions ❖ Proposed Disk Embeddings. ❖ A general framework for DAG embeddings. ❖ Hyperbolic DE. ❖ Proved existing methods = DE with extra assumptions. ❖ Order Embeddings (Vendrov 2016) = Euclidean DE ❖ Hyperbolic Cones (Ganea 2018) = Spherical DE ❖ Evaluation of effectiveness by experiment.
40. Future work This paper Tree-like Reversed tree-like Complex DAGs
 e.g. citation network
41. Thank you for listening!
42. “ ”