Slide 1
Slide 1 text
scouty Knowledge Graph and Embedding Night
Hyperbolic Disk
Embeddings for
Directed Acyclic Graphs
2019.2.14
Ryota Suzuki
scouty Inc.
Slide 2
Slide 2 text
ࣗݾհ
❖ Ryota Suzuki (͵͵͖)
Twitter: nunuki_ GitHub: nunukim
/&$
म࢜՝ఔ
ੑཧ
ػցֶश Τ͵δχΞ
2018.11
தԝݚڀॴ
৴߸ॲཧ / ػցֶश
Slide 3
Slide 3 text
Hyperbolic Disk Embedding for Directed Acyclic Graphs
https://arxiv.org/pdf/1902.04335.pdf
Slide 4
Slide 4 text
Contents
❖ Introduction
❖ Disk Embedding Models
❖ Learning Method
❖ Experiments
Slide 5
Slide 5 text
Introduction
Slide 6
Slide 6 text
Embedding methods
❖ Embedding symbolic entity
in continuous space (e.g. Euclidean space)
❖ Used as a first layer of NN
❖ Obtains relation of entities
as geometrical structure
❖ This paper: Embedding directed acyclic graphs (DAGs)
vwoman
− vman
+ vking
= vqueen
Slide 7
Slide 7 text
Directed Acyclic Graphs (DAGs)
❖ Directed graphs without cycles.
❖ Directed reachability between nodes forms a partial ordering.
B
A
Root nodes Leaf nodes
Intermediate nodes
Slide 8
Slide 8 text
Partially ordered set (poset)
❖ Transitive, reflexive, anti-symmetric binary relation.
❖ Transitivity: x ⪯ y ∧ y ⪯ z ⇒ x ⪯ z
Lower cone
Upper cone
C⪯
(x) = {y|y ⪯ x}
C⪰
(x) = {y|y ⪰ x}
y ⪯ x
C⪯
(x) ⊆ C⪯
(y)
⇔
Partial ordering
= Inclusive relation of lower cones
x
Proposition 1
x
y
Slide 9
Slide 9 text
Embedding DAGs in poset
❖ Embedding graph structure so that transitive relation
(reachability) is preserved.
Ancestors = Upper cone
Descendants = Lower cone
Slide 10
Slide 10 text
Existing Methods
Order Embeddings
(Vendrov; ICLR 2016)
Hyperbolic Entailment Cones
(Ganea; ICML 2018)
Lower cones
Slide 11
Slide 11 text
Hierarchies and complex DAGs
Complex DAG
Tree-like hierarchy
Exponentially increasing descendants,
few ancestors
Exponentially increasing
descendants AND ancestors
• Organization chart
• Semantic hierarchy of words
• Citation network
• Family genealogy
• Dependency network of libraries
• Manufacturing system …
• Translational
symmetry
• Reversibility
Slide 12
Slide 12 text
Disadvantages of existing methods
(Vendrov; ICLR 2016) (Ganea; ICML 2018)
Assuming the special “origin” (≒ the root of tree).
⇒ Implicitly assuming tree-like structures.
The “origin"
Slide 13
Slide 13 text
Contribution of the paper
❖ Proposed Disk Embeddings.
❖ A general framework for DAG embeddings.
❖ Hyperbolic DE.
❖ Proved existing methods = DE with extra assumptions.
❖ Order Embeddings (Vendrov 2016) = Euclidean DE
❖ Hyperbolic Cones (Ganea 2018) = Spherical DE
❖ Evaluation of effectiveness by experiment.
Slide 14
Slide 14 text
Disk Embeddings
Slide 15
Slide 15 text
Quasi-metric spaces
❖ Symmetric (metric)
Quasi-metrics on
∥x∥1
= 1 ∥x∥∞
= 1
∥x∥2
= 1
∥x∥W
= 1
w1
w2
w3
L1-distance Euclidean distance Uniform distance
Polyhedral quasi-metric
ℝn
∥x∥W
:= max
i
{w⊤
i
x}
Proposition 2
❖ Asymmetric
Unit disk
Slide 16
Slide 16 text
Formal disks
❖ (X,d): quasi-metric space (X: set, d: quasi-metric on X)
x
r
x
r
Closed disk
D(x, r)
❖ Inclusive Relation of closed (open) disks
D(x, r) ⊆ D(y, s) ⟺ D∘(x, r) ⊆ D∘(y, s) ⟺ d(x, y) ≤ s − r
y
s
r
x y
s
r
x
(x, r) ⊑ (y, s)
⇔
Def
Formal disk: abstraction of open and closed disks
Open disk
D∘(x, r)
Partial ordering
∈
B+(X) = X × ℝ+
Slide 17
Slide 17 text
Generalized Formal Disks
❖ Partial ordering of formal disks
❖ Radii need not to be positive
⇒ Generalized formal disks
❖ Transitive symmetry
❖ Reversibility
(y, s) ⊑ (x, r) ⇔ d(x, y) ≤ r − s
Fig. 1 Lower cones of formal disks
Radius
Center
Negative radius
Slide 18
Slide 18 text
Disk Embedding Models
Slide 19
Slide 19 text
Disk Embeddings
❖ Given quasi-metric space (X,d)
❖ Embedding DAGs:
❖ Nodes: formal disk
❖ Edges: Partial ordering of formal disks
(xi
, ri
)
(xi
, ri
) ⊑ (xj
, rj
)
United
Kingdom
Ireland
(island)
Ireland
(state)
Nothern
Ireland
England Scotland
Wales
Fig. 2
2D Euclidean DE (r > 0)
= 2D Euler diagrams of circles
Slide 20
Slide 20 text
= Inclusion of projected disks
= Disk Embedding
Equivalence of Existing Methods
x
y
DAG Embedding
= Partial ordering
= Inclusion of lower cones
Motivation
Slide 21
Slide 21 text
Order Embeddings (Vendrov 2016)
(a) (b)
Reversed product order on
n
⋀
i=1
xi
≥ yi dW
(u, v) ≤ r − s
Formal disks on hyperplane Hn−1
(u, r) =
(
Px, a −
1
n
n
∑
i=1
xi)
ϕord
Order Embeddings Euclidean Disk Embedding
ℝn
Polyhedral quasi-metric
Slide 22
Slide 22 text
Hyperbolic Cones (Ganea 2018)
Reversed product order on
ψ(x) ≥ Ξ(x, y) d
(u, v) ≤ r − s
Formal disks on (n-1)-sphere Sn−1
(u, r) =
x
∥x∥
, arcsin
(
1 + ∥x∥2
2∥x∥
sin θ0)
ϕhyp
Hyperbolic Entailment Cones Spherical Disk Embedding
ℝn
(a) (b)
Slide 23
Slide 23 text
Vector and Disk Embeddings
Base Space Vector Embedding Disk Embedding
Euclidean space
Word2Vec etc.
(Minkolov 2013)
Order Embeddings
(Vendrov 2016)
Sphere Several approaches
Hyperbolic
Entailment Cones
(Ganea 2018)
Slide 24
Slide 24 text
Vector and Disk Embeddings
Base Space Vector Embedding Disk Embedding
Euclidean space
Word2Vec etc.
(Minkolov 2013)
Order Embeddings
(Vendrov 2016)
Sphere Several approaches
Hyperbolic
Entailment Cones
(Ganea 2018)
Hyperbolic space
Poincaré Embedding
(Nickel 2017)
Slide 25
Slide 25 text
Hyperbolic geometry
❖ Non-Euclidean geometry
❖ Constant negative curvature
❖ Interior angles of triangle < 180°
❖ Several equivalent models
❖ Poincaré ball model
❖ Hyperboloid (Lorentz) model
❖ Can embed any weighted tree
w/o losing geometrical properties
Poincaré disk model
http://nunuki.hatenablog.com/entry/2018/12/15/055136
Slide 26
Slide 26 text
Poincaré Embedding
❖ Embedding entities into
hyperbolic space such that
distances are preserved.
❖ Only few (~5) dimensional
PE outperforms 200D
Euclidean methods
Slide 27
Slide 27 text
Hyperbolic Disk Embedding
Base Space Vector Embedding Disk Embedding
Euclidean space
Word2Vec etc.
(Minkolov 2013)
Order Embeddings
(Vendrov 2016)
Sphere Several approaches
Hyperbolic
Entailment Cones
(Ganea 2018)
Hyperbolic space
Poincaré Embedding
(Nickel 2017)
Hyperbolic
Disk Embedding
Slide 28
Slide 28 text
Hyperbolic Disk Embedding
❖ Disk Embedding in hyperbolic geometry
2
1
3
4
5
6
2
1
3
4
5
6
7
Euclidean Hyperbolic
More neighboring disks in hyperbolic space
All disks are of
the same radius
Slide 29
Slide 29 text
Learning Method
Slide 30
Slide 30 text
Loss functions
❖ Margin Loss
Eij
= d(xi
, xj
) − ri
+ rj
h+
(x) = max(x, 0)
Positive samples Negative samples
❖ Inclusion of formal disks
(x, r) ⊑ (y, s) ⟺ d(x, y) ≤ s − r
Eij
Slide 31
Slide 31 text
Loss functions of existing methods
This paper
Vanishing gradient
(Vendor 2016, Ganda 2018)
Slide 32
Slide 32 text
Riemannian SGD
❖ SGD on Riemannian manifold.
❖ Gradient → Riemannian gradient
❖ Delta update → Update via exponential map
Ordinary (Euclidean) SGD
Θt+1
= Θt
− η∇L
Riemannian SGD
Θt+1
= expΘt
(−η∇X
L)
Exponential map Riemannian gradient
Slide 33
Slide 33 text
RSGD on formal disks
❖ Collection of formal disks
❖ Gradient on B(X)
❖ Exponential map on B(X)
B(X) = X × ℝ
Riemannian manifold
∇B(X)
L = (∇X
L,
∂L
∂r )
expB(X)
(x,r)
(u, t) = (expX
x
(u), r + t)
Euclidian space
Slide 34
Slide 34 text
Frequently used geometries
❖ Euclidean geometry
❖ Ordinary SGD
❖ Spherical geometry
❖ Hyperbolic geometry
Slide 35
Slide 35 text
Experiments
Slide 36
Slide 36 text
Experimental Data
❖ WordNet
❖ Large lexical database
❖ Contains “is-a” relation of nouns
❖ Tree-like structure
❖ Reversed WordNet
❖ Reversed relations of WordNet
❖ Non tree-like structure
Slide 37
Slide 37 text
Baseline methods
❖ Poincare Embedding (Nickel 2017)
❖ Order Embedding (Vendrov 2016)
❖ Equivalent to Euclidean Disk Embedding
❖ Hyperbolic Entailment Cones (Ganea 2018)
❖ Equivalent to Spherical Disk Embedding
Slide 38
Slide 38 text
Results
Tree-like data: improvement on loss function
Reversed tree-like data: maintains reversibility (allows negative radius)
Slide 39
Slide 39 text
Conclusions
❖ Proposed Disk Embeddings.
❖ A general framework for DAG embeddings.
❖ Hyperbolic DE.
❖ Proved existing methods = DE with extra assumptions.
❖ Order Embeddings (Vendrov 2016) = Euclidean DE
❖ Hyperbolic Cones (Ganea 2018) = Spherical DE
❖ Evaluation of effectiveness by experiment.
Slide 40
Slide 40 text
Future work
This paper
Tree-like Reversed tree-like Complex DAGs
e.g. citation network
Slide 41
Slide 41 text
Thank you for listening!
Slide 42
Slide 42 text
“
”