Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ICML2019読み会: Hyperbolic Disk Embeddings for Directed Acyclic Graphs

ICML2019読み会: Hyperbolic Disk Embeddings for Directed Acyclic Graphs

ICLR/ICML2019 読み会 (https://connpass.com/event/138672/) での発表資料です。

Ryota Suzuki / ぬぬき

July 21, 2019
Tweet

More Decks by Ryota Suzuki / ぬぬき

Other Decks in Technology

Transcript

  1. Hyperbolic Disk Embeddings

    for Directed Acyclic Graphs
    ICML2019 ಡΈձ
    Ryota Suzuki
    ©2019 LAPRAS Inc.

    View full-size slide

  2. 2
    Researcher / Algorithm Engineer
    ɾNEC → LAPRAS (2018.10)

    ɾػցֶशɾ৴߸ॲཧɺ෺ཧ(෺ੑཧ࿦)
    ɾtwitter: nunuki_
    ɾLAPRAS: https://lapras.com/public/1HETMLB
    ɾHolacracy roles:
    ɹɹAlgorithm Prototyping / ࿦จࣥච / ੒ՌמऔΓϚϯ / σʔλ෼ੳ /
    ɹɹWebApp ࣮૷ / ϓϩμΫτಛڐઓུ / ϓϩμΫτ๏ྩ९क / etc.
    Ryota Suzuki (ླ໦ ྄ଠ)

    View full-size slide

  3. 3
    ※ Ҏ߱ɺ΄΅ӳޠͰ͢

    View full-size slide

  4. *OUSPEVDUJPO

    View full-size slide

  5. 5
    Embedding Methods
    Vector representation of discrete entities
    • Natural languages, graphs, …
    • Data structure is encoded as geometrical properties
    • Used as first layer of neural networks
    [Mikolov, NIPS 2013]

    View full-size slide

  6. 6
    Embedding Methods
    ʙ
    Word2Vec

    [Mikolov, NIPS2013]
    Various Applications
    Poincaré Embedding

    [Nickel, NIPS 2017]
    Order Embedding

    [Vendrov, ICLR2016]
    Probabilistic

    Order Embedding

    [Lai, EACL2017]
    Hyperbolic

    Entailment Cones

    [Ganea, ICML2018]
    Box Embedding

    [Vilnis ACL 2018]
    Lorentz Embedding

    [Nickel, ICML 2018]
    Smoothing

    Box Embedding

    [Li, ICLR 2019]
    Hyperbolic

    Disk Embeddings

    [Suzuki, ICML 2019]
    &NCFEEJOH
    )ZQFSCPMJD
    0SEFS
    1SPCBCJMJTUJD

    View full-size slide

  7. 7
    Embedding Directed Acyclic Graphs (DAGs)
    • Semantics, citation networks, genealogical networks
    • Asymmetric / transitive relation of nodes

    → Partially ordered set (poset)
    DAG Embedding
    • Embedding nodes into continuous poset

    so that transitive relation is preserved.
    DAG Embedding Models

    View full-size slide

  8. 8
    Existing Methods
    Order Embeddings

    [Vendrov, ICLR 2016]
    Hyperbolic Entailment Cones

    [Ganea, ICML 2018]

    View full-size slide

  9. 9
    2D Euclidean Disk Embedding = Eular Diagram
    • Disk Embedding: General framework for embedding DAGs
    Our Contributions
    • Novel Hyperbolic Disk Embedding
    • Experiments
    • Theorems: Existing methods are special cases of DE

    (+ extra restrictions)

    View full-size slide

  10. %JTL&NCFEEJOHT

    View full-size slide

  11. 11
    Disk Embedding
    Disk Embedding: DAG relation → Inclusive relation of disks
    Disk = Center + Radius
    X y
    x
    dX
    (x, y) + rx
    − ry
    ≤ 0
    Dx
    = (x, rx
    )

    View full-size slide

  12. 12
    Embedding DAGs
    Ancestors = Upper cone
    Descendants = Lower cone
    Transitive relation of DAG induces “cones” in the embedding space.

    View full-size slide

  13. 13
    Key Idea
    y
    x Relation of DAG nodes

    View full-size slide

  14. 13
    Key Idea
    = Inclusion of lower cones
    y
    x Relation of DAG nodes

    View full-size slide

  15. 13
    Key Idea
    = Inclusion of projected disks
    X = Inclusion of lower cones
    y
    x Relation of DAG nodes

    View full-size slide

  16. 13
    Key Idea
    = Inclusion of projected disks
    X = Inclusion of lower cones
    y
    x Relation of DAG nodes
    X y
    x
    dX
    (x, y) + rx
    − ry
    ≤ 0
    = Disk Embedding

    View full-size slide

  17. %JTL&NCFEEJOHBT(FOFSBM'SBNFXPSL

    View full-size slide

  18. 15
    Disk Embedding as General Framework
    Disk Embedding is easily extensible to general metric spaces
    Further generalizations
    • Metric spaces → Quasi-metric spaces
    • (Closed) Disks → Formal disks
    Existing methods can be understood as special cases of DEs
    X y
    x

    View full-size slide

  19. 16
    Quasi-metric Spaces
    ∥x∥W
    = 1
    w1
    w2
    w3
    ∥x∥W
    := max
    i
    {w⊤
    i
    x}
    Proposition 2
    ∥x∥1
    = 1 ∥x∥∞
    = 1
    ∥x∥2
    = 1
    L1-distance Euclidean distance Uniform distance
    Symmetric quasi-metric (metric)
    Asymmetric quasi-metric
    Unit disk
    Polyhedral quasimetric

    View full-size slide

  20. 17
    Formal Disks
    Abstraction of closed / open disks
    y
    x
    y
    x
    Closed disk Open disk
    Dx
    ⊆ Dy
    D∘
    x
    ⊆ D∘
    y
    (x, r)
    (x, rx
    ) ⊑ (y, ry
    )
    d(x, y) + rx
    − ry
    ≤ 0
    Formal Disk
    Radii can be negative → Reversibility of Disk Embeddings

    View full-size slide

  21. 18
    Existing Methods
    Order Embeddings

    [Vendrov, ICLR 2016]
    Hyperbolic Entailment Cones

    [Ganea, ICML 2018]
    Lower Cones

    View full-size slide

  22. 19
    Equivalence of models
    Order Embeddings
    [Vendrov 2016]
    (a)
    ϕ
    Euclidean Disk Embedding
    (w/ polyhedral quasimetric)
    xk
    ≥ yk
    for k = 1,⋯, n
    (x′, r) =
    (
    Px, a −
    1
    n
    n

    k=1
    xk)
    dW
    (x′, y′) + rx
    − ry
    ≤ 0

    View full-size slide

  23. 20
    Equivalence of models
    (a) (b)
    ϕ
    Spherical Disk Embedding
    d
    (x′, y′) + rx
    − ry
    ≤ 0
    Hyperbolic Entailment Cones

    [Ganea, ICML 2018]
    (x′, r) =
    x
    ∥x∥
    , arcsin
    (
    1 + ∥x∥2
    2∥x∥
    sin θ0)
    ψ(x) ≥ Ξ(x, y)

    View full-size slide

  24. 21
    Limitations of Existing Methods
    Hierarchy Non-Hierarchy
    Tree-like Complex DAG
    Root No root

    View full-size slide

  25. 22
    Limitations of Existing Methods
    Order Embeddings

    [Vendrov, ICLR 2016]
    Hyperbolic Entailment Cones

    [Ganea, ICML 2018]
    “Origin”
    Existing methods implicitly assumes the existence of the origin

    ≒ Assuming rooted hierarchies

    View full-size slide

  26. 23
    Advantage of Disk Embedding
    Extends existing methods
    • Negative radius: reversibility & translational symmetry
    • Avoid gradient vanishing on loss functions
    Applicable for various (quasi-)metric spaces.
    Positive sample
    Negative sample
    Case of
    Negative Radius Loss Functions
    Ours
    Existing

    methods

    View full-size slide

  27. )ZQFSCPMJD%JTL&NCFEEJOH

    View full-size slide

  28. 25
    Base Space Vector Embedding Disk Embedding
    Euclidean space
    Word2Vec etc.

    (Minkolov 2013)
    Order Embeddings

    (Vendrov 2016)
    Sphere Several approaches
    Hyperbolic Entailment
    Cones

    (Ganea 2018)
    Hyperbolic space
    Poincaré Embedding

    (Nickel 2017)
    Hyperbolic

    Disk Embedding

    View full-size slide

  29. 26
    Hyperbolic Disk Embedding
    2
    1
    3
    4
    5
    6
    2
    1
    3
    4
    5
    6
    7
    More neighboring disks in hyperbolic space
    All disks are of the same radius
    Euclidean Hyperbolic

    View full-size slide

  30. 28
    Experiments
    Hierarchy Non-Hierarchy
    Tree-like Reversed Tree-like Complex DAG
    This Paper

    View full-size slide

  31. 29
    Experimental Results
    Hierarchy
    Non-

    Hierarchy
    Our Disk Embedding models achieved significant
    improvement especially in non-hierarchical data

    View full-size slide

  32. $PODMVTJPO'VUVSF8PSLT

    View full-size slide

  33. 31
    • Disk Embedding: General framework for embedding DAGs
    Conclusion
    • Novel Hyperbolic Disk Embedding
    • Experiments
    • Theorems: Existing methods are special cases of DE

    (+ extra restrictions)

    View full-size slide

  34. 32
    Future Works: Experiments
    Hierarchy Non-Hierarchy
    Tree-like Reversed Tree-like Complex DAG
    Future Work

    View full-size slide

  35. 33
    Future Works: Theories
    ʙ
    Word2Vec

    [Mikolov, NIPS2013]
    Various Applications
    Poincaré Embedding

    [Nickel, NIPS 2017]
    Order Embedding

    [Vendrov, ICLR2016]
    Probabilistic

    Order Embedding

    [Lai, EACL2017]
    Hyperbolic

    Entailment Cones

    [Ganea, ICML2018]
    Box Embedding

    [Vilnis ACL 2018]
    Lorentz Embedding

    [Nickel, ICML 2018]
    Smoothing

    Box Embedding

    [Li, ICLR 2019]
    Hyperbolic

    Disk Embeddings

    [Suzuki, ICML 2019]
    &NCFEEJOH
    )ZQFSCPMJD
    0SEFS
    1SPCBCJMJTUJD
    .PSFHFOFSBMJ[BUJPOT

    GPSQSPCBCJMJTUJDNPEFMT

    View full-size slide

  36. 5IBOLZPVGPSMJTUFOJOH

    View full-size slide