Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AAAI-2016 Invited Talk

Ozan Sener
February 15, 2016

AAAI-2016 Invited Talk

Extended presentation of my RSS paper rCRF: Recursive Belief Estimation over CRFs in RGB-D Activity Videos

Ozan Sener

February 15, 2016
Tweet

More Decks by Ozan Sener

Other Decks in Research

Transcript

  1. rCRF - Recursive
    Conditional Random Fields
    for Human Activity Understanding
    Ozan Sener, Ashutosh Saxena

    View full-size slide

  2. courtesy of Koppula

    View full-size slide

  3. courtesy of Koppula

    View full-size slide

  4. How to Find MAP
    Compute features
    Define the energy function
    Solve the Combinatorial Optimization
    Activity/Object labels
    for the past and future
    Sample
    Possible
    Futures
    [Koppula et al RSS/ICML, Kitani et
    al ECCV, Jiang et al. RSS/CVPR, Lan
    et al ECCV, Hu et al RSS/ICRA ]

    View full-size slide

  5. Shortcomings
    Compute features
    Define the energy function
    Solve the Combinatorial Optimization
    Activity/Object labels for the
    past and future
    Sample
    Possible
    Futures
    We also need
    probabilities in addition
    to MAP solution
    (future is unknown)

    View full-size slide

  6. Shortcomings
    Compute features
    Define the energy function
    Solve the Combinatorial Optimization
    Activity/Object labels for the
    past and future
    Sample
    Possible
    Futures
    We also need
    probabilities in addition
    to MAP solution
    (future is unknown)
    on
    Space with dimension~ 1o6xT ~ 103600
    (#ObjLabels#Objectsx#ActLabels)Time

    View full-size slide

  7. Tale of Two Approaches
    Graphical Models
    (discriminative)
    Can handle
    exponentially large
    space
    Recursive Bayesian
    (generative)
    Can compute full
    belief and reason
    about uncertinity

    View full-size slide

  8. Structured Diversity
    Although the state dimensionality is high, probability concentrates on a few modes

    View full-size slide

  9. Structured Diversity
    Modes are likely and structurally diverse
    yt,i
    = arg max
    y
    belt
    (
    y
    )
    s.t.
    (
    y, yt,i
    )
    8j < i

    View full-size slide

  10. HMM – Recursive Belief Estimation
    HMM Derivation [Rabiner]
    belt(
    y
    ) / p(
    y
    t =
    y
    |
    x
    1, . . . ,
    x
    t)
    | {z }
    ↵t(y)
    p(
    x
    t+1, . . . ,
    x
    T |
    y
    t =
    y
    )
    | {z }
    t(y)
    ↵t(
    y
    t) = p(
    x
    t|
    y
    t)
    X
    yt 1
    ↵t 1(
    y
    t 1)p(
    y
    t|
    y
    t 1)
    t(
    y
    t) =
    X
    yt+1
    p(
    x
    t+1|
    y
    t+1) t+1(
    y
    t+1)p(
    y
    t+1|
    y
    t)

    View full-size slide

  11. rCRF: Structured Diversity meets HMM
    A Belief over rCRF is a CRF
    bel
    (
    yt
    )
    /
    exp
    2
    4
    X
    v,w2Et

    Eb
    ( v, w) ˜
    Eb
    ( v, w)

    X
    v2Vt
    0
    @Eu
    ( v) ˜
    Eu
    ( v) +
    X
    yt 1
    ↵t 1
    (
    yt 1
    ) log
    p
    (
    yt
    v
    |yt 1
    v )
    1
    X
    yt+1
    t+1
    (
    yt+1
    )
    bel
    (
    yt+1
    ) log
    p
    (
    yt+1
    v
    |yt
    v)
    1
    A
    3
    5
    Binary Term
    Unary Term

    View full-size slide

  12. rCRF: Structured Diversity w/ HMM
    Sampling over rCRF is equivalent to solving CRF
    Binary Term
    Unary Term
    penalize
    similarity
    to previous
    solutions

    View full-size slide

  13. rCRF: Algorithm in a Nutshell
    Compute energy function for frame-wise
    CRF
    Forward-Backward loop for message
    passing
    Compute the energy function of rCRF
    Sample by using Lagrangian
    relaxation [Batra et al]

    View full-size slide

  14. rCRF: Algorithm in a Nutshell
    All extensions of recursive belief
    estimation framework are applicable
    For example,
    •  Non-Markovian edges (eg. skip-
    connections, auto-regressive HMM,
    Input-Output HMM)

    View full-size slide

  15. Efficiency and Accuracy Improvement
    rCRF is 30x faster than the state-of-the-art algorithms and runs in real-time
    Accurate handling of uncertainty also increases accuracy

    View full-size slide

  16. Resulting Belief

    View full-size slide

  17. Efficiency and Accuracy Improvement
    Resulting belief also stays informative trough time

    View full-size slide

  18. Computes probabilities for past/present/future with no
    random sampling
    rCRF: Algorithm
    O

    (TNOLOLA)3

    ! O

    T (NOLOLA)3

    Finds all plausible solutions
    Extends existing learned models and inference algorithms

    View full-size slide