Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AAAI-2016 Invited Talk

Ozan Sener
February 15, 2016

AAAI-2016 Invited Talk

Extended presentation of my RSS paper rCRF: Recursive Belief Estimation over CRFs in RGB-D Activity Videos

Ozan Sener

February 15, 2016
Tweet

More Decks by Ozan Sener

Other Decks in Research

Transcript

  1. rCRF - Recursive
    Conditional Random Fields
    for Human Activity Understanding
    Ozan Sener, Ashutosh Saxena

    View Slide

  2. courtesy of Koppula

    View Slide

  3. courtesy of Koppula

    View Slide

  4. View Slide

  5. How to Find MAP
    Compute features
    Define the energy function
    Solve the Combinatorial Optimization
    Activity/Object labels
    for the past and future
    Sample
    Possible
    Futures
    [Koppula et al RSS/ICML, Kitani et
    al ECCV, Jiang et al. RSS/CVPR, Lan
    et al ECCV, Hu et al RSS/ICRA ]

    View Slide

  6. View Slide

  7. Shortcomings
    Compute features
    Define the energy function
    Solve the Combinatorial Optimization
    Activity/Object labels for the
    past and future
    Sample
    Possible
    Futures
    We also need
    probabilities in addition
    to MAP solution
    (future is unknown)

    View Slide

  8. Shortcomings
    Compute features
    Define the energy function
    Solve the Combinatorial Optimization
    Activity/Object labels for the
    past and future
    Sample
    Possible
    Futures
    We also need
    probabilities in addition
    to MAP solution
    (future is unknown)
    on
    Space with dimension~ 1o6xT ~ 103600
    (#ObjLabels#Objectsx#ActLabels)Time

    View Slide

  9. Tale of Two Approaches
    Graphical Models
    (discriminative)
    Can handle
    exponentially large
    space
    Recursive Bayesian
    (generative)
    Can compute full
    belief and reason
    about uncertinity

    View Slide

  10. Structured Diversity
    Although the state dimensionality is high, probability concentrates on a few modes

    View Slide

  11. Structured Diversity
    Modes are likely and structurally diverse
    yt,i
    = arg max
    y
    belt
    (
    y
    )
    s.t.
    (
    y, yt,i
    )
    8j < i

    View Slide

  12. HMM – Recursive Belief Estimation
    HMM Derivation [Rabiner]
    belt(
    y
    ) / p(
    y
    t =
    y
    |
    x
    1, . . . ,
    x
    t)
    | {z }
    ↵t(y)
    p(
    x
    t+1, . . . ,
    x
    T |
    y
    t =
    y
    )
    | {z }
    t(y)
    ↵t(
    y
    t) = p(
    x
    t|
    y
    t)
    X
    yt 1
    ↵t 1(
    y
    t 1)p(
    y
    t|
    y
    t 1)
    t(
    y
    t) =
    X
    yt+1
    p(
    x
    t+1|
    y
    t+1) t+1(
    y
    t+1)p(
    y
    t+1|
    y
    t)

    View Slide

  13. rCRF: Structured Diversity meets HMM
    A Belief over rCRF is a CRF
    bel
    (
    yt
    )
    /
    exp
    2
    4
    X
    v,w2Et

    Eb
    ( v, w) ˜
    Eb
    ( v, w)

    X
    v2Vt
    0
    @Eu
    ( v) ˜
    Eu
    ( v) +
    X
    yt 1
    ↵t 1
    (
    yt 1
    ) log
    p
    (
    yt
    v
    |yt 1
    v )
    1
    X
    yt+1
    t+1
    (
    yt+1
    )
    bel
    (
    yt+1
    ) log
    p
    (
    yt+1
    v
    |yt
    v)
    1
    A
    3
    5
    Binary Term
    Unary Term

    View Slide

  14. rCRF: Structured Diversity w/ HMM
    Sampling over rCRF is equivalent to solving CRF
    Binary Term
    Unary Term
    penalize
    similarity
    to previous
    solutions

    View Slide

  15. rCRF: Algorithm in a Nutshell
    Compute energy function for frame-wise
    CRF
    Forward-Backward loop for message
    passing
    Compute the energy function of rCRF
    Sample by using Lagrangian
    relaxation [Batra et al]

    View Slide

  16. rCRF: Algorithm in a Nutshell
    All extensions of recursive belief
    estimation framework are applicable
    For example,
    •  Non-Markovian edges (eg. skip-
    connections, auto-regressive HMM,
    Input-Output HMM)

    View Slide

  17. Efficiency and Accuracy Improvement
    rCRF is 30x faster than the state-of-the-art algorithms and runs in real-time
    Accurate handling of uncertainty also increases accuracy

    View Slide

  18. Resulting Belief

    View Slide

  19. Efficiency and Accuracy Improvement
    Resulting belief also stays informative trough time

    View Slide

  20. Computes probabilities for past/present/future with no
    random sampling
    rCRF: Algorithm
    O

    (TNOLOLA)3

    ! O

    T (NOLOLA)3

    Finds all plausible solutions
    Extends existing learned models and inference algorithms

    View Slide

  21. Thank you

    View Slide