Ozan Sener
February 15, 2016
160

# AAAI-2016 Invited Talk

Extended presentation of my RSS paper rCRF: Recursive Belief Estimation over CRFs in RGB-D Activity Videos

## Ozan Sener

February 15, 2016

## Transcript

1. ### rCRF - Recursive Conditional Random Fields for Human Activity Understanding

Ozan Sener, Ashutosh Saxena

4. ### How to Find MAP Compute features Deﬁne the energy function

Solve the Combinatorial Optimization Activity/Object labels for the past and future Sample Possible Futures [Koppula et al RSS/ICML, Kitani et al ECCV, Jiang et al. RSS/CVPR, Lan et al ECCV, Hu et al RSS/ICRA ]
5. ### Shortcomings Compute features Define the energy function Solve the Combinatorial

Optimization Activity/Object labels for the past and future Sample Possible Futures We also need probabilities in addition to MAP solution (future is unknown)
6. ### Shortcomings Compute features Define the energy function Solve the Combinatorial

Optimization Activity/Object labels for the past and future Sample Possible Futures We also need probabilities in addition to MAP solution (future is unknown) on Space with dimension~ 1o6xT ~ 103600 (#ObjLabels#Objectsx#ActLabels)Time
7. ### Tale of Two Approaches Graphical Models (discriminative) Can handle exponentially

large space Recursive Bayesian (generative) Can compute full belief and reason about uncertinity
8. ### Structured Diversity Although the state dimensionality is high, probability concentrates

on a few modes
9. ### Structured Diversity Modes are likely and structurally diverse yt,i =

arg max y belt ( y ) s.t. ( y, yt,i ) 8j < i
10. ### HMM – Recursive Belief Estimation HMM Derivation [Rabiner] belt( y

) / p( y t = y | x 1, . . . , x t) | {z } ↵t(y) p( x t+1, . . . , x T | y t = y ) | {z } t(y) ↵t( y t) = p( x t| y t) X yt 1 ↵t 1( y t 1)p( y t| y t 1) t( y t) = X yt+1 p( x t+1| y t+1) t+1( y t+1)p( y t+1| y t)
11. ### rCRF: Structured Diversity meets HMM A Belief over rCRF is

a CRF bel ( yt ) / exp 2 4 X v,w2Et ⇣ Eb ( v, w) ˜ Eb ( v, w) ⌘ X v2Vt 0 @Eu ( v) ˜ Eu ( v) + X yt 1 ↵t 1 ( yt 1 ) log p ( yt v |yt 1 v ) 1 X yt+1 t+1 ( yt+1 ) bel ( yt+1 ) log p ( yt+1 v |yt v) 1 A 3 5 Binary Term Unary Term
12. ### rCRF: Structured Diversity w/ HMM Sampling over rCRF is equivalent

to solving CRF Binary Term Unary Term penalize similarity to previous solutions
13. ### rCRF: Algorithm in a Nutshell Compute energy function for frame-wise

CRF Forward-Backward loop for message passing Compute the energy function of rCRF Sample by using Lagrangian relaxation [Batra et al]
14. ### rCRF: Algorithm in a Nutshell All extensions of recursive belief

estimation framework are applicable For example, •  Non-Markovian edges (eg. skip- connections, auto-regressive HMM, Input-Output HMM)
15. ### Efficiency and Accuracy Improvement rCRF is 30x faster than the

state-of-the-art algorithms and runs in real-time Accurate handling of uncertainty also increases accuracy

time
18. ### Computes probabilities for past/present/future with no random sampling rCRF: Algorithm

O ⇣ (TNOLOLA)3 ⌘ ! O ⇣ T (NOLOLA)3 ⌘ Finds all plausible solutions Extends existing learned models and inference algorithms