Ozan Sener
February 15, 2016
130

# AAAI-2016 Invited Talk

Extended presentation of my RSS paper rCRF: Recursive Belief Estimation over CRFs in RGB-D Activity Videos

## Ozan Sener

February 15, 2016

## Transcript

1. rCRF - Recursive
Conditional Random Fields
for Human Activity Understanding
Ozan Sener, Ashutosh Saxena

2. courtesy of Koppula

3. courtesy of Koppula

4. How to Find MAP
Compute features
Deﬁne the energy function
Solve the Combinatorial Optimization
Activity/Object labels
for the past and future
Sample
Possible
Futures
[Koppula et al RSS/ICML, Kitani et
al ECCV, Jiang et al. RSS/CVPR, Lan
et al ECCV, Hu et al RSS/ICRA ]

5. Shortcomings
Compute features
Define the energy function
Solve the Combinatorial Optimization
Activity/Object labels for the
past and future
Sample
Possible
Futures
We also need
to MAP solution
(future is unknown)

6. Shortcomings
Compute features
Define the energy function
Solve the Combinatorial Optimization
Activity/Object labels for the
past and future
Sample
Possible
Futures
We also need
to MAP solution
(future is unknown)
on
Space with dimension~ 1o6xT ~ 103600
(#ObjLabels#Objectsx#ActLabels)Time

7. Tale of Two Approaches
Graphical Models
(discriminative)
Can handle
exponentially large
space
Recursive Bayesian
(generative)
Can compute full
belief and reason

8. Structured Diversity
Although the state dimensionality is high, probability concentrates on a few modes

9. Structured Diversity
Modes are likely and structurally diverse
yt,i
= arg max
y
belt
(
y
)
s.t.
(
y, yt,i
)
8j < i

10. HMM – Recursive Belief Estimation
HMM Derivation [Rabiner]
belt(
y
) / p(
y
t =
y
|
x
1, . . . ,
x
t)
| {z }
↵t(y)
p(
x
t+1, . . . ,
x
T |
y
t =
y
)
| {z }
t(y)
↵t(
y
t) = p(
x
t|
y
t)
X
yt 1
↵t 1(
y
t 1)p(
y
t|
y
t 1)
t(
y
t) =
X
yt+1
p(
x
t+1|
y
t+1) t+1(
y
t+1)p(
y
t+1|
y
t)

11. rCRF: Structured Diversity meets HMM
A Belief over rCRF is a CRF
bel
(
yt
)
/
exp
2
4
X
v,w2Et

Eb
( v, w) ˜
Eb
( v, w)

X
v2Vt
0
@Eu
( v) ˜
Eu
( v) +
X
yt 1
↵t 1
(
yt 1
) log
p
(
yt
v
|yt 1
v )
1
X
yt+1
t+1
(
yt+1
)
bel
(
yt+1
) log
p
(
yt+1
v
|yt
v)
1
A
3
5
Binary Term
Unary Term

12. rCRF: Structured Diversity w/ HMM
Sampling over rCRF is equivalent to solving CRF
Binary Term
Unary Term
penalize
similarity
to previous
solutions

13. rCRF: Algorithm in a Nutshell
Compute energy function for frame-wise
CRF
Forward-Backward loop for message
passing
Compute the energy function of rCRF
Sample by using Lagrangian
relaxation [Batra et al]

14. rCRF: Algorithm in a Nutshell
All extensions of recursive belief
estimation framework are applicable
For example,
•  Non-Markovian edges (eg. skip-
connections, auto-regressive HMM,
Input-Output HMM)

15. Efficiency and Accuracy Improvement
rCRF is 30x faster than the state-of-the-art algorithms and runs in real-time
Accurate handling of uncertainty also increases accuracy

16. Resulting Belief

17. Efficiency and Accuracy Improvement
Resulting belief also stays informative trough time

18. Computes probabilities for past/present/future with no
random sampling
rCRF: Algorithm
O

(TNOLOLA)3

! O

T (NOLOLA)3

Finds all plausible solutions
Extends existing learned models and inference algorithms

19. Thank you