AAAI-2016 Invited Talk

Extended presentation of my RSS paper rCRF: Recursive Belief Estimation over CRFs in RGB-D Activity Videos

Transcript

1. rCRF - Recursive Conditional Random Fields for Human Activity Understanding Ozan Sener, Ashutosh Saxena
2. courtesy of Koppula
3. courtesy of Koppula
5. How to Find MAP Compute features Define the energy function Solve the Combinatorial Optimization Activity/Object labels for the past and future Sample Possible Futures [Koppula et al RSS/ICML, Kitani et al ECCV, Jiang et al. RSS/CVPR, Lan et al ECCV, Hu et al RSS/ICRA ]
7. Shortcomings Compute features Define the energy function Solve the Combinatorial Optimization Activity/Object labels for the past and future Sample Possible Futures We also need probabilities in addition to MAP solution (future is unknown)
8. Shortcomings Compute features Define the energy function Solve the Combinatorial Optimization Activity/Object labels for the past and future Sample Possible Futures We also need probabilities in addition to MAP solution (future is unknown) on Space with dimension~ 1o6xT ~ 103600 (#ObjLabels#Objectsx#ActLabels)Time
9. Tale of Two Approaches Graphical Models (discriminative) Can handle exponentially large space Recursive Bayesian (generative) Can compute full belief and reason about uncertinity
10. Structured Diversity Although the state dimensionality is high, probability concentrates on a few modes
11. Structured Diversity Modes are likely and structurally diverse yt,i = arg max y belt ( y ) s.t. ( y, yt,i ) 8j < i
12. HMM – Recursive Belief Estimation HMM Derivation [Rabiner] belt( y ) / p( y t = y | x 1, . . . , x t) | {z } ↵t(y) p( x t+1, . . . , x T | y t = y ) | {z } t(y) ↵t( y t) = p( x t| y t) X yt 1 ↵t 1( y t 1)p( y t| y t 1) t( y t) = X yt+1 p( x t+1| y t+1) t+1( y t+1)p( y t+1| y t)
13. rCRF: Structured Diversity meets HMM A Belief over rCRF is a CRF bel ( yt ) / exp 2 4 X v,w2Et ⇣ Eb ( v, w) ˜ Eb ( v, w) ⌘ X v2Vt 0 @Eu ( v) ˜ Eu ( v) + X yt 1 ↵t 1 ( yt 1 ) log p ( yt v |yt 1 v ) 1 X yt+1 t+1 ( yt+1 ) bel ( yt+1 ) log p ( yt+1 v |yt v) 1 A 3 5 Binary Term Unary Term
14. rCRF: Structured Diversity w/ HMM Sampling over rCRF is equivalent to solving CRF Binary Term Unary Term penalize similarity to previous solutions
15. rCRF: Algorithm in a Nutshell Compute energy function for frame-wise CRF Forward-Backward loop for message passing Compute the energy function of rCRF Sample by using Lagrangian relaxation [Batra et al]
16. rCRF: Algorithm in a Nutshell All extensions of recursive belief estimation framework are applicable For example, •  Non-Markovian edges (eg. skip- connections, auto-regressive HMM, Input-Output HMM)
17. Efficiency and Accuracy Improvement rCRF is 30x faster than the state-of-the-art algorithms and runs in real-time Accurate handling of uncertainty also increases accuracy
18. Resulting Belief
19. Efficiency and Accuracy Improvement Resulting belief also stays informative trough time
20. Computes probabilities for past/present/future with no random sampling rCRF: Algorithm O ⇣ (TNOLOLA)3 ⌘ ! O ⇣ T (NOLOLA)3 ⌘ Finds all plausible solutions Extends existing learned models and inference algorithms
21. Thank you