A-Exam Slides

Transcript

1. Unsupervised Discovery of Structure in Human Videos Ozan Sener Joint work with Ashutosh Saxena, Silvio Savarese, Ashesh Jain and Amir Zamir Committee: Ashutosh Saxena, David Mimno, Emin Gun Sirer
2. We envision robots doing human like activities while working with humans courtesy of Sung et al. courtesy of Koppula et al.
3. understanding humans, their environments, objects and activities It requires
4. What is the next step? What is he doing? How can I perform X activity?
5. Understanding Videos Image Centric Video is an trivial extensions of images Video Centric We need video specific features/models [Kantarov CVP14, Hou et al ECCV14, THUMOS15, Schmid CVPR15]
6. Understanding Videos Image Centric Rich models like CRF Easy to model context Super linear in #of-frames Hard to obtain supervision (~10s activitiy, ~10s objects) Video Centric Scales linearly in #of-frames Requires only frame labels Does not model the context Inefficient (~30sec for ~1sec of vid) Exclusively supervised Hard to scale in #of-videos (~100s videos)
7. What we have? ~30sec to process 1sec of video support ~10 activities learning from ~100 videos covering only indoor/sport environments What we need? real-time any activity learn from all available information any environment ?
8. Discover, understand and share the underlying semantic structure of the videos.
9. Structured understanding of a single video Large-scale understanding of video collections Sharing knowledge to other domains and modalities Outline
10. Structured understanding of a single video Large-scaled understanding of video collections Sharing knowledge to other domains and modalities Outline
11. Revisit the Image Based Approach O2 O1 H Context of humans and objects are successfully modeled as CRFs P ( O1,...,T 1 , O1,...,T 2 , H1,...,T | 1,...,T O1 , 1,...,T O2 , 1,...,T H ) ⇠ exp 0 @ X v2V E ( v) + X v,w2E E ( v , w) 1 A
12. How to Find MAP[Koppula RSS 2013] Compute features Define the energy function Solve the Combinatorial Optimization Activity/Object labels for the past and future Sample Possible Futures
13. Shortcomings[Koppula RSS 2013] Compute features Define the energy function Solve the Combinatorial Optimization Activity/Object labels for the past and future Sample Possible Futures We also need probabilities in addition to MAP solution (future is unknown) Dimension ~ 1o6xT ~ 103600 (#ObjLabels#Objectsx#ActLabels)Time O ⇣ (TNOLOLA)3 ⌘
14. Structured Diversity Although the state dimensionality is high, probability concentrates on a few modes
15. Structured Diversity Modes are likely and structurally diverse yt,i = arg max y belt ( y ) s.t. ( y, yt,i ) 8j < i
16. HMM – Recursive Belief Estimation HMM Derivation [Rabiner] belt( y ) / p( y t = y | x 1, . . . , x t) | {z } ↵t(y) p( x t+1, . . . , x T | y t = y ) | {z } t(y) ↵t( y t) = p( x t| y t) X yt 1 ↵t 1( y t 1)p( y t| y t 1) t( y t) = X yt+1 p( x t+1| y t+1) t+1( y t+1)p( y t+1| y t)
17. rCRF: Structured Diversity meets HMM Proposition: A Belief over rCRF is a CRF bel ( yt ) / exp 2 4 X v,w2Et ⇣ Eb ( v, w) ˜ Eb ( v, w) ⌘ X v2Vt 0 @Eu ( v) ˜ Eu ( v) + X yt 1 ↵t 1 ( yt 1 ) log p ( yt v |yt 1 v ) 1 X yt+1 t+1 ( yt+1 ) bel ( yt+1 ) log p ( yt+1 v |yt v) 1 A 3 5 Binary Term Unary Term
18. rCRF: Algorithm Compute energy function for each frame-wise CRF Forward-Backward loop for message passing Compute the energy function of rCRF Sample by using Lagrangian relaxation [Batra et al]
19. rCRF: Algorithm Compute energy function for each frame-wise CRF Forward-Backward loop for message passing Compute the energy function of rCRF Sample by using Lagrangian relaxation [Batra et al] O ⇣ (TNOLOLA)3 ⌘ ! O ⇣ T (NOLOLA)3 ⌘ Computes probabilities for past/present/future states as a part of the formulation with no random sampling
20. Resulting Belief
21. Efficiency and Accuracy Improvement rCRF is 30x faster than the state-of-the-art algorithms and runs in real-time Accurate handling of uncertainty also increases accuracy
22. Efficiency and Accuracy Improvement Resulting belief also stays informative trough time
23. Structured understanding of a single video Large-scaled understanding of video collection Sharing knowledge to other domains and modalities Outline
24. is Unsupervised Learning Possible Is there an underlying structure in the YouTube “How-To” videos?
25. Is there an underlying structure in the YouTube “How-To” videos? 1st Result
26. Is there an underlying structure in the YouTube “How-To” videos? 2nd Result
27. We discover activities by using a NP-Bayes approach We learn a multi-modal dictionary Summary of the Approach We automatically download and filter large multi-modal activity corpus from YouTube
28. Dictionary Learning (Language) We use the tf-idf metric by considering each video as a document. We choose the K mostfrequent words with max tf−idf Dictionary for category “Hard Boil an Egg” with K=50 sort, place, water, egg, bottom, fresh, pot, crack, cold, cover, time, overcooking, hot, shell, stove, turn, cook, boil, break, pinch, salt, peel, lid, point, haigh, rules, perfectly, hard, smell, fast, soft, chill, ice, bowl, remove, aside, store, set, temperature, coagulates, yolk, drain, swirl, shake, white, roll, handle, surface, flat
29. Dictionary Learning (Visual) Multi Video Edges Proposal Graph for Video 1 Proposal Graph for Video 2 Proposal Graph for Video 3
30. Dictionary Learning (Visual) Multi Video Edges Proposal Graph for Video 1 Proposal Graph for Video 2 Proposal Graph for Video 3 arg max X i2V x (i)T A (i) x (i) x (i)T x (i) + X i2V X j2N (i) x (i)T A (i,j) x (j) x (i)T 11T x (j) where V is set of videos, N ( i ) is neighbour videos of i , and A is similarity matrices This function is quasi convex and can be optimized via SGD as arg max X i2V x (i)T A (i) x (i) x (i)T x (i) + X i2V X j2N (i) x (i)T A (i,j) x (j) x (i)T 11T x (j) where V is set of videos, N ( i ) is neighbour videos of i , and A is similarity matrices r x (i) = 2 A ( i ) x ( i ) 2 x ( i )r(i) x ( i )T x ( i ) + X i2N Ai , j xj x ( j )T1r(i,j) x ( i )T11T x ( j )
31. Learned Dictionaries Semantically Correct
32. Learned Dictionaries Semantically Correct
33. Learned Dictionaries Accuracy vs Semantic Meaning
34. Representing Each Frame
35. Unsupervised Discovery via NP-Bayes k = 1, . . . , 1 i = 1, . . . , N c y2 · · · ⌘i fi y1 · · · kth activity step ⇡i B0 (·) ith video wk z4 y3 z3 y4 z1 z2 ✓k We jointly model activities and videos
36. Unsupervised Discovery via NP-Bayes k = 1, . . . , 1 i = 1, . . . , N c y2 · · · ⌘i fi y1 · · · kth activity step ⇡i B0 (·) ith video wk z4 y3 z3 y4 z1 z2 ✓k Each activity is modeled as likelihood of seeing each dictionary item. e.g. probability of having a word “egg” and having object 5 ✓k
37. Unsupervised Discovery via NP-Bayes k = 1, . . . , 1 i = 1, . . . , N c y2 · · · ⌘i fi y1 · · · kth activity step ⇡i B0 (·) ith video wk z4 y3 z3 y4 z1 z2 ✓k Each videos choose subset of activities via Indian Buffet Process fi
38. Unsupervised Discovery via NP-Bayes k = 1, . . . , 1 i = 1, . . . , N c y2 · · · ⌘i fi y1 · · · kth activity step ⇡i B0 (·) ith video wk z4 y3 z3 y4 z1 z2 ✓k And transition probabilities between activities ⌘i
39. Unsupervised Discovery via NP-Bayes k = 1, . . . , 1 i = 1, . . . , N c y2 · · · ⌘i fi y1 · · · kth activity step ⇡i B0 (·) ith video wk z4 y3 z3 y4 z1 z2 ✓k Given activities/transition probabilities, it is HMM
40. Unsupervised Discovery via NP-Bayes k = 1, . . . , 1 i = 1, . . . , N c y2 · · · ⌘i fi y1 · · · kth activity step ⇡i B0 (·) ith video wk z4 y3 z3 y4 z1 z2 ✓k We learn by Gibbs Sampling
41. Discovered Activities
43. Evaluation Both modalities are complementary and joint modeling is necessary! Multi-video mid-level descriptions are critical for the accuracy
44. Structured understanding of a single video Large-scaled unsupervised understanding of human activities Sharing knowledge to other domains and modalities Outline
45. Graph Perspective of Large-Scaled Activities How to make pancakes
46. Graph Perspective of Large-Scaled Activities How to make pancakes
47. Graph Perspective of Large-Scaled Activities How to make pancakes Egg Heat Pan Flip Beat Flour
48. Graph Perspective of Large-Scaled Activities How to make pancakes Egg Heat Pan Flip Beat Flour
49. Can we go further? RoboBrain Snapshot of the RoboBrain graph 45,000 concepts (nodes) 98,000 relations (edges) Connecting knowledge from Internet sources and manyprojects
50. How to scale the knowledge cv cv cv cv cv cv cv cup water pour liquid bottle kept fridge appearance has_grasp Input is in the form of “feeds” A feed is collection of binary relations Concept Concept Relation Concept Concept Relation …
51. How to scale the knowledge cv cv cv cv cv cv cv cup water pour liquid bottle kept fridge appearance has_grasp Bottle has appearance Table has appearance
52. How to scale the knowledge cv cv cv cv cv cv cv cup water pour liquid bottle kept fridge appearance has_grasp Bottle has appearance Table has appearance cv table
53. How to scale the knowledge cv cv cv cv cv cv cv cup water pour liquid bottle kept fridge appearance has_grasp Cup spatially distributed Table spatially distributed cv table
54. System Architecture
55. How we processed 25k videos in a day
56. Robotics-as-a-Service We shared a reliable interface to multiple universities. Humans to Robots Lab @ Brown University successfully used RoboBrain-as-a-Service
57. What we have? ~30sec to process 1sec of video support ~10 activities learning from ~100 videos covering only indoor sport environments What we need? real-time any activity learn from all available information any environment humans go rCRF with structured div unsupervised learning w/ NP-Bayes Large-scaled learning on YouTube Using multiple domains via RB
58. How can we scale further? Linking more and more modalities and domains with efficient and theoretically sound models
59. Cross-Domain Information Videos with no Structure Images and Words with Structure How to transfer knowledge?
60. Transductive Approach (ongoing/future work) How to handle domain shift? Domain adaptation vs Domain invariance Domain invariant feature Learning followed by Induction Induction followed by transformation
61. Transductive Approach (ongoing/future work) Domain invariant feature Learning followed by Induction Induction followed by transformation It is generally hard Sometimes impossible [Vapnik] What if such feature does not exist
62. Domain Transduction [Ongoing work] It might be possible to solve Domain transfer with no induction We are developing a max-margin framework based on coordinate-ascent of transduction and domain adaptation [ongoing work]
63. Adaptive Transduction [Future Work] Some domains are Intractably large like YouTube etc. Can we solve the problem adaptively
64. Adaptive Transduction – Robot in the Loop Sampled Videos Domain Transduction Adaptive Sampling
65. rCRF: Recursive Belief Estimation over CRFs in RGB-D Activity Videos Ozan Sener, Ashutosh Saxena. In RSS 2015 Unsupervised Semantic Parsing of Video Collections Ozan Sener, Amir Zamir, Silvio Savarese, Ashutosh Saxena. In ICCV 2015 RoboBrain: Large-Scale Knowledge Engine for Robots Ashutosh Saxena, Ashesh Jain, Ozan Sener, Aditya Jami, Dipendra K. Misra, Hema S. Koppula. In ISRR 2015 3D Semantic Parsing of Large-Scale Buildings Iro Armeni, Ozan Sener et al. (In submission to CVPR 2016) Unsupervised Discovery of Spatio-Temporal Semantic Descriptors Under preperation for TPAMI submission
66. Joint Work With Ashutosh Saxena Silvio Savarese Ozan Sener Ashesh Jain Deedy Das Amir R. Zamir Aditya Jami Dipendra K. Misra Jay Hack Hema Koppula
67. Thank You