Energy Minimization for Multiple Object Tracking

A talk held at the RMRC Workshop in Sydney

Transcript

1. | Anton Milan | Energy Minimization for Multi-Object Tracking | 1 Energy Minimization for Multiple Object Tracking December 2, 2013 Anton Milan (Andriyenko) Supervisors: Stefan Roth & Konrad Schindler
2. | Anton Milan | Energy Minimization for Multi-Object Tracking | 2 The Task Input:
3. | Anton Milan | Energy Minimization for Multi-Object Tracking | 3 The Task Desired output: (manually annotated data)
4. | Anton Milan | Energy Minimization for Multi-Object Tracking | 4 The Task Results: (Discrete-continuous energy minimization [Milan et al. 2013])
5. | Anton Milan | Energy Minimization for Multi-Object Tracking | 5 Road Safety ! Applications ! Surveillance Life Sciences Sports / Entertainment
6. | Anton Milan | Energy Minimization for Multi-Object Tracking | 6 Tracking-by-detection
7. | Anton Milan | Energy Minimization for Multi-Object Tracking | 7 Online vs. Offline Approaches online eg. Kalman filter, particle filter offline non-recursive (deferred logic) Most current approaches
8. | Anton Milan | Energy Minimization for Multi-Object Tracking | 8 Challenges: Data poor data clutter occlusion visual similarity
9. | Anton Milan | Energy Minimization for Multi-Object Tracking | 9 Challenges: Optimization ? ? Brute force approach is intractable! Huge state space Continuous trajectory estimation is difficult. Target 1, …, N? Target N+1? False Detection? time space ? ? ? ? ? ? ? ? ? ? ? ? ? ? number of targets physical constraints
10. | Anton Milan | Energy Minimization for Multi-Object Tracking | 10 Previous Approaches • Focused mainly on data association ➢ discrete state space ➢ constrained to detection responses [Jiang et al. '07] [Huang et al. '08] [Zhang et al. '08] [Pirsiavash et al. '11] [Brendel et al. '11] [Benfold & Reid '11] [Yang & Nevatia '12] occlusion
11. | Anton Milan | Energy Minimization for Multi-Object Tracking | 11 Energy-based Multi-Target Tracking • Goal: Design an energy function that ➢ accurately represents the problem at hand, • complete and appropriate state representation ➢ captures important aspects / dependencies, • dynamics, exclusion, occlusion,... ➢ can be optimized efficiently. • global or “strong” local minima
12. | Anton Milan | Energy Minimization for Multi-Object Tracking | 12 discrete continuous discrete-continuous Energy Domain
13. | Anton Milan | Energy Minimization for Multi-Object Tracking | 13 Discrete Optimization Network Flow Graph-based Approaches [Zhang et al. '08] [Pirsiavash et al. '11] [Henriques et al. '11] [Butt & Collins '13] ... Integer Linear Programming (ILP) [Morefield '77] [Jiang et al. '07] [Berclaz et al. '09] [Andriyenko et al. '10] [Berclaz et al. '11] ... [Wu et al. '11] [Brendel et al. '11] [Zamir et al. '12] [Yang & Nevatia '12] ...
14. | Anton Milan | Energy Minimization for Multi-Object Tracking | 14 discrete continuous discrete-continuous Energy Domain
15. | Anton Milan | Energy Minimization for Multi-Object Tracking | 15 Continuous Energy Minimization state X = all targets in all frames E( , , ) [Andriyenko & Schindler CVPR '11, Andriyenko et al. ICCV-WS '11, Milan et al., PAMI '14] • No restrictions on the energy ➢ dynamics, exclusion, persistence... • Entirely in continuous space • Solve to local optimality with greedy jump moves
16. | Anton Milan | Energy Minimization for Multi-Object Tracking | 16 Continuous Energy Function E =E obs +aE dyn +bE exc +cE per +dE reg data physically-based priors regularizer dynamics exclusion persistence parsimony high energy low energy ∑ i≠ j ∣ ∣X i −X j ∣ ∣−2 −∑ g ∣ ∣X i −D g ∣ ∣−2 ∑ i ∣ ∣v i t −v i t 1∣ ∣2 N+∑ i 1/length i ∑ i 1exp1−b X i −1
17. | Anton Milan | Energy Minimization for Multi-Object Tracking | 17 Non-convex Optimization • conjugate gradient descent for local optimization • several runs from different initializations • discontinuous jumps to determine dimensionality shrink add split E(X) X Jump moves Conjugate gradient descent grow remove merge
18. | Anton Milan | Energy Minimization for Multi-Object Tracking | 18 Continuous Optimization (with discontinuous jumps)
19. | Anton Milan | Energy Minimization for Multi-Object Tracking | 19 Continuous Optimization Video Results
20. | Anton Milan | Energy Minimization for Multi-Object Tracking | 20 Data Association ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Target 1, …, N? Target N+1? False Detection? time space
21. | Anton Milan | Energy Minimization for Multi-Object Tracking | 21 discrete continuous discrete-continuous Energy Domain
22. | Anton Milan | Energy Minimization for Multi-Object Tracking | 22 Discrete-Continuous Optimization E( , , , , , ) [Andriyenko et al. CVPR '12, Milan et al. CVPR '13] • Unified energy for both data association and trajectory estimation • Powerful discrete optimization • Natural continuous space
23. | Anton Milan | Energy Minimization for Multi-Object Tracking | 23 Discrete-Continuous Energy E(f, ) = Unaries + Pairwise + Label cost data smoothness exclusion dynamics, occlusion persistence, regularizer collisions time ∑ g ∣ ∣X i −D g ∣ ∣ ∑ E s δ[a−b] ∑ E x (1−δ[a−b])
24. | Anton Milan | Energy Minimization for Multi-Object Tracking | 24 Discrete-Continuous Optimization α Alternating Optimization Least Squares / L-BFGS E(f) E( ) (modified) - expansion
25. | Anton Milan | Energy Minimization for Multi-Object Tracking | 25 Discrete-Continuous Optimization
26. | Anton Milan | Energy Minimization for Multi-Object Tracking | 26 • Quadratic Boolean Programming (QBP) • Lagrangian Dual Decomposition • Training Detector with Tracker-in-the-Loop ➢ Joint people detector ➢ Mining occlusion patterns Posters 2A (P2A-13) Wednesday Morning [Leibe et al. CVPR '07] [Wu et al. CVPR '12] [Tang et al. ICCV '13] Detector MOTA HOG 19.1 % DPM 21.8 % Joint-Design 23.0 % Joint-Learn 1st iter. 23.4 % Joint-Learn 2nd iter. 26.8 % PETS-S1L2 Combining Detection and Tracking
27. | Anton Milan | Energy Minimization for Multi-Object Tracking | 27 Summary • Continuous Energy Minimization ✔ Accurate representation and modeling ✔ State-of-the-art results despite local optima ✗ No explicit data association • Discrete-Continuous Energy Minimization ✔ Unified energy for data association and trajectory estimation ✔ Powerful discrete optimization techniques can be applied ✔ Accurate and complete state representation
28. | Anton Milan | Energy Minimization for Multi-Object Tracking | 28 Further Directions • Far from solved in challenging / crowded conditions • Heavy / long-term occlusions • Accurate cue extraction • Proper benchmarking, cf. [Milan et al., CVPR-WS '13]
29. | Anton Milan | Energy Minimization for Multi-Object Tracking | 29 Thank you! Energy Minimization for Multiple Object Tracking