Share
Embed
Energy Minimization for Multiple Object Tracking
A talk held at the RMRC Workshop in Sydney
Anton Milan
December 02, 2013
Transcript
29 slides
-
| Anton Milan | Energy Minimization for Multi-Object Tracking |
1 Energy Minimization for Multiple Object Tracking December 2, 2013 Anton Milan (Andriyenko) Supervisors: Stefan Roth & Konrad Schindler -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
2 The Task Input: -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
3 The Task Desired output: (manually annotated data) -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
4 The Task Results: (Discrete-continuous energy minimization [Milan et al. 2013]) -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
5 Road Safety ! Applications ! Surveillance Life Sciences Sports / Entertainment -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
6 Tracking-by-detection -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
7 Online vs. Offline Approaches online eg. Kalman filter, particle filter offline non-recursive (deferred logic) Most current approaches -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
8 Challenges: Data poor data clutter occlusion visual similarity -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
9 Challenges: Optimization ? ? Brute force approach is intractable! Huge state space Continuous trajectory estimation is difficult. Target 1, …, N? Target N+1? False Detection? time space ? ? ? ? ? ? ? ? ? ? ? ? ? ? number of targets physical constraints -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
10 Previous Approaches • Focused mainly on data association ➢ discrete state space ➢ constrained to detection responses [Jiang et al. '07] [Huang et al. '08] [Zhang et al. '08] [Pirsiavash et al. '11] [Brendel et al. '11] [Benfold & Reid '11] [Yang & Nevatia '12] occlusion -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
11 Energy-based Multi-Target Tracking • Goal: Design an energy function that ➢ accurately represents the problem at hand, • complete and appropriate state representation ➢ captures important aspects / dependencies, • dynamics, exclusion, occlusion,... ➢ can be optimized efficiently. • global or “strong” local minima -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
12 discrete continuous discrete-continuous Energy Domain -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
13 Discrete Optimization Network Flow Graph-based Approaches [Zhang et al. '08] [Pirsiavash et al. '11] [Henriques et al. '11] [Butt & Collins '13] ... Integer Linear Programming (ILP) [Morefield '77] [Jiang et al. '07] [Berclaz et al. '09] [Andriyenko et al. '10] [Berclaz et al. '11] ... [Wu et al. '11] [Brendel et al. '11] [Zamir et al. '12] [Yang & Nevatia '12] ... -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
14 discrete continuous discrete-continuous Energy Domain -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
15 Continuous Energy Minimization state X = all targets in all frames E( , , ) [Andriyenko & Schindler CVPR '11, Andriyenko et al. ICCV-WS '11, Milan et al., PAMI '14] • No restrictions on the energy ➢ dynamics, exclusion, persistence... • Entirely in continuous space • Solve to local optimality with greedy jump moves -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
16 Continuous Energy Function E =E obs +aE dyn +bE exc +cE per +dE reg data physically-based priors regularizer dynamics exclusion persistence parsimony high energy low energy ∑ i≠ j ∣ ∣X i −X j ∣ ∣−2 −∑ g ∣ ∣X i −D g ∣ ∣−2 ∑ i ∣ ∣v i t −v i t 1∣ ∣2 N+∑ i 1/length i ∑ i 1exp1−b X i −1 -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
17 Non-convex Optimization • conjugate gradient descent for local optimization • several runs from different initializations • discontinuous jumps to determine dimensionality shrink add split E(X) X Jump moves Conjugate gradient descent grow remove merge -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
18 Continuous Optimization (with discontinuous jumps) -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
19 Continuous Optimization Video Results -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
20 Data Association ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Target 1, …, N? Target N+1? False Detection? time space -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
21 discrete continuous discrete-continuous Energy Domain -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
22 Discrete-Continuous Optimization E( , , , , , ) [Andriyenko et al. CVPR '12, Milan et al. CVPR '13] • Unified energy for both data association and trajectory estimation • Powerful discrete optimization • Natural continuous space -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
23 Discrete-Continuous Energy E(f, ) = Unaries + Pairwise + Label cost data smoothness exclusion dynamics, occlusion persistence, regularizer collisions time ∑ g ∣ ∣X i −D g ∣ ∣ ∑ E s δ[a−b] ∑ E x (1−δ[a−b]) -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
24 Discrete-Continuous Optimization α Alternating Optimization Least Squares / L-BFGS E(f) E( ) (modified) - expansion -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
25 Discrete-Continuous Optimization -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
26 • Quadratic Boolean Programming (QBP) • Lagrangian Dual Decomposition • Training Detector with Tracker-in-the-Loop ➢ Joint people detector ➢ Mining occlusion patterns Posters 2A (P2A-13) Wednesday Morning [Leibe et al. CVPR '07] [Wu et al. CVPR '12] [Tang et al. ICCV '13] Detector MOTA HOG 19.1 % DPM 21.8 % Joint-Design 23.0 % Joint-Learn 1st iter. 23.4 % Joint-Learn 2nd iter. 26.8 % PETS-S1L2 Combining Detection and Tracking -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
27 Summary • Continuous Energy Minimization ✔ Accurate representation and modeling ✔ State-of-the-art results despite local optima ✗ No explicit data association • Discrete-Continuous Energy Minimization ✔ Unified energy for data association and trajectory estimation ✔ Powerful discrete optimization techniques can be applied ✔ Accurate and complete state representation -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
28 Further Directions • Far from solved in challenging / crowded conditions • Heavy / long-term occlusions • Accurate cue extraction • Proper benchmarking, cf. [Milan et al., CVPR-WS '13] -
| Anton Milan | Energy Minimization for Multi-Object Tracking |
29 Thank you! Energy Minimization for Multiple Object Tracking