HADES: Hierarchical Approximate Decoding for Structured Prediction

A 30-min talk covering my MSc. thesis defense at the Data Analytics Lab at ETH Zürich. For information, visit here: http://e-collection.library.ethz.ch/view/eth:48405

Transcript

1. HADES : Hierarchical Approximate Decoding for Structured Prediction Tribhuvanesh Orekondy ETH Zürich Advisors Martin Jaggi Aurelien Lucchi Thomas Hofmann
2. Problem Statement
3. 3 Building Car Road Computer Vision Setting: Structured Output Learning This is a tagged sentence DT VBZ DT JJ NN Natural Language Processing
4. Setting: Structured Output Learning 4 DT VBZ DT JJ NN Predict discrete output variable This is a tagged sentence Observed input variable Learn good predictor (parameterized by )
5. Setting: Structured Output Learning 5 Learning = Estimation of Structured SVM
6. Setting: Structured Output Learning 6 Optimization
7. 7 Problem Maximization Oracle This is a tagged sentence DT VBZ DT JJ NN ? ? ? ? ?
8. 8 Problem Maximization Oracle ? ? ? ? ? Y1 Y2 Y3 Y4 Y5
9. Problem Maximization Oracle Y1 Y4 Y2 Y5 Y3 Y6 9 Image Segmentation bike person
10. Approximate solutions 10 Maximization Oracle “.. learning can fail even with an approximate inference method with rigorous approximation guarantees ..” This thesis How can we learn using approximate solutions for Image Segmentation?
11. Approach
12. Y ρl-Maximization Oracle Hierarchical Decomposition Approach - Optimization 12 Y1 Y2 Y3 Maximization Oracle Level 3
13. Approach - Optimization 13 Recall (in each iteration )
14. Approach - Optimization 14 Recall (in each iteration ) Block-Coordinate Frank Wolfe
15. Approach - Optimization 15 Hierarchical Decoding Strategy (in each iteration ) ρl-Maximization Oracle Step-size Update }
16. Application - Image Segmentation 16
18. Y1 18 Y2 Y3 Surrogate CRF
19. Y1 19 Y2 Y3 Coarse Fine
20. 20 Coarse Fine
21. Approach - Optimization 21 Hierarchical Decoding Strategy Coarse-to-fine decoding
22. Surrogate CRF 22 Want
23. Surrogate CRF 23 For some ,
24. Surrogate CRF 24
25. Surrogate CRF 25 Unaries
26. Surrogate CRF 26 Pairwise
27. Propositions 27 Equivalence of CRFs Equivalent loss-augmented decoding Hierarchical decoding
28. Results
29. • Data • MSRC-21 29 Results - Setup cat road grass cow cow
30. • Data • MSRC-21
 • dissolvestruct 30 Results - Setup
31. • Data • MSRC-21
 • dissolvestruct
 • Features • Unary: CNN • Pairwise: Orientation, Edge Intensity 31 Results - Setup
32. • Data • MSRC-21
 • dissolvestruct
 • Features • Unary: CNN • Pairwise: Orientation, Edge Intensity
 • Hierarchy • HSLIC: 6 Levels
 32 Results - Setup
33. • Data • MSRC-21
 • dissolvestruct
 • Features • Unary: CNN • Pairwise: Orientation, Edge Intensity
 • Hierarchy • HSLIC: 6 Levels
 • Evaluation Metric 33 Results - Setup
34. 34 Results - Accuracy 75% 26.03 ± 1.3 55.08 ± 1.57 80 - 87%
35. Results - Passes 35 4x 57x
36. Results - Step-sizes & Energy 36
37. • RPL
 Resume Previous Level 37 Results - RPL + STUBR • STUBR
 Stub Repetitions
38. 38 Results - RPL + STUBR
39. Results - RPL + STUBR 39 BARE RPL + STUBR
40. • Motivation
 Max-oracles are expensive in Computer Vision.
 Learning using approximate oracles are not well understood. • Approach
 Coarse-to-fine approximation-based BCFW-variant.
 Hierarchical Surrogate CRF model for Image Segmentation. • Results
 Approximate decoding is 50-60x faster.
 75% mark obtained 1.5x-4x faster. 40 Conclusion
41. THANK YOU!
42. Addendum
44. • Surrogate Energy Function
 • Unary Factor El(Y = y |X = x ; w ) = X u2 ˜ Vl h w D yu , xu i + X (u,v)2 ˜ El w P yuyv xu xi xj xk xu = X i2atm(u) xi SURROGATE CRF - DEFINITION ˜ Gl = ( ˜ Vl, ˜ El) “atom” “supernode”
45. • Surrogate Energy Function
 • Pairwise Factor X ( u,v ) 2 ˜ El P(yu, yv; wP ) = X ( u,v ) 2 ˜ El X ( i,j ) 2 atmE ( u,v ) P(Yi = yu, Yj = yv; wP ) | {z } Supernode transition + X u2 ˜ Vl X ( i,j ) 2 atmE ( u ) P(Yi = Yj = yu; wP ) | {z } Static transition El(Y = y |X = x ; w ) = X u2 ˜ Vl h w D yu , xu i + X (u,v)2 ˜ El w P yuyv SURROGATE CRF - DEFINITION ˜ Gl = ( ˜ Vl, ˜ El) “atom” “supernode”
46. PROPOSITIONS† • Equivalence of CRFs
 
 • Equivalent Loss-Augmented Decoding
 
 • Hierarchical Decoding argmin y2Ym l E( y ; x m, w ) ⌘ argmin y2 ˜ Ym l El( y ; x m, w ) argmin y2Yl E( y ; x m, w ) ( y , y m) ⌘ argmin y2 ˜ Yl El( y ; x m, w ) ( y , y m) min y2Yl+1 E( y ; x , w )  min y2Yl E( y ; x , w ) † Proofs excluded
47. APPROXIMATION QUALITY • Gauge
 • Additive Error E( y ⇤; x , w )  El( y ⇤; x , w )  E( y ⇤; x , w ) + ⇢(l) E⇤ P  E⇤ l  E⇤ P + ⇢(l) E⇤ l E⇤ P  ( P Nl) · 2 BRU + ( Z Tl) · 2 BRP P # Nodes – level P Nl # Nodes – level l Z # Edges – level P Tl # Super-node transitions – level l kwi k  B k xi k  RU k P ( yi, yj) k  RP
48. Motivation - Image Segmentation 48
50. Application - Image Segmentation 50 Maximization Oracle
51. Application - Image Segmentation 51
52. Results - Unaries vs. Pairwise 52
53. Results - Unaries vs. Pairwise 53
54. Results - -threshold 54
55. Results - Fixed Schedule 55
56. Results - Stub Repetitions 56
57. Results - Resume Previous Level 57
58. Results - Resume Previous Level 58
59. 59 Results - BP iterations 20 iterations 40 iterations 60 iterations
60. Results - Unaries vs. Pairwise 60 Bin 2 SLIC HSLIC
61. Results - Unaries vs. Pairwise 61 Bin 3 SLIC HSLIC