Reading Circle (Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation)

Explanation of Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation [Wang+, CVPR20]

Transcript

1. Reading Circle Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation [Wang+, CVPR20]
2. Semantic Segmentation Assign a semantic category to each pixel. 2 Supervised Dataset : image & pixel-level class label ✗ huge annotation cost ↓ Weakly-supervised Dataset : image & image-level class label
3. Weakly-Supervised SS Methods Three steps to train on the image-level label. 1. predict an initial category-wise response map to localize the object 2. refine the initial response as the pseudo GT 3. train the segmentation network based on pseudo labels 3 1 2 3
4. Weakly-Supervised SS Methods Three steps to train on the image-level label. 1. predict an initial category-wise response map to localize the object 2. refine the initial response as the pseudo GT 3. train the segmentation network based on pseudo labels 4 1
5. What’s New Introduce a self-supervised equivariant attention mechanism (SEAM). - Narrow the supervision gap between fully and weakly supervised semantic segmentation 5
6. What’s New Introduce a self-supervised equivariant attention mechanism (SEAM). - Focus on affine transformation 6 Previous Proposed
7. What’s New Introduce a self-supervised equivariant attention mechanism (SEAM). - Focus on affine transformation 7 Previous Proposed CAM varies depending on the size of the input image
8. What’s New Introduce a self-supervised equivariant attention mechanism (SEAM). - Focus on affine transformation 8 Previous Proposed Consistent CAM regardless of the size of the input image
9. Network Architecture of SEAM Use Siamese Network and three kinds of Losses. 9
10. Network Architecture of SEAM Use Siamese Network and three kinds of Losses. 10
11. Pixel Correlation Module (PCM) Modify CAM by self attention mechanism. 11
12. Pixel Correlation Module (PCM) Modify CAM by self attention mechanism. 12 “Non-local neural networks” [Wang+, CVPR18]
13. Self Attention [Wang+, CVPR18] Non-local mean operation x : input signal y : output signal g : representation function f : similarity function (scalar) 13 Gaussian Embedded Gaussian Dot product Concatenation
14. Self Attention [Wang+, CVPR18] Non-local block 14
15. Pixel Correlation Module (PCM) Modify CAM by self attention mechanism. 15 “Non-local neural networks” [Wang+, CVPR18]
16. Pixel Correlation Module (PCM) Modify CAM by self attention mechanism. 16
17. Network Architecture of SEAM Use Siamese Network and three kinds of Losses. 17
18. Loss Design of SEAM 1. Class Loss : multi-label soft margin loss 18 original CAM
19. Network Architecture of SEAM Use Siamese Network and three kinds of Losses. 19
20. Loss Design of SEAM 2. Equivariant Regularization (ER) Loss 20 Consistency between before and after affine transformation
21. Network Architecture of SEAM Use Siamese Network and three kinds of Losses. 21
22. Loss Design of SEAM 2. Equivariant Cross Regularization (ECR) Loss 22 to further improve the ability of network for equivariance learning
23. Network Architecture Use Siamese Network and three kinds of Losses. 23
24. Dataset PASCAL VOC 2012 semantic segmentation benchmark 21 categories one or multiple object class 1,464 images in training set 1,449 images in validation set 1,456 images in test set 24
25. Result -CAM- 25 Proposed Baseline GT Original
26. Result -Semantic Segmentation- 26 Input GT Ours
27. Quantitative Comparison Evaluation of transformations on equivariant regularization - evaluation metric : ↑ mIoU (%) 27
28. Quantitative Comparison Category performance comparison - evaluation metric : ↑ mIoU (%) 28
29. Quantitative Comparison Evaluation of WSSS performance 29 [Chang+, CVPR20] val : 66.1 test : 65.9
30. Conclusion Weakly-supervised Learning 1. generate pseudo GT label 2. apply supervised learning Weakly-supervised Semantic Segmentation Self-supervised Equivariant Attention Mechanism ↓ appropriate CAM ↓ better pseudo GT label 30