(CVPR2022) Integrative Few-Shot Learning for Classification and Segmentation

Transcript

1. Integrative Few-Shot Learning for Classification and Segmentation CVPR 2022 ~Paper reading seminar~ Kazuya Nishimura
2. About me  Name: Kazuya Nishimura (Bise laboratory:D2)  Birth of place: Shimane  Graduate: National Institute of Technology, Matsue College  Research interest: semi or weakly supervised learning Cell mitosis detection [EMBC 2020, CVPRW second prize] Cell segmentation [MIA2021, MICCAI 2019] Cell tracking [ECCV2020]
3. Summary of introduce paper Integrative Few-Shot Learning for Classification and Segmentation 3 3 labeled image for different class object = Support set Recognition target = Query image Aim: identify class and perform segmentation Inputs Outputs
4. Summary of introduce paper Integrative Few-Shot Learning for Classification and Segmentation 4 3 labeled image for different class object = Support set Aim: identify class and perform segmentation If query does not belong support set, the method predict false Inputs Outputs Recognition target = Query image
5. Few-shot learning  Semi-supervised learning: Small amount of labeled data + large amount of unlabeled data  Few-shot learning: Small amount of labeled data + pretrained CNN model 5 How to effectively use few labeled data? How to effectively use unlabeled data?
6. Few-shot learning  Semi-supervised learning: Small amount of labeled data + large amount of unlabeled data  Few-shot learning: Small amount of labeled data + pretrained CNN model N way K shot 6 𝑆𝑆 = 𝑥𝑥𝑠𝑠 𝑖𝑖, 𝑦𝑦𝑠𝑠 𝑖𝑖 𝑦𝑦𝑠𝑠 𝑖𝑖 ∈ 𝐶𝐶𝑠𝑠 𝑖𝑖=0 𝑁𝑁𝑁𝑁 How to effectively use few labeled data? How to effectively use unlabeled data? N class K images for each class
7. Few-shot learning  Semi-supervised learning: Small amount of labeled data + large amount of unlabeled data  Few-shot learning: Small amount of labeled data + pretrained CNN model N way K shot 7 𝑆𝑆 = 𝑥𝑥𝑠𝑠 𝑖𝑖, 𝑦𝑦𝑠𝑠 𝑖𝑖 𝑦𝑦𝑠𝑠 𝑖𝑖 ∈ 𝐶𝐶𝑠𝑠 𝑖𝑖=0 𝑁𝑁𝑁𝑁 How to effectively use few labeled data? How to effectively use unlabeled data? N class K images for each class CNN Image net pretrained K class does not include Imagenet class
8. Background: few-shot classification and segmentation 8 CNN  Few-shot classification [Chen+ ICLR 2019] Support set “ferret” “crab” 𝑥𝑥𝑠𝑠 𝑖𝑖 𝑥𝑥𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑖𝑖 … “ferret” … “crab” Multi-class output
9.  Few-shot segmentation [Rakelly+, ICLR 2018] Background: few-shot classification and segmentation 10 CNN CNN Estimate salient object 𝑥𝑥𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑗𝑗 Support sample  Few-shot classification [Chen+ ICLR 2019] Support set “ferret” “crab” 𝑥𝑥𝑠𝑠 𝑖𝑖 𝑥𝑥𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑖𝑖 … One sample is used “ferret” … “crab” Multi-class output
10.  Few-shot segmentation [Rakelly+, ICLR 2018] Background: few-shot classification and segmentation 11 CNN CNN Estimate salient object 𝑥𝑥𝑠𝑠 𝑖𝑖 𝑥𝑥𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑖𝑖 Support set …  Few-shot classification [Chen+ ICLR 2019] Support set “ferret” “crab” 𝑥𝑥𝑠𝑠 𝑖𝑖 𝑥𝑥𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑖𝑖 … “ferret” CNN Support set 𝑥𝑥𝑠𝑠 𝑖𝑖 𝑥𝑥𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑗𝑗 𝑦𝑦𝑠𝑠 𝑖𝑖 … Few-shot classification and segmentation Estimate N class segment … “crab” Multi-class output
11. Support set Query Idea: new architecture for few-shot classification and segmentation 12 Classification Segmentation Inputs Outputs New architecture Multi-class classification Multi-class segmentation
12. Support set Query Idea: new architecture for few-shot classification and segmentation 13 Classification Segmentation Inputs Outputs New architecture Outputs are different from previous  Weakly supervised architecture!
13. Overview of method 14 CNN CNN Support set Query Attentive Squeeze network
14. Overview of method 15 CNN CNN Support set Query Attentive Squeeze network Classification Max pooling
15. Overview of method 16 CNN CNN Support set Query Attentive Squeeze network Classification Max pooling Argmax Segmentation
16. Overview of method 17 CNN CNN Support set Query Attentive Squeeze network Classification Max pooling Argmax Segmentation Loss Gt Loss Gt
17. Overview of method 18 CNN CNN Support set Query Classification Max pooling Argmax Segmentation Loss Gt Loss Gt Attentive Squeeze network
18. Attentive Squeeze network  Multi scale Hypercorrelation module The network can aggregate multi-scale correlation features 19 Support multi-scale features Extracted form ResNet Query multi-scale features Extracted form ResNet Cosine similarity !
19. Experiments: Comparisons 20  Dataset: Pascal-5i (famous few shot segmentation dataset) Method 1 way 1 shot 2 way 1 shot Classification acc. Segmentation mIou classification segmentation Wang+, ICCV 2019 69.0 36.2 50.9 37.2 Tian+, TPAMI 2020 74.6 43.0 41.0 35.3 Min, ICCV 2021 83.7 49.7 67.3 43.5 Ours 84.9 52.3 68.3 47.8 1 class contain one sample 2 class contain e sample
20. Experiments: Comparisons 21  Dataset: COCO-20i (famous few shot segmentation dataset) Method 1 way 1 shot 2 way 1 shot Classification acc. Segmentation mIou classification segmentation Wang+, ICCV 2019 66.7 25.2 48.5 23.6 Tian+, TPAMI 2020 71.4 31.9 36.5 22.6 Min, ICCV 2021 77.0 34.3 62.5 29.5 Ours 78.6 35.8 63.1 31.6 1 class contain one sample 2 class contain e sample
21. Experiments: Performance of N way settings 22 Improvement is slight
22. Experiments: Performance of N way settings 23 Improvement is slight Thanks to argmax operation
23. Qualitative result  2 way 1shot 24 1st labeled 2nd labeled Prediction Ground-truth True negative case 1st labeled include input 2nd labeled include input both labeled include input
24. Summary  The paper introduced “few-shot classification and segmentation” The task integrate classification and segmentation Given query and support set, the model output class and segment  Technical novelty Introduce weakly-supervised segmentation representation to this task By considering other class, the method improved performance Proposed Attentive Squeeze network It aggregate multi-scale Hypercorrelation information  What I thought 25