Deep Relative Attribute

Presented at Winter Seminar Series 2015 at Sharif. http://ssc.ce.sharif.edu/events/wss/2015/

Transcript

1. Deep Relative Attributes Yaser Souri, Erfan Noury, Ehsan Adeli-Mosabbeb
2. Visual Recognition Typical Recognition Dog Panda Bear Dog
3. Visual Recognition Attribute Recognition 4 legs white 4 legs black&white 4 legs brown 4 legs black&white
4. Attributes Mid-Level semantic properties, shared by objects Human-understandable and machine-detectable Ferrari, Vittorio, and Andrew Zisserman. "Learning visual attributes." NIPS 2007. Farhadi, Ali, et al. "Describing objects by their attributes." CVPR 2009.
5. Attributes Applications: • Zero-shot learning • Visual Search • Interactive Recognition Lampert, Christoph H., et al. "Learning to detect unseen object classes by between-class attribute transfer." CVPR 2009. Kovashka, Adriana, et al. "Whittlesearch: Image search with relative attribute feedback."CVPR 2012. Branson, Steve, et al. "Visual recognition with humans in the loop." ECCV 2010.
6. Relative Attributes The problem with Binary/Categorical Attributes Smiling Not Smiling ?
7. Relative Attributes The problem with Binary/Categorical Attributes Smiling Not Smiling > > less than more than Devi Parikh, and Kristen Grauman. "Relative attributes." ICCV 2011. Marr Prize
8. Learn a function from a set of pairwise supervisions Relative Attributes Devi Parikh, and Kristen Grauman. "Relative attributes." ICCV 2011.
9. Previous works lack: • Use hand-engineered features (GIST, HOG, …) • Use complicated and often special purpose models We propose: • Using Convolutional Neural Networks and a simple ranking model • Learning features and ranking end-to-end Relative Attributes Related Work Devi Parikh, and Kristen Grauman. "Relative attributes." ICCV 2011. Li, Shaoxin, et al. "Relative forest for attribute prediction." ACCV 2012. Yu, Anbo, and Kristen Grauman. "Fine-grained visual comparisons with local learning." CVPR 2014. Sandeep, Ramachandruni N., et al. "Relative Parts: Distinctive Parts for Learning Relative Attributes." CVPR 2014.
10. Deep Relative Attributes Test time linear and global ranking model ConvNet
11. Deep Relative Attributes Train time Burges, Chris, et al. "Learning to rank using gradient descent." ICML 2005.
12. Deep Relative Attributes Train time Burges, Chris, et al. "Learning to rank using gradient descent." ICML 2005.
13. Deep Relative Attributes Train time Burges, Chris, et al. "Learning to rank using gradient descent." ICML 2005.
14. Deep Relative Attributes Details • Each attribute separately • ConvNet: VGG16 (conv1 - fc7) • Initial weights of the ConvNet: pre-trained on ILSVRC2014 • Optimization: ◦ SGD with minibatch of size 32 (16 x 2) ◦ RMSProp + weight decay ◦ Gradient clipping ◦ Random shuffling of data for each epoch + flip augmentation for small datasets • Code: Theano + Lasagne
15. Results Global Ranking
16. Results Relative Attribute Prediction
17. Results Relative Attribute Prediction
18. Results Relative Attribute Prediction
19. Results Relative Attribute Prediction
20. Results Saliency Maps and Localization Simonyan, Karen, et al. "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps." ICLR Workshop 2014.
21. Results Saliency Maps and Localization
22. Analysis Feature learning effect Pretrained features tSNE Embedding
23. Analysis Feature learning effect Learned features tSNE Embedding
24. Conclusion A method for end-to-end feature learning and ranking of images State-of-the-art relative attribute prediction results Our model is able to localize attributes Code and models will be released soon Paper at arXiv: http://arxiv.org/abs/1512.04103
25. ? Thank you