Deep Relative Attributes Yaser Souri, Erfan Noury, Ehsan Adeli-Mosabbeb 

Visual Recognition Typical Recognition Dog Panda Bear Dog 

Visual Recognition Attribute Recognition 4 legs white 4 legs black&white 4 legs brown 4 legs black&white 

Attributes Mid-Level semantic properties, shared by objects Human-understandable and machine-detectable Ferrari, Vittorio, and Andrew Zisserman. "Learning visual attributes." NIPS 2007. Farhadi, Ali, et al. "Describing objects by their attributes." CVPR 2009. 

Attributes Applications: ● Zero-shot learning ● Visual Search ● Interactive Recognition Lampert, Christoph H., et al. "Learning to detect unseen object classes by between-class attribute transfer." CVPR 2009. Kovashka, Adriana, et al. "Whittlesearch: Image search with relative attribute feedback."CVPR 2012. Branson, Steve, et al. "Visual recognition with humans in the loop." ECCV 2010. 

Relative Attributes The problem with Binary/Categorical Attributes Smiling Not Smiling ? 

Relative Attributes The problem with Binary/Categorical Attributes Smiling Not Smiling > > less than more than Devi Parikh, and Kristen Grauman. "Relative attributes." ICCV 2011. Marr Prize 

Learn a function from a set of pairwise supervisions Relative Attributes Devi Parikh, and Kristen Grauman. "Relative attributes." ICCV 2011. 

Previous works lack: ● Use hand-engineered features (GIST, HOG, …) ● Use complicated and often special purpose models We propose: ● Using Convolutional Neural Networks and a simple ranking model ● Learning features and ranking end-to-end Relative Attributes Related Work Devi Parikh, and Kristen Grauman. "Relative attributes." ICCV 2011. Li, Shaoxin, et al. "Relative forest for attribute prediction." ACCV 2012. Yu, Anbo, and Kristen Grauman. "Fine-grained visual comparisons with local learning." CVPR 2014. Sandeep, Ramachandruni N., et al. "Relative Parts: Distinctive Parts for Learning Relative Attributes." CVPR 2014. 

Deep Relative Attributes Test time linear and global ranking model ConvNet 

Deep Relative Attributes Train time Burges, Chris, et al. "Learning to rank using gradient descent." ICML 2005. 

Deep Relative Attributes Train time Burges, Chris, et al. "Learning to rank using gradient descent." ICML 2005. 

Deep Relative Attributes Train time Burges, Chris, et al. "Learning to rank using gradient descent." ICML 2005. 

Deep Relative Attributes Details ● Each attribute separately ● ConvNet: VGG16 (conv1 - fc7) ● Initial weights of the ConvNet: pre-trained on ILSVRC2014 ● Optimization: ○ SGD with minibatch of size 32 (16 x 2) ○ RMSProp + weight decay ○ Gradient clipping ○ Random shuffling of data for each epoch + flip augmentation for small datasets ● Code: Theano + Lasagne 

Results Global Ranking 

Results Relative Attribute Prediction 

Results Relative Attribute Prediction 

Results Relative Attribute Prediction 

Results Relative Attribute Prediction 

Results Saliency Maps and Localization Simonyan, Karen, et al. "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps." ICLR Workshop 2014. 

Results Saliency Maps and Localization 

Analysis Feature learning effect Pretrained features tSNE Embedding 

Analysis Feature learning effect Learned features tSNE Embedding 

Conclusion A method for end-to-end feature learning and ranking of images State-of-the-art relative attribute prediction results Our model is able to localize attributes Code and models will be released soon Paper at arXiv: http://arxiv.org/abs/1512.04103 

? Thank you