Search fashion items by colors

This is english translated version of following slide.
https://speakerdeck.com/shoby/hurirufalseshang-pin-wose-tejian-suo-tekiruyounisitahua

Fashion Tech Meetup #2
http://fashion-tech.connpass.com/event/28335/

Transcript

1. Search fashion items by colors Fablic, inc. Shota Izaka ⛄
2. Shota Izaka • Fablic, inc. • Developing Fril • iOS / Ruby on Rails / Image Processing • Search by colors / Apple Watch App / Swift migration  !TIPCZTIPCZ
3. What is Fril? • Fashion market place app • Similar to Poshmark • Buy and sell fashion items • Fril mediates trading 
4. Outline 
5. Outline • What is color search? • Why we implemented it? • Implementation • Create color palette • Color detection by image processing • Result 
6. What is color search? 
7. 
8. What is color search? • Narrow down items by colors • e.g. One-piece dress & white • e.g. Coat & brown • Detect colors automatically • Sellers input nothing • Color detection from item images 
9. Why we implemented it? 
10. Why we implemented it? • Most users choosing clothing colors • Questionnaire • Interview • Search word logs • To specialize in fashion • Fashion specialized search features 
11. Implementation #1 Create color palette 
12. Use RGB values  255 * 255 * 255 = 16,581,375 colors
13. Create color palette  12 colors ☺
14. Why 12 colors? 
15. Why 12 colors? • From human color recognition research[1] • All languages have basic 11 color terms • 11 colors are universal • Red, yellow, green, blue, orange, purple, pink, brown, gray, black, white • From search queries including color names • Listed queries in descending order • Merged into basic 11 colors (e.g camel is brown) • Leaved beige because it can't merge  [1] Berlin B.and Kay P, “Basic Color Terms:Their Universality and Evolution.”
16. Implementation #2 Color detection by image processing 
17. Color detection by image processing • Color correction by white balancing • Remove background • Detect a mode color • Choose nearest color palette 
18. Color correction by white balancing 
19. Color correction by white balancing • Correct colors of images taken under fluorescent lamp • Typical fluorescent lights make images greenish • Fashion images usually taken inside a room • Correct colors with Retinex[2] • A method modeled human visual features • Do not affect photoshopped images  [2] E. Land and J. McCann, “Lightness and Retinex theory,”
20. require 'RMagick' image = ImageList.new('image.jpg') red_extrema = image.channel_extrema(Magick::RedChannel) green_extrema = image.channel_extrema(Magick::GreenChannel) blue_extrema = image.channel_extrema(Magick::BlueChannel) red_max = red_extrema.last.to_f green_max = green_extrema.last.to_f blue_max = blue_extrema.last.to_f white_balanced_image = image white_balanced_image = white_balanced_image.convolve_channel(1, [green_max / red_max], Magick::RedChannel) white_balanced_image = white_balanced_image.convolve_channel(1, [green_max / blue_max], Magick::BlueChannel)  Retinex using RMagick
21.  Before After green white
22. Remove background 
23. Remove background • Eliminate the effect of background colors • Make background mask to remove • Make background mask • Detect edges • Flood magenta from corners • Extract magenta regions as background mask 
24.  Original Detect edges Flood magenta Background mask
25. Detect a mode color 
26. Detect a mode color • Detect a mode color from the central portion • Fashion items usually appear in center of photo • Central portion is less affected by background colors • Detect a mode color • Crop the central portion • Reduce colors into 8 • Composite background mask • Make histogram and detect a mode 
27.  Original Crop center ˍ Reduce colors Composite mask Make histogram & Detect mode #A3886B
28. Choose nearest color palette 
29. Choose nearest color palette • RGB to 12 colors • 255*255*255ʢ16,581,375ʣ → 12 • Use color name survey[3] • 196,608 RGB values are named • We converted to HSL and used the range of each color names • Choose nearest color palette • Conver RGB to HSL • Choose nearest palette considering HSL range  IUUQCMPHYLDEDPNDPMPSTVSWFZSFTVMUT
30. Me: ”Succeeded!” 
31. A internal user: “It’s strange” 
32. General colors != Fashion colors 
33. Color recognition is different by fashion knowledge 
34.  8IBU
    DPMPS
    JT
    UIJT HSFFO #81D4CB
35.  5J⒎BOZ
    lCMVFz
36. Tuned up with showing to internal uses 
37.  Completed
38. Result 
39. Accuracy of color detection 
40. Accuracy of color detection • Checked 30 items visually in major categories • Average is 48.36% • Well appeared colors are well • Black: 90.67% • Red: 79.33% • Pink: 72.67% • White: 56.67% • Less appeared colors affected by false detection of other colors 
41. Item found proportion in search result 
42. Item found ratio in search result • We defined item found proportion • Percentage of users who opened a item in search result • Item found • Keyword search: about 69% • Color search: about 75% • Color search become easier to find items than keyword search 
43. Summary 
44. Summary • We implemented color search • Researched user buying behavior and implemented fashion specialized search • Color palette were created from human recognition research and search queries • Correcting color, removing background, cropping center and detect nearest color palette • Color search become easier to find items than keyword search 
45. Thanks  ⛄