Search fashion items by colors

A003bc0b49c7e5adf746b6d73cb57956?s=47 shoby
March 22, 2016

Search fashion items by colors

A003bc0b49c7e5adf746b6d73cb57956?s=128

shoby

March 22, 2016
Tweet

Transcript

  1. Search fashion items by colors Fablic, inc. Shota Izaka ⛄

  2. Shota Izaka • Fablic, inc. • Developing Fril • iOS

    / Ruby on Rails / Image Processing • Search by colors / Apple Watch App / Swift migration  !TIPCZTIPCZ
  3. What is Fril? • Fashion market place app • Similar

    to Poshmark • Buy and sell fashion items • Fril mediates trading 
  4. Outline 

  5. Outline • What is color search? • Why we implemented

    it? • Implementation • Create color palette • Color detection by image processing • Result 
  6. What is color search? 

  7. 

  8. What is color search? • Narrow down items by colors

    • e.g. One-piece dress & white • e.g. Coat & brown • Detect colors automatically • Sellers input nothing • Color detection from item images 
  9. Why we implemented it? 

  10. Why we implemented it? • Most users choosing clothing colors

    • Questionnaire • Interview • Search word logs • To specialize in fashion • Fashion specialized search features 
  11. Implementation #1 Create color palette 

  12. Use RGB values  255 * 255 * 255 =

    16,581,375 colors
  13. Create color palette  12 colors ☺

  14. Why 12 colors? 

  15. Why 12 colors? • From human color recognition research[1] •

    All languages have basic 11 color terms • 11 colors are universal • Red, yellow, green, blue, orange, purple, pink, brown, gray, black, white • From search queries including color names • Listed queries in descending order • Merged into basic 11 colors (e.g camel is brown) • Leaved beige because it can't merge  [1] Berlin B.and Kay P, “Basic Color Terms:Their Universality and Evolution.”
  16. Implementation #2 Color detection by image processing 

  17. Color detection by image processing • Color correction by white

    balancing • Remove background • Detect a mode color • Choose nearest color palette 
  18. Color correction by white balancing 

  19. Color correction by white balancing • Correct colors of images

    taken under fluorescent lamp • Typical fluorescent lights make images greenish • Fashion images usually taken inside a room • Correct colors with Retinex[2] • A method modeled human visual features • Do not affect photoshopped images  [2] E. Land and J. McCann, “Lightness and Retinex theory,”
  20. require 'RMagick' image = ImageList.new('image.jpg') red_extrema = image.channel_extrema(Magick::RedChannel) green_extrema =

    image.channel_extrema(Magick::GreenChannel) blue_extrema = image.channel_extrema(Magick::BlueChannel) red_max = red_extrema.last.to_f green_max = green_extrema.last.to_f blue_max = blue_extrema.last.to_f white_balanced_image = image white_balanced_image = white_balanced_image.convolve_channel(1, [green_max / red_max], Magick::RedChannel) white_balanced_image = white_balanced_image.convolve_channel(1, [green_max / blue_max], Magick::BlueChannel)  Retinex using RMagick
  21.  Before After green white

  22. Remove background 

  23. Remove background • Eliminate the effect of background colors •

    Make background mask to remove • Make background mask • Detect edges • Flood magenta from corners • Extract magenta regions as background mask 
  24.  Original Detect edges Flood magenta Background mask

  25. Detect a mode color 

  26. Detect a mode color • Detect a mode color from

    the central portion • Fashion items usually appear in center of photo • Central portion is less affected by background colors • Detect a mode color • Crop the central portion • Reduce colors into 8 • Composite background mask • Make histogram and detect a mode 
  27.  Original Crop center ˍ Reduce colors Composite mask Make

    histogram & Detect mode #A3886B
  28. Choose nearest color palette 

  29. Choose nearest color palette • RGB to 12 colors •

    255*255*255ʢ16,581,375ʣ → 12 • Use color name survey[3] • 196,608 RGB values are named • We converted to HSL and used the range of each color names • Choose nearest color palette • Conver RGB to HSL • Choose nearest palette considering HSL range  IUUQCMPHYLDEDPNDPMPSTVSWFZSFTVMUT
  30. Me: ”Succeeded!” 

  31. A internal user: “It’s strange” 

  32. General colors != Fashion colors 

  33. Color recognition is different by fashion knowledge 

  34.  8IBUDPMPSJTUIJT HSFFO #81D4CB

  35.  5J⒎BOZlCMVFz

  36. Tuned up with showing to internal uses 

  37.  Completed

  38. Result 

  39. Accuracy of color detection 

  40. Accuracy of color detection • Checked 30 items visually in

    major categories • Average is 48.36% • Well appeared colors are well • Black: 90.67% • Red: 79.33% • Pink: 72.67% • White: 56.67% • Less appeared colors affected by false detection of other colors 
  41. Item found proportion in search result 

  42. Item found ratio in search result • We defined item

    found proportion • Percentage of users who opened a item in search result • Item found • Keyword search: about 69% • Color search: about 75% • Color search become easier to find items than keyword search 
  43. Summary 

  44. Summary • We implemented color search • Researched user buying

    behavior and implemented fashion specialized search • Color palette were created from human recognition research and search queries • Correcting color, removing background, cropping center and detect nearest color palette • Color search become easier to find items than keyword search 
  45. Thanks  ⛄