Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search fashion items by colors

shoby
March 22, 2016

Search fashion items by colors

shoby

March 22, 2016
Tweet

More Decks by shoby

Other Decks in Technology

Transcript

  1. Search fashion items by colors
    Fablic, inc.
    Shota Izaka

    View Slide

  2. Shota Izaka
    • Fablic, inc.
    • Developing Fril
    • iOS / Ruby on Rails / Image Processing
    • Search by colors / Apple Watch App / Swift
    migration

    !TIPCZTIPCZ

    View Slide

  3. What is Fril?
    • Fashion market place app
    • Similar to Poshmark
    • Buy and sell fashion items
    • Fril mediates trading

    View Slide

  4. Outline

    View Slide

  5. Outline
    • What is color search?
    • Why we implemented it?
    • Implementation
    • Create color palette
    • Color detection by image processing
    • Result

    View Slide

  6. What is color search?

    View Slide


  7. View Slide

  8. What is color search?
    • Narrow down items by colors
    • e.g. One-piece dress & white
    • e.g. Coat & brown
    • Detect colors automatically
    • Sellers input nothing
    • Color detection from item images

    View Slide

  9. Why we implemented it?

    View Slide

  10. Why we implemented it?
    • Most users choosing clothing colors
    • Questionnaire
    • Interview
    • Search word logs
    • To specialize in fashion
    • Fashion specialized search features

    View Slide

  11. Implementation #1
    Create color palette

    View Slide

  12. Use RGB values


    255 * 255 * 255 = 16,581,375 colors

    View Slide

  13. Create color palette

    12 colors ☺

    View Slide

  14. Why 12 colors?

    View Slide

  15. Why 12 colors?
    • From human color recognition research[1]
    • All languages have basic 11 color terms
    • 11 colors are universal
    • Red, yellow, green, blue, orange, purple, pink,
    brown, gray, black, white
    • From search queries including color names
    • Listed queries in descending order
    • Merged into basic 11 colors (e.g camel is brown)
    • Leaved beige because it can't merge
    [1] Berlin B.and Kay P, “Basic Color Terms:Their Universality and Evolution.”

    View Slide

  16. Implementation #2
    Color detection by image processing

    View Slide

  17. Color detection by image processing
    • Color correction by white balancing
    • Remove background
    • Detect a mode color
    • Choose nearest color palette

    View Slide

  18. Color correction by white balancing

    View Slide

  19. Color correction by white balancing
    • Correct colors of images taken under
    fluorescent lamp
    • Typical fluorescent lights make images greenish
    • Fashion images usually taken inside a room
    • Correct colors with Retinex[2]
    • A method modeled human visual features
    • Do not affect photoshopped images
    [2] E. Land and J. McCann, “Lightness and Retinex theory,”

    View Slide

  20. require 'RMagick'
    image = ImageList.new('image.jpg')
    red_extrema = image.channel_extrema(Magick::RedChannel)
    green_extrema = image.channel_extrema(Magick::GreenChannel)
    blue_extrema = image.channel_extrema(Magick::BlueChannel)
    red_max = red_extrema.last.to_f
    green_max = green_extrema.last.to_f
    blue_max = blue_extrema.last.to_f
    white_balanced_image = image
    white_balanced_image = white_balanced_image.convolve_channel(1,
    [green_max / red_max], Magick::RedChannel)
    white_balanced_image = white_balanced_image.convolve_channel(1,
    [green_max / blue_max], Magick::BlueChannel)

    Retinex using RMagick

    View Slide


  21. Before After
    green white

    View Slide

  22. Remove background

    View Slide

  23. Remove background
    • Eliminate the effect of background colors
    • Make background mask to remove
    • Make background mask
    • Detect edges
    • Flood magenta from corners
    • Extract magenta regions as background mask

    View Slide


  24. Original Detect edges
    Flood magenta Background mask

    View Slide

  25. Detect a mode color

    View Slide

  26. Detect a mode color
    • Detect a mode color from the central portion
    • Fashion items usually appear in center of photo
    • Central portion is less affected by background
    colors
    • Detect a mode color
    • Crop the central portion
    • Reduce colors into 8
    • Composite background mask
    • Make histogram and detect a mode

    View Slide


  27. Original Crop center
    ˍ Reduce colors
    Composite mask
    Make histogram
    & Detect mode
    #A3886B

    View Slide

  28. Choose nearest color palette

    View Slide

  29. Choose nearest color palette
    • RGB to 12 colors
    • 255*255*255ʢ16,581,375ʣ → 12
    • Use color name survey[3]
    • 196,608 RGB values are named
    • We converted to HSL and used the range of each
    color names
    • Choose nearest color palette
    • Conver RGB to HSL
    • Choose nearest palette considering HSL range
    IUUQCMPHYLDEDPNDPMPSTVSWFZSFTVMUT

    View Slide

  30. Me: ”Succeeded!”


    View Slide

  31. A internal user: “It’s strange”


    View Slide

  32. General colors != Fashion colors

    View Slide

  33. Color recognition is different by
    fashion knowledge

    View Slide


  34. 8IBUDPMPSJTUIJT
    HSFFO
    #81D4CB

    View Slide


  35. 5J⒎BOZlCMVFz

    View Slide

  36. Tuned up with
    showing to internal uses


    View Slide


  37. Completed

    View Slide

  38. Result

    View Slide

  39. Accuracy of color detection

    View Slide

  40. Accuracy of color detection
    • Checked 30 items visually in major
    categories
    • Average is 48.36%
    • Well appeared colors are well
    • Black: 90.67%
    • Red: 79.33%
    • Pink: 72.67%
    • White: 56.67%
    • Less appeared colors affected by false
    detection of other colors

    View Slide

  41. Item found proportion
    in search result

    View Slide

  42. Item found ratio in search result
    • We defined item found proportion
    • Percentage of users who opened a item in search
    result
    • Item found
    • Keyword search: about 69%
    • Color search: about 75%
    • Color search become easier to find items
    than keyword search

    View Slide

  43. Summary

    View Slide

  44. Summary
    • We implemented color search
    • Researched user buying behavior and
    implemented fashion specialized search
    • Color palette were created from human
    recognition research and search queries
    • Correcting color, removing background,
    cropping center and detect nearest color
    palette
    • Color search become easier to find items
    than keyword search

    View Slide

  45. Thanks


    View Slide