Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search fashion items by colors

shoby
March 22, 2016

Search fashion items by colors

shoby

March 22, 2016
Tweet

More Decks by shoby

Other Decks in Technology

Transcript

  1. Search fashion items by colors
    Fablic, inc.
    Shota Izaka

    View full-size slide

  2. Shota Izaka
    • Fablic, inc.
    • Developing Fril
    • iOS / Ruby on Rails / Image Processing
    • Search by colors / Apple Watch App / Swift
    migration

    !TIPCZTIPCZ

    View full-size slide

  3. What is Fril?
    • Fashion market place app
    • Similar to Poshmark
    • Buy and sell fashion items
    • Fril mediates trading

    View full-size slide

  4. Outline
    • What is color search?
    • Why we implemented it?
    • Implementation
    • Create color palette
    • Color detection by image processing
    • Result

    View full-size slide

  5. What is color search?

    View full-size slide

  6. What is color search?
    • Narrow down items by colors
    • e.g. One-piece dress & white
    • e.g. Coat & brown
    • Detect colors automatically
    • Sellers input nothing
    • Color detection from item images

    View full-size slide

  7. Why we implemented it?

    View full-size slide

  8. Why we implemented it?
    • Most users choosing clothing colors
    • Questionnaire
    • Interview
    • Search word logs
    • To specialize in fashion
    • Fashion specialized search features

    View full-size slide

  9. Implementation #1
    Create color palette

    View full-size slide

  10. Use RGB values


    255 * 255 * 255 = 16,581,375 colors

    View full-size slide

  11. Create color palette

    12 colors ☺

    View full-size slide

  12. Why 12 colors?

    View full-size slide

  13. Why 12 colors?
    • From human color recognition research[1]
    • All languages have basic 11 color terms
    • 11 colors are universal
    • Red, yellow, green, blue, orange, purple, pink,
    brown, gray, black, white
    • From search queries including color names
    • Listed queries in descending order
    • Merged into basic 11 colors (e.g camel is brown)
    • Leaved beige because it can't merge
    [1] Berlin B.and Kay P, “Basic Color Terms:Their Universality and Evolution.”

    View full-size slide

  14. Implementation #2
    Color detection by image processing

    View full-size slide

  15. Color detection by image processing
    • Color correction by white balancing
    • Remove background
    • Detect a mode color
    • Choose nearest color palette

    View full-size slide

  16. Color correction by white balancing

    View full-size slide

  17. Color correction by white balancing
    • Correct colors of images taken under
    fluorescent lamp
    • Typical fluorescent lights make images greenish
    • Fashion images usually taken inside a room
    • Correct colors with Retinex[2]
    • A method modeled human visual features
    • Do not affect photoshopped images
    [2] E. Land and J. McCann, “Lightness and Retinex theory,”

    View full-size slide

  18. require 'RMagick'
    image = ImageList.new('image.jpg')
    red_extrema = image.channel_extrema(Magick::RedChannel)
    green_extrema = image.channel_extrema(Magick::GreenChannel)
    blue_extrema = image.channel_extrema(Magick::BlueChannel)
    red_max = red_extrema.last.to_f
    green_max = green_extrema.last.to_f
    blue_max = blue_extrema.last.to_f
    white_balanced_image = image
    white_balanced_image = white_balanced_image.convolve_channel(1,
    [green_max / red_max], Magick::RedChannel)
    white_balanced_image = white_balanced_image.convolve_channel(1,
    [green_max / blue_max], Magick::BlueChannel)

    Retinex using RMagick

    View full-size slide


  19. Before After
    green white

    View full-size slide

  20. Remove background

    View full-size slide

  21. Remove background
    • Eliminate the effect of background colors
    • Make background mask to remove
    • Make background mask
    • Detect edges
    • Flood magenta from corners
    • Extract magenta regions as background mask

    View full-size slide


  22. Original Detect edges
    Flood magenta Background mask

    View full-size slide

  23. Detect a mode color

    View full-size slide

  24. Detect a mode color
    • Detect a mode color from the central portion
    • Fashion items usually appear in center of photo
    • Central portion is less affected by background
    colors
    • Detect a mode color
    • Crop the central portion
    • Reduce colors into 8
    • Composite background mask
    • Make histogram and detect a mode

    View full-size slide


  25. Original Crop center
    ˍ Reduce colors
    Composite mask
    Make histogram
    & Detect mode
    #A3886B

    View full-size slide

  26. Choose nearest color palette

    View full-size slide

  27. Choose nearest color palette
    • RGB to 12 colors
    • 255*255*255ʢ16,581,375ʣ → 12
    • Use color name survey[3]
    • 196,608 RGB values are named
    • We converted to HSL and used the range of each
    color names
    • Choose nearest color palette
    • Conver RGB to HSL
    • Choose nearest palette considering HSL range
    IUUQCMPHYLDEDPNDPMPSTVSWFZSFTVMUT

    View full-size slide

  28. Me: ”Succeeded!”


    View full-size slide

  29. A internal user: “It’s strange”


    View full-size slide

  30. General colors != Fashion colors

    View full-size slide

  31. Color recognition is different by
    fashion knowledge

    View full-size slide


  32. 8IBUDPMPSJTUIJT
    HSFFO
    #81D4CB

    View full-size slide


  33. 5J⒎BOZlCMVFz

    View full-size slide

  34. Tuned up with
    showing to internal uses


    View full-size slide

  35. Accuracy of color detection

    View full-size slide

  36. Accuracy of color detection
    • Checked 30 items visually in major
    categories
    • Average is 48.36%
    • Well appeared colors are well
    • Black: 90.67%
    • Red: 79.33%
    • Pink: 72.67%
    • White: 56.67%
    • Less appeared colors affected by false
    detection of other colors

    View full-size slide

  37. Item found proportion
    in search result

    View full-size slide

  38. Item found ratio in search result
    • We defined item found proportion
    • Percentage of users who opened a item in search
    result
    • Item found
    • Keyword search: about 69%
    • Color search: about 75%
    • Color search become easier to find items
    than keyword search

    View full-size slide

  39. Summary
    • We implemented color search
    • Researched user buying behavior and
    implemented fashion specialized search
    • Color palette were created from human
    recognition research and search queries
    • Correcting color, removing background,
    cropping center and detect nearest color
    palette
    • Color search become easier to find items
    than keyword search

    View full-size slide