Search fashion items by colors

Slide 1

Slide 1 text

Search fashion items by colors Fablic, inc. Shota Izaka ⛄

Slide 2

Slide 2 text

Shota Izaka • Fablic, inc. • Developing Fril • iOS / Ruby on Rails / Image Processing • Search by colors / Apple Watch App / Swift migration !TIPCZTIPCZ

Slide 3

Slide 3 text

What is Fril? • Fashion market place app • Similar to Poshmark • Buy and sell fashion items • Fril mediates trading

Slide 4

Slide 4 text

Outline

Slide 5

Slide 5 text

Outline • What is color search? • Why we implemented it? • Implementation • Create color palette • Color detection by image processing • Result

Slide 6

Slide 6 text

What is color search?

Slide 7

Slide 7 text

Slide 8

Slide 8 text

What is color search? • Narrow down items by colors • e.g. One-piece dress & white • e.g. Coat & brown • Detect colors automatically • Sellers input nothing • Color detection from item images

Slide 9

Slide 9 text

Why we implemented it?

Slide 10

Slide 10 text

Why we implemented it? • Most users choosing clothing colors • Questionnaire • Interview • Search word logs • To specialize in fashion • Fashion specialized search features

Slide 11

Slide 11 text

Implementation #1 Create color palette

Slide 12

Slide 12 text

Use RGB values 255 * 255 * 255 = 16,581,375 colors

Slide 13

Slide 13 text

Create color palette 12 colors ☺

Slide 14

Slide 14 text

Why 12 colors?

Slide 15

Slide 15 text

Why 12 colors? • From human color recognition research[1] • All languages have basic 11 color terms • 11 colors are universal • Red, yellow, green, blue, orange, purple, pink, brown, gray, black, white • From search queries including color names • Listed queries in descending order • Merged into basic 11 colors (e.g camel is brown) • Leaved beige because it can't merge [1] Berlin B.and Kay P, “Basic Color Terms:Their Universality and Evolution.”

Slide 16

Slide 16 text

Implementation #2 Color detection by image processing

Slide 17

Slide 17 text

Color detection by image processing • Color correction by white balancing • Remove background • Detect a mode color • Choose nearest color palette

Slide 18

Slide 18 text

Color correction by white balancing

Slide 19

Slide 19 text

Color correction by white balancing • Correct colors of images taken under fluorescent lamp • Typical ﬂuorescent lights make images greenish • Fashion images usually taken inside a room • Correct colors with Retinex[2] • A method modeled human visual features • Do not affect photoshopped images [2] E. Land and J. McCann, “Lightness and Retinex theory,”

Slide 20

Slide 20 text

require 'RMagick' image = ImageList.new('image.jpg') red_extrema = image.channel_extrema(Magick::RedChannel) green_extrema = image.channel_extrema(Magick::GreenChannel) blue_extrema = image.channel_extrema(Magick::BlueChannel) red_max = red_extrema.last.to_f green_max = green_extrema.last.to_f blue_max = blue_extrema.last.to_f white_balanced_image = image white_balanced_image = white_balanced_image.convolve_channel(1, [green_max / red_max], Magick::RedChannel) white_balanced_image = white_balanced_image.convolve_channel(1, [green_max / blue_max], Magick::BlueChannel) Retinex using RMagick

Slide 21

Slide 21 text

Before After green white

Slide 22

Slide 22 text

Remove background

Slide 23

Slide 23 text

Remove background • Eliminate the effect of background colors • Make background mask to remove • Make background mask • Detect edges • Flood magenta from corners • Extract magenta regions as background mask

Slide 24

Slide 24 text

Original Detect edges Flood magenta Background mask

Slide 25

Slide 25 text

Detect a mode color

Slide 26

Slide 26 text

Detect a mode color • Detect a mode color from the central portion • Fashion items usually appear in center of photo • Central portion is less affected by background colors • Detect a mode color • Crop the central portion • Reduce colors into 8 • Composite background mask • Make histogram and detect a mode

Slide 27

Slide 27 text

Original Crop center ˍ Reduce colors Composite mask Make histogram & Detect mode #A3886B

Slide 28

Slide 28 text

Choose nearest color palette

Slide 29

Slide 29 text

Choose nearest color palette • RGB to 12 colors • 255*255*255ʢ16,581,375ʣ → 12 • Use color name survey[3] • 196,608 RGB values are named • We converted to HSL and used the range of each color names • Choose nearest color palette • Conver RGB to HSL • Choose nearest palette considering HSL range IUUQCMPHYLDEDPNDPMPSTVSWFZSFTVMUT

Slide 30

Slide 30 text

Me: ”Succeeded!”

Slide 31

Slide 31 text

A internal user: “It’s strange”

Slide 32

Slide 32 text

General colors != Fashion colors

Slide 33

Slide 33 text

Color recognition is different by fashion knowledge

Slide 34

Slide 34 text

8IBUDPMPSJTUIJT HSFFO #81D4CB

Slide 35

Slide 35 text

5J⒎BOZlCMVFz

Slide 36

Slide 36 text

Tuned up with showing to internal uses

Slide 37

Slide 37 text

Completed

Slide 38

Slide 38 text

Result

Slide 39

Slide 39 text

Accuracy of color detection

Slide 40

Slide 40 text

Accuracy of color detection • Checked 30 items visually in major categories • Average is 48.36% • Well appeared colors are well • Black: 90.67% • Red: 79.33% • Pink: 72.67% • White: 56.67% • Less appeared colors affected by false detection of other colors

Slide 41

Slide 41 text

Item found proportion in search result

Slide 42

Slide 42 text

Item found ratio in search result • We deﬁned item found proportion • Percentage of users who opened a item in search result • Item found • Keyword search: about 69% • Color search: about 75% • Color search become easier to ﬁnd items than keyword search

Slide 43

Slide 43 text

Summary

Slide 44

Slide 44 text

Summary • We implemented color search • Researched user buying behavior and implemented fashion specialized search • Color palette were created from human recognition research and search queries • Correcting color, removing background, cropping center and detect nearest color palette • Color search become easier to ﬁnd items than keyword search

Slide 45

Slide 45 text

Thanks ⛄