PicCell : Flooding your service with AI

Joonsik Baek
LINE Plus LPE4 MediaPlatform Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/F1-6

Transcript

1. 2019 DevDay
   PicCell : Flooding Your Service
   With AI
   > Joonsik Baek
   > LINE Plus LPE4 MediaPlatform Software Engineer
   

   View Slide

2. Flood Your Service With AI
   2
   

   View Slide

3. Flood Your Service With AI
   3
   

   View Slide

4. View Slide

5. 5
   ?
   

   View Slide

6. 6
   !
   

   View Slide

7. 7
   > Why and How PicCell Was Made
   > Case Studies
   > Roadmap
   

   View Slide

8. 8
   Why PicCell Was Made
   

   View Slide

9. 9
   Why PicCell Was Made
   

   View Slide

10. 10
    Low ROI
    Why PicCell Was Made
    

    View Slide

11. 11
    ???
    Why PicCell Was Made
    

    View Slide

12. 12
    ???
    Why PicCell Was Made
    

    View Slide

13. 13
    Why PicCell Was Made
    

    View Slide

14. 14
    Why PicCell Was Made
    

    View Slide

15. 15
    More Than

    40 Services
    Why PicCell Was Made
    

    View Slide

16. 16
    Developed by ???
    

    View Slide

17. 17
    Media
    Platform
    Who Is Qualified
    

    View Slide

18. 18
    Media
    Platform
    Who Is Qualified
    

    View Slide

19. 19
    Media
    Platform
    Who Is Qualified
    - More Than 80 Services
    

    View Slide

20. 20
    Media
    Platform
    OBSObject Storage
    Who Is Qualified
    - More Than 80 Services
    

    View Slide

21. 21
    Media
    Platform
    Video Image Audio File
    Who Is Qualified
    OBSObject Storage
    - More Than 80 Services
    

    View Slide

22. 22
    Media
    Platform
    Video Image Audio File
    Storage Delivery
    Processing
    Who Is Qualified
    OBSObject Storage
    - More Than 80 Services
    

    View Slide

23. 23
    Media
    Platform
    Video Image Audio File
    Storage Delivery
    Processing
    Who Is Qualified
    OBSObject Storage
    - More Than 80 Services
    - MAX 1.5 Tbps Traffic
    - 100 PB Storage
    

    View Slide

24. 24
    Media
    Platform
    Video Image Audio File
    Storage Delivery
    Processing
    Who Is Qualified
    OBSObject Storage
    - More Than 80 Services
    - MAX 1.5 Tbps Traffic
    - 100 PB Storage
    

    View Slide

25. 25
    Media
    Platform
    Video Image Audio File
    Storage Delivery
    Processing
    Who Is Qualified
    OBSObject Storage
    - More Than 80 Services
    - MAX 1.5 Tbps Traffic
    - 100 PB Storage
    

    View Slide

26. 26
    ? ? ?
    

    View Slide

27. 27
    ? ?
    Adult

    Content

    Filter
    ?
    

    View Slide

28. ResNet 50

    Based
    28
    Adult Content Filter
    

    View Slide

29. 3 Classes

    (Adult, Sexy, Normal)
    ResNet 50

    Based
    29
    Adult Content Filter
    

    View Slide

30. 3.5M

    Training Samples
    ResNet 50

    Based
    30
    Adult Content Filter
    3 Classes

    (Adult, Sexy, Normal)
    

    View Slide

31. > 0.9 F1 Score
    3.5M

    Training Samples
    ResNet 50

    Based
    31
    Adult Content Filter
    3 Classes

    (Adult, Sexy, Normal)
    1.1M Test Set
    

    View Slide

32. 300ms

    Processing Time
    > 0.9 F1 Score
    3.5M

    Training Samples
    ResNet 50

    Based
    32
    Adult Content Filter
    3 Classes

    (Adult, Sexy, Normal)
    1.1M Test Set
    

    View Slide

33. 33
    18
    Adult Content Filter
    

    View Slide

34. 34
    18
    "
    "
    "
    ""
    "
    Adult Content Filter
    

    View Slide

35. View Slide

36. View Slide

37. 37
    18
    Sync
    Async
    Gateway
    

    View Slide

38. 38
    18
    Sync
    Async
    Gateway Pacman
    

    View Slide

39. 39
    18
    Sync
    Async
    Gateway Pacman
    Realtime Processing
    

    View Slide

40. 40
    Req. #1
    Req. #2
    Req. #3
    Callback. #1
    Callback. #3
    Guaranteed Callback
    Callback. #2
    

    View Slide

41. 41
    Req. #1
    Req. #2
    Req. #3
    Callback. #1
    Callback. #2
    Callback. #3
    Guaranteed Callback
    

    View Slide

42. 42
    Req. #1
    Req. #2
    Req. #3
    Callback. #1
    Callback. #2
    Callback. #3
    Guaranteed Callback
    #
    

    View Slide

43. 43
    Consumer
    Publisher Message Queue
    Guaranteed Callback
    

    View Slide

44. 44
    Consumer
    Publisher Message Queue
    Ack
    Guaranteed Callback
    

    View Slide

45. 45
    Consumer
    Publisher Message Queue
    Nack
    Guaranteed Callback
    

    View Slide

46. 46
    Consumer
    Publisher Message Queue
    Dead-Letter Queue

    With TTL
    Nack
    Guaranteed Callback
    

    View Slide

47. 47
    18
    Sync
    Async
    Gateway Pacman
    DLQ
    Spiderman
    $
    Guaranteed

    Callback
    

    View Slide

48. 48
    Result

    Message Queue
    Dead-Letter Queue

    With TTL
    Nack
    Guaranteed Callback
    Pacman Spiderman
    $ Guaranteed

    Callback
    

    View Slide

49. 49
    Expensive Duplicated Processing
    

    View Slide

50. 50
    Expensive Duplicated Processing
    

    View Slide

51. 51
    GPU
    Expensive Duplicated Processing
    

    View Slide

52. 52
    %
    =
    GPU
    Expensive Duplicated Processing
    

    View Slide

53. 53
    18
    Sync
    Async
    Gateway
    Pacman
    DLQ
    Spiderman
    $
    Guaranteed

    Callback
    

    View Slide

54. 54
    Gateway
    DLQ
    Search Result
    Write Request
    Process
    Write Result
    Req. For
    Deliver Result
    

    View Slide

55. 55
    Gateway
    DLQ
    Search Result
    Write Request
    …
    Search Result
    Process
    Write Result
    Req. For
    Req. For
    …
    Deliver Result
    Deliver Result
    

    View Slide

56. 56
    Image
    Adult Content Filter for Video
    

    View Slide

57. 57
    Image
    Video
    Adult Content Filter for Video
    

    View Slide

58. 58
    ???
    Image
    Video
    Adult Content Filter for Video
    

    View Slide

59. 59
    Adult Content Filter for Video
    

    View Slide

60. 60
    JPG JPG JPG
    Adult Content Filter for Video
    

    View Slide

61. 61
    JPG JPG JPG
    {;} {;} {;} {;}
    Adult Content Filter for Video
    

    View Slide

62. 62
    Timberman
    

    View Slide

63. 63
    18
    Sync
    Async
    Gateway Pacman
    DLQ
    Spiderman
    $
    Timberman
    Guaranteed

    Callback
    

    View Slide

64. 64
    Image
    Video
    Adult Content Filter for Video
    

    View Slide

65. 65
    The Most Pleasant LINE Ever
    

    View Slide

66. 66
    Illegal Advertisements in Timeline
    

    View Slide

67. 67
    Illegal Advertisement
    Adult Content
    Illegal Advertisements in Timeline
    

    View Slide

68. 68
    Illegal Advertisement
    Adult Content
    Illegal Advertisements in Timeline
    

    View Slide

69. 69
    ?
    Adult

    Content

    Filter
    Illegal

    Ad.
    Detection
    ?
    

    View Slide

70. 70
    Illegal

    Ad.
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    

    View Slide

71. 71
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    Illegal

    Ad.
    

    View Slide

72. 72
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    Illegal

    Ad.
    

    View Slide

73. 73
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    Illegal

    Ad.
    

    View Slide

74. 74
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    ✅ Product Classification
    - Sunglasses, Shoes, Watch or Handbag
    Illegal

    Ad.
    

    View Slide

75. 75
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    ✅ Product Classification
    - Sunglasses, Shoes, Watch or Handbag
    Illegal

    Ad.
    

    View Slide

76. 76
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    ✅ Product Classification
    - Sunglasses, Shoes, Watch or Handbag
    ✅ QR Code Reader
    - URL Extraction
    Illegal

    Ad.
    

    View Slide

77. 77
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    ✅ Product Classification
    - Sunglasses, Shoes, Watch or Handbag
    ✅ QR Code Reader
    - URL Extraction
    ✅ OCR (Optical Character Recognition)
    Illegal

    Ad.
    

    View Slide

78. 78
    Illegal Advertisements in Timeline
    Imitation Product
    QR Code
    Advertising Text on Image
    ✅ Product Classification
    - Sunglasses, Shoes, Watch or Handbag
    ✅ QR Code Reader
    - URL Extraction
    ✅ OCR (Optical Character Recognition)
    - JP / KR / EN / TH / TW / CN / ID
    Illegal

    Ad.
    

    View Slide

79. 79
    18
    Sync
    Async
    Gateway Pacman
    DLQ
    Spiderman
    $
    Timberman
    '
    (
    ⌚
    *
    Guaranteed

    Callback
    

    View Slide

80. 80
    Illegal Ad Free
    

    View Slide

81. 81
    Trial and Error via LINE LIVE
    

    View Slide

82. 18
    82
    Trial and Error via LINE LIVE
    

    View Slide

83. 18
    83
    Trial and Error via LINE LIVE
    - Gun
    - Knife
    - Cigarette
    - Electric Cigarette
    - Sensitive Symbols
    

    View Slide

84. 84
    Trial and Error via LINE LIVE
    - Gun
    - Knife
    - Cigarette
    - Electric Cigarette
    - Sensitive Symbols
    +
    18
    

    View Slide

85. 85
    18 ??
    Trial and Error via LINE LIVE
    +
    - Gun
    - Knife
    - Cigarette
    - Electric Cigarette
    - Sensitive Symbols
    

    View Slide

86. 86
    18 ??
    Trial and Error via LINE LIVE
    +
    - Gun
    - Knife
    - Cigarette
    - Electric Cigarette
    - Sensitive Symbols
    F1 Score
    0.9 0.3
    

    View Slide

87. 87
    18
    Service Specific Training
    

    View Slide

88. 88
    18
    Service Specific Training
    

    View Slide

89. 89
    JPG
    JPG
    JPG
    JPG
    JSON
    JSON
    JSON
    JSON
    Service Specific Training
    

    View Slide

90. 90
    JPG
    JPG
    JPG
    JPG
    JSON
    JSON
    JSON
    JSON
    - User Consent
    - Disciplined
    Service Specific Training
    

    View Slide

91. 91
    JPG
    JPG
    JPG
    JPG
    JSON
    JSON
    JSON
    JSON
    Content
    Deep-Learning
    Engineer
    Service Specific Training
    Training Sample

    Gateway
    Legal Review
    +
    

    View Slide

92. 92
    JPG
    JPG
    JPG
    JPG
    JSON
    JSON
    JSON
    JSON
    Content
    DL Engineer
    Access-Token
    Service Specific Training
    Training Sample

    Gateway
    

    View Slide

93. 93
    Librarian
    

    View Slide

94. 94
    18
    Sync
    Async '
    (
    ⌚
    *
    Gateway
    Spiderman
    $
    Timberman
    DLQ
    +
    Pacman
    Librarian
    Access

    Token
    

    View Slide

95. F1 Score
    0.3 0.84
    40,000

    Training Samples
    3,000
    Training Samples
    95
    Service Specific Training
    

    View Slide

96. Back-Office Support
    Adult Content Filter
    Object Detection
    …
    OCR - JP/EN/CN/TW/ID/KR
    AD Detection
    96
    …
    

    View Slide

97. Back-Office Support
    Adult Content Filter
    Object Detection
    15M / Day
    …
    OCR - JP/EN/CN/TW/ID/KR
    AD Detection
    97
    …
    

    View Slide

98. Back-Office Support
    Adult Content Filter
    Object Detection
    15M / Day
    1.3M / Day
    15M / Day
    3.5M / Day
    …
    OCR - JP/EN/CN/TW/ID/KR
    AD Detection
    98
    …
    

    View Slide

99. 99
    0
    80000
    160000
    240000
    320000
    400000
    2017 2018 2019
    Daily Monitored Images by Operators
    Monitoring Cost Reduction
    

    View Slide

100. 100
     0
     80000
     160000
     240000
     320000
     400000
     2017 2018 2019
     Daily Monitored Images by Operators
     Monitoring Cost Reduction
     

     View Slide

101. Case Studies
     101
     

     View Slide

102. 102
     Closing the Distance
     

     View Slide

103. 103
     CLOSING
     THE DISTANCE
     LINEΛ௨ͯ͡ɺੈքதͷਓͱਓɺ
     ਓͱ৘ใɾαʔϏεͱͷڑ཭ΛॖΊ·͢
     LINE
     Closing the Distance
     

     View Slide

104. 104
     Translation Infra.
     X
     Closing the Distance
     

     View Slide

105. 105
     Closing the Distance
     ,
     

     View Slide

106. 106
     Closing the Distance
     ,
     - . / 0 1 2
     1
     —

     6
     

     View Slide

107. 107
     Closing the Distance
     ,
     2
     1
     0
     .
     -
     1
     -
     

     View Slide

108. 108
     Closing the Distance
     ,
     2
     1
     0
     .
     -
     1
     1
     —

     2
     1
     —

     3
     1
     —

     2
     -
     

     View Slide

109. 109
     Closing the Distance
     

     View Slide

110. 110
     Closing the Distance
     

     View Slide

111. 111
     Closing the Distance
     

     View Slide

112. 112
     Closing the Distance
     

     View Slide

113. 113
     Closing the Distance
     

     View Slide

114. 114
     18
     Sync
     Async '
     (
     ⌚
     *
     Gateway
     Librarian Spiderman
     $
     Timberman
     DLQ
     +
     Pacman
     Access

     Token
     

     View Slide

115. 115
     Expansion Into Finance
     LINE FINANCE
     

     View Slide

116. 116
     Expansion Into Finance
     KYC
     3
     LINE FINANCE
     

     View Slide

117. 117
     eKYC
     KYC
     3 !
     Expansion Into Finance
     LINE FINANCE
     

     View Slide

118. 118
     eKYC
     KYC
     3 !
     Expansion Into Finance
     444

     444
     LINE FINANCE
     

     View Slide

119. Financial

     Solution
     119
     ?
     Adult

     Content

     Filter
     Illegal

     Ad.
     Detection
     

     View Slide

120. 120
     Expansion Into Finance
     LINE FINANCE
     eKYC
     

     View Slide

121. ID card
     Recognition
     121
     - Jane Doe
     Expansion Into Finance
     LINE FINANCE
     eKYC
     

     View Slide

122. ID card
     Recognition
     122
     - Jane Doe
     Expansion Into Finance
     LINE FINANCE
     eKYC
     

     View Slide

123. ID card
     Recognition
     123
     - Jane Doe
     Expansion Into Finance
     LINE FINANCE
     eKYC
     

     View Slide

124. ID card
     Recognition
     Face
     Verification
     124
     - Jane Doe
     5
     +
     Expansion Into Finance
     LINE FINANCE
     eKYC
     

     View Slide

125. ID card
     Recognition
     Face
     Verification
     125
     - Jane Doe
     5
     +
     Expansion Into Finance
     x 0.5 Speed
     LINE FINANCE
     eKYC
     

     View Slide

126. ID card
     Recognition
     Face
     Verification
     Secured Infra. (JP)
     126
     - Jane Doe
     5
     +
     Expansion Into Finance
     LINE FINANCE
     eKYC
     

     View Slide

127. ID card
     Recognition
     Face
     Verification
     Secured Infra. (JP)
     127
     - Jane Doe
     5
     +
     Expansion Into Finance
     LINE FINANCE
     eKYC
     

     View Slide

128. Roadmap
     128
     

     View Slide

129. Deep-Learning Training Sample Portal
     129
     Training Sample Portal
     

     View Slide

130. Deep-Learning Training Sample Portal
     130
     JPG
     JPG
     JPG
     JPG
     JSON
     JSON
     JSON
     JSON
     Normal

     Librarian
     Deep-Learning
     Engineers
     

     View Slide

131. Deep-Learning Training Sample Portal
     131
     JPG
     JPG
     JPG
     JPG
     JSON
     JSON
     JSON
     JSON
     Normal

     Librarian
     Deep-Learning
     Engineers
     Refine Training Sample With Correction API
     

     Adult
     Correction API
     

     View Slide

132. 132
     Only Corrected Samples
     Only Samples From JP
     Training Sample Portal
     Deep-Learning Training Sample Portal
     Librarian
     

     View Slide

133. 133
     Only Corrected Samples
     Only Samples From JP
     Samples

     From

     JP
     Corrected

     Samples
     Training Sample Portal
     Deep-Learning Training Sample Portal
     Librarian
     

     View Slide

134. Cloud Vision API
     Cognitive Service
     134
     

     View Slide

135. Cloud Vision API
     Cognitive Service
     135
     

     View Slide

136. Cloud Vision API
     Cognitive Service
     136
     

     View Slide

137. Cloud Vision API
     Cloud Vision API
     Cognitive Service
     137
     

     View Slide

138. View Slide

139. View Slide

140. Thank You for Your Attention
     140
     

     View Slide