グラフィックデザインへのCV技術の応用@関東CV勉強会

20190414 関東CV勉強会で発表した際のスライドです。

Transcript

1.    
   CV
       
2.       D2 
    Vision & Language Computational Graphic Design ←New! 2 ktrk15
3. 38DH,#+0"-/ 6=5&.*/!0+ ­ @ : %($ 1<?C  )0&'$ B  ­  J 2 
   4LKEF
   EF 2
   >B  ­   ; 
   A7  9  
    IG M  3
4. )% *  4 )% *E@: graphic designF
   - 6D , A>:5=1
   C?  0<
   84(+.B9; 2/  *     
       !# " )" $#+73 WASEDA INNOVATION LEADING LAB ࣮ମ৘ใֶ  ࣮ମ৘ใֶത࢜ϓϩάϥϜͰ͸ɼػցͱ৘ใ ͷ༥߹ΛςʔϚʹଟ෼໺ͷֶੜ͕ͦΕͧΕͷ ྖҬΛԣஅͯ͠ݚڀɾٞ࿦Λߦͳ͍ͬͯ·͢ɽ ࠓճ͸ɼෳ਺ͷମݧܕలࣔΛ௨ͯ͠ࢲͨͪͷ ߟ͑Δ࣮ମ৘ใֶͷੈքΛ͓ݟ͍ͤͨ͠·͢ɽ ձ৔ɿ߸ؗ  ໌࣏௨Γ ੢ૣҴాӺ ࢸ
   ৽॓ ਖ਼໳ தఉ IUUQXXXMFBEJOHTOXBTFEBBDKQ ࣮ମ৘ใֶϦʔσΟϯάϓϩάϥϜ͸ɺ༏ लͳֶੜΛඋ͑޿͘࢈ֶ׭ʹΘͨΓάϩʔ όϧʹ׆༂͢ΔϦʔμʔ΁ͱಋͨ͘Ίɺઐ ໳෼໺ͷ࿮Λ௒͑ͯത࢜՝ఔલظɾޙظҰ ؏ֶͨ͠ҐϓϩάϥϜΛߏஙɾల։͢Δจ ෦ՊֶলͷࣄۀͰ͢ɻ8BTFEB
   *OOPWBUJPO
   -FBEJOH
   -BC ͸࣮ମ৘ใֶϦʔσΟϯάϓ ϩάϥϜͷֶੜஂମͰ͢ɻ
   Communication Robot ಀ͛ͨΓসͬͨΓɼਓؒͷ ༷ʑͳಈ͖΍ද৘ʹ൓Ԡ͢ ΔϩϘοτɽ࣮ࡍʹΠϯλ RoboCup ϩϘοτͷࡇయ3PCP$VQʹ ग़৔͢ΔͨΊ੡࡞͍ͯ͠Δ ϩϘοτɽࠓճͷڝٕͰ͋ ΔαοΧʔΛߦ͏ͨΊʹ༷ Drone ࠷ۙΑࣖ͘ʹ͢Δυϩʔϯɽ ࣮ࡍʹ໨ʹͨ͜͠ͱ͸͋Γ ·͔͢ʁυϩʔϯΛ࢖ͬͨ গ͠ಛघͳϝσΟΞදݱΛ ϥΫγϣϯΛͱָͬͯ͠Μ ͰΈ͍ͯͩ͘͞ɽ ʑͳػೳΛ౥ࡌ͍ͯ͠·͢ɽ ָ͓͠Έ͍ͩ͘͞ɽ ࣮ମ৘ใֶϦʔσΟϯά ϓϩάϥϜͰͷ׆ಈΛɼ ϙελʔలࣔͱͱ΋ʹ ͝঺հ͠·͢ɽ Poster &! '+ Wikipedia
5. 5 άϥϑΟοΫσβΠϯͱ͸ : graphic design   
       !# " 
    
    WASEDA INNOVATION LEADING LAB ࣮ମ৘ใֶ  ࣮ମ৘ใֶത࢜ϓϩάϥϜͰ͸ɼػցͱ৘ใ ͷ༥߹ΛςʔϚʹଟ෼໺ͷֶੜ͕ͦΕͧΕͷ ྖҬΛԣஅͯ͠ݚڀɾٞ࿦Λߦͳ͍ͬͯ·͢ɽ ࠓճ͸ɼෳ਺ͷମݧܕలࣔΛ௨ͯ͠ࢲͨͪͷ ߟ͑Δ࣮ମ৘ใֶͷੈքΛ͓ݟ͍ͤͨ͠·͢ɽ ձ৔ɿ߸ؗ  ໌࣏௨Γ ੢ૣҴాӺ ࢸ
   ৽॓ ਖ਼໳ தఉ IUUQXXXMFBEJOHTOXBTFEBBDKQ ࣮ମ৘ใֶϦʔσΟϯάϓϩάϥϜ͸ɺ༏ लͳֶੜΛඋ͑޿͘࢈ֶ׭ʹΘͨΓάϩʔ όϧʹ׆༂͢ΔϦʔμʔ΁ͱಋͨ͘Ίɺઐ ໳෼໺ͷ࿮Λ௒͑ͯത࢜՝ఔલظɾޙظҰ ؏ֶͨ͠ҐϓϩάϥϜΛߏஙɾల։͢Δจ ෦ՊֶলͷࣄۀͰ͢ɻ8BTFEB
   *OOPWBUJPO
   -FBEJOH
   -BC ͸࣮ମ৘ใֶϦʔσΟϯάϓ ϩάϥϜͷֶੜஂମͰ͢ɻ
   Communication Robot ಀ͛ͨΓসͬͨΓɼਓؒͷ ༷ʑͳಈ͖΍ද৘ʹ൓Ԡ͢ ΔϩϘοτɽ࣮ࡍʹΠϯλ RoboCup ϩϘοτͷࡇయ3PCP$VQʹ ग़৔͢ΔͨΊ੡࡞͍ͯ͠Δ ϩϘοτɽࠓճͷڝٕͰ͋ ΔαοΧʔΛߦ͏ͨΊʹ༷ Drone ࠷ۙΑࣖ͘ʹ͢Δυϩʔϯɽ ࣮ࡍʹ໨ʹͨ͜͠ͱ͸͋Γ ·͔͢ʁυϩʔϯΛ࢖ͬͨ গ͠ಛघͳϝσΟΞදݱΛ ϥΫγϣϯΛͱָͬͯ͠Μ ͰΈ͍ͯͩ͘͞ɽ ʑͳػೳΛ౥ࡌ͍ͯ͠·͢ɽ ָ͓͠Έ͍ͩ͘͞ɽ ࣮ମ৘ใֶϦʔσΟϯά ϓϩάϥϜͰͷ׆ಈΛɼ ϙελʔలࣔͱͱ΋ʹ ͝঺հ͠·͢ɽ Poster 
    ϙελʔ Wikipedia
6. 
     6 https://paper-leaf.com/blog/2012/10/principles-of-design-quick-reference-poster/
7. )% *  7 )% *E@: graphic designF
   - 6D , A>:5=1
   C?  0<
   84(+.B9; 2/  *     
       !# " )" $#+73 WASEDA INNOVATION LEADING LAB ࣮ମ৘ใֶ  ࣮ମ৘ใֶത࢜ϓϩάϥϜͰ͸ɼػցͱ৘ใ ͷ༥߹ΛςʔϚʹଟ෼໺ͷֶੜ͕ͦΕͧΕͷ ྖҬΛԣஅͯ͠ݚڀɾٞ࿦Λߦͳ͍ͬͯ·͢ɽ ࠓճ͸ɼෳ਺ͷମݧܕలࣔΛ௨ͯ͠ࢲͨͪͷ ߟ͑Δ࣮ମ৘ใֶͷੈքΛ͓ݟ͍ͤͨ͠·͢ɽ ձ৔ɿ߸ؗ  ໌࣏௨Γ ੢ૣҴాӺ ࢸ
   ৽॓ ਖ਼໳ தఉ IUUQXXXMFBEJOHTOXBTFEBBDKQ ࣮ମ৘ใֶϦʔσΟϯάϓϩάϥϜ͸ɺ༏ लͳֶੜΛඋ͑޿͘࢈ֶ׭ʹΘͨΓάϩʔ όϧʹ׆༂͢ΔϦʔμʔ΁ͱಋͨ͘Ίɺઐ ໳෼໺ͷ࿮Λ௒͑ͯത࢜՝ఔલظɾޙظҰ ؏ֶͨ͠ҐϓϩάϥϜΛߏஙɾల։͢Δจ ෦ՊֶলͷࣄۀͰ͢ɻ8BTFEB
   *OOPWBUJPO
   -FBEJOH
   -BC ͸࣮ମ৘ใֶϦʔσΟϯάϓ ϩάϥϜͷֶੜஂମͰ͢ɻ
   Communication Robot ಀ͛ͨΓসͬͨΓɼਓؒͷ ༷ʑͳಈ͖΍ද৘ʹ൓Ԡ͢ ΔϩϘοτɽ࣮ࡍʹΠϯλ RoboCup ϩϘοτͷࡇయ3PCP$VQʹ ग़৔͢ΔͨΊ੡࡞͍ͯ͠Δ ϩϘοτɽࠓճͷڝٕͰ͋ ΔαοΧʔΛߦ͏ͨΊʹ༷ Drone ࠷ۙΑࣖ͘ʹ͢Δυϩʔϯɽ ࣮ࡍʹ໨ʹͨ͜͠ͱ͸͋Γ ·͔͢ʁυϩʔϯΛ࢖ͬͨ গ͠ಛघͳϝσΟΞදݱΛ ϥΫγϣϯΛͱָͬͯ͠Μ ͰΈ͍ͯͩ͘͞ɽ ʑͳػೳΛ౥ࡌ͍ͯ͠·͢ɽ ָ͓͠Έ͍ͩ͘͞ɽ ࣮ମ৘ใֶϦʔσΟϯά ϓϩάϥϜͰͷ׆ಈΛɼ ϙελʔలࣔͱͱ΋ʹ ͝঺հ͠·͢ɽ Poster &! '+ Wikipedia
8.  /' )% *  5 )% *E@: graphic designF
   - 6D , A>:5=1
   C?  0<
   84(+.B9; 2/  *     
       !# " )" $#+73 WASEDA INNOVATION LEADING LAB ࣮ମ৘ใֶ  ࣮ମ৘ใֶത࢜ϓϩάϥϜͰ͸ɼػցͱ৘ใ ͷ༥߹ΛςʔϚʹଟ෼໺ͷֶੜ͕ͦΕͧΕͷ ྖҬΛԣஅͯ͠ݚڀɾٞ࿦Λߦͳ͍ͬͯ·͢ɽ ࠓճ͸ɼෳ਺ͷମݧܕలࣔΛ௨ͯ͠ࢲͨͪͷ ߟ͑Δ࣮ମ৘ใֶͷੈքΛ͓ݟ͍ͤͨ͠·͢ɽ ձ৔ɿ߸ؗ  ໌࣏௨Γ ੢ૣҴాӺ ࢸ
   ৽॓ ਖ਼໳ தఉ IUUQXXXMFBEJOHTOXBTFEBBDKQ ࣮ମ৘ใֶϦʔσΟϯάϓϩάϥϜ͸ɺ༏ लͳֶੜΛඋ͑޿͘࢈ֶ׭ʹΘͨΓάϩʔ όϧʹ׆༂͢ΔϦʔμʔ΁ͱಋͨ͘Ίɺઐ ໳෼໺ͷ࿮Λ௒͑ͯത࢜՝ఔલظɾޙظҰ ؏ֶͨ͠ҐϓϩάϥϜΛߏஙɾల։͢Δจ ෦ՊֶলͷࣄۀͰ͢ɻ8BTFEB
   *OOPWBUJPO
   -FBEJOH
   -BC ͸࣮ମ৘ใֶϦʔσΟϯάϓ ϩάϥϜͷֶੜஂମͰ͢ɻ
   Communication Robot ಀ͛ͨΓসͬͨΓɼਓؒͷ ༷ʑͳಈ͖΍ද৘ʹ൓Ԡ͢ ΔϩϘοτɽ࣮ࡍʹΠϯλ RoboCup ϩϘοτͷࡇయ3PCP$VQʹ ग़৔͢ΔͨΊ੡࡞͍ͯ͠Δ ϩϘοτɽࠓճͷڝٕͰ͋ ΔαοΧʔΛߦ͏ͨΊʹ༷ Drone ࠷ۙΑࣖ͘ʹ͢Δυϩʔϯɽ ࣮ࡍʹ໨ʹͨ͜͠ͱ͸͋Γ ·͔͢ʁυϩʔϯΛ࢖ͬͨ গ͠ಛघͳϝσΟΞදݱΛ ϥΫγϣϯΛͱָͬͯ͠Μ ͰΈ͍ͯͩ͘͞ɽ ʑͳػೳΛ౥ࡌ͍ͯ͠·͢ɽ ָ͓͠Έ͍ͩ͘͞ɽ ࣮ମ৘ใֶϦʔσΟϯά ϓϩάϥϜͰͷ׆ಈΛɼ ϙελʔలࣔͱͱ΋ʹ ͝঺հ͠·͢ɽ Poster &! '+ Wikipedia 8  &!  .$ #1 ­ !*-%
   ­ ")
   ­ 0+ ­   ­   ­ ,( …
9. Computational Graphic Design 9  %   -" / 0 
   1 ,*$  ­ '( # + ­  ).&
   !
10.  B>67
    D =0"(<' #5967 ­ A;C*"2<C*(<' ­ %,&0 ­ ;8! $3"4- #5967 ­ ! ) !?. ­ :3/1 + ;8@9 10
11. Learning Visual Importance for Graphic Designs and Data Visualizations [UIST17] 11 0*#/"41.3$ #/"5 +! 1.3$ #/ " 
    FCN%- )&
    Graphic Design (2  3$
    Data Visualization /,  ' 
12. 12 a https://www.youtube.com/watch?v=13YZPndRJdE   
     Learning Visual Importance for Graphic Designs and Data Visualizations [UIST17]
13. Task-driven Webpage Saliency [ECCV18] 13   
        
14. Task-driven Webpage Saliency [ECCV18] 14 Semantic layout  
        !
15.  " '0>Cute, Vintage ? /%- .DNN&)1< '09+=$!$ 5(38
    75,0002 $5( $ 46 15 What Characterizes Personalities of Graphic Designs? [SIGGRAPH18] ;1#,:*>'0 )1?
16. What Characterizes Personalities of Graphic Designs? [SIGGRAPH18]   
     16
17. What Characterizes Personalities of Graphic Designs? [SIGGRAPH18] 17  
          
18. What Characterizes Personalities of Graphic Designs? [SIGGRAPH18] 18 ← Element-level design suggestion ↑ Example-based personality transfer Personality scoring net.
       
19. Rico: A Mobile App Dataset [UIST17],[UIST18] 19   
    
20. Learning to Extract Semantic Structure from Documents Using Multimodal FCNN [CVPR17] Document$ Semantic segmentation -/%!2 " Text Embedding Map 3 )&1(
    -/%!2OCR.0  
    20 ,#+4*' 5
21. Modeling Fonts in Context: Font Prediction on Web Designs [PG18] 21  #  & $ 
    Web  )
    HTML
      "!  ( &'%*+
22. Modeling Fonts in Context: Font Prediction on Web Designs [PG18] 22 #5.Autoencoder$0
    " 3 1+!L2+GAN 
    '2(  " 3 /64) &-
    *,%78
23. Modeling Fonts in Context: Font Prediction on Web Designs [PG18] 23 Web 
        
       
24. Learning Layouts for Single-Page Graphic Designs [TVCG14] 24 !%   ' $# → &(" alignment ) * 
     ­ alignment ­ balance ­ white space ­ overlap …
25. 25 "   $ ! → #%
    & ' Learning Layouts for Single-Page Graphic Designs [TVCG14]
26. 26 https://www.youtube.com/watch?v=cHb11WG23K8    
     DesignScape: Design with Interactive Layout Suggestions [CHI15]
27. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators [ICLR19]  
      " GAN  #%$ Wireframe Renderer 27 Generator Encoder Stacked Relation Module Real / Fake Prediction fc fc fc fc fc fc fc Decoder CNN Differentiable Wireframe Rendering Wireframe Layout f(p1 ,x1 ) f(p2 ,x2 ) …. f(pi ,xi ) …. f(pn ,xn ) f(p1 ,x1 ) f(p2 ,x2 ) …. f(pi ,xi ) …. f(pn ,xn ) f’(p1 ,x1 ) f'(p2 ,x2 ) …. f'(pi ,xi ) …. f'(pn ,xn ) Self-attention ➕ Wireframe Rendering Discriminator (Graphic -> Visual Domain) Class Probabilities Geometric Parameters {(p1 , θ 1 ), … ,(pN , θ N )} p1 θ 1 p2 θ 2 • • • pN θ N Random Input Class Probabilities Geometric Parameters {(p1 ’, θ 1 ’), … ,(pN ’, θ N ’)} p1 ’ θ 1 ’ p2 ’ θ 2 ’ • • • pN ’ θ N ’ Class Probabilities Geometric Parameters {(p1 ’, θ 1 ’), … ,(pN ’, θ N ’)} p1 ’ θ 1 ’ p2 ’ θ 2 ’ • • • pN ’ θ N ’ !"& '
28. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators [ICLR19] 28    Semantic Layouts Real Pages Paragra ph Heading Figure Table List Caption 
      
29. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators [ICLR19] 29 Wireframe rendering    
     DCGAN Paragraph Heading Figure Table List Caption Real data Relation-based
30. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators [ICLR19] 30 Girl Sun Tree Boy Glasses Hat    
     Wireframe rendering Real data Relation-based
31.  B>67
    D =0"(<' #5967 ­ A;C*"2<C*(<' ­ %,&0 ­ ;8! $3"4- #5967 ­ ! ) !?. ­ :3/1 + ;8@9 31
32. *) :-#9"12
    + CV3,7;0 ­ FCNs, Siamese network, GAN,'54.#9"3 8!6  ($   & 9/% 32
33.   # $%
     ' & "! Hand-crafted *+ )( , → Data-driven  33
34. 97 Data-driven $+!I ­ ,->G 6$ * (D/)  0 C$ *H2BGANA&)+" :;K ­ 6$ *=<A@8
    5C$ *F  $+!.'(%#3 E?J41 34
35.  "('%!&# ) × CV>?@- ­ B9*1A0>?
    ,:*<7>? &+$.D;E53  35 4/ 2G  F=&+$8  C
    6  HH