人間のクリエイティブな生産能力の支援と拡張・Towards Augmenting Creative Processes with Machine Learning

Creative processes, such as illustration, writing, content designing, etc., play an important role in human society and communication. In this talk, I will explain on the possibility of supporting and augmenting creative processes with machine learning, allowing for both faster and higher quality results. A part from more philosophical aspects of creativity, I will discuss network model based approaches and their application to interactive inking of rough sketches.

Transcript

1. 人間のク リ エイ ティ ブな生産能力の支援と 拡張 Towards Augmenting Creative Processes with Machine Learning Edgar SIMO-SERRA 13th of Octobre 2018 Waseda University
2. Self-Introduction • Spanish descent, born in USA • Moved to Barcelona at 11 • July 2015 PhD from BarcelonaTech • August 2015 Waseda University Junior Researcher • April 2017 Waseda University Junior Researcher (Assistant Professor) • April 2018 PRESTO Researcher • September 2018 Waseda University Assistant Professor 2
3. Self-Introduction 2
4. Regarding Pottery Stoneware (陶器) • Porous due to higher sand content • Generally glazed to to make it non-porous • Gritty texture Porcelain (磁器) • Highest quality • Very hard and strong • Very smooth texture • Hardest to make 3
5. Regarding Pottery 1. Molding 1.1 Create and define the basic form 1.2 Wait until partially dry 1.3 Initial modification of the form (荒削り ) 1.4 Wait until fully dry 1.5 Definition of the form (本削り ) 1.6 First firing (~800C, 素焼き ) 2. Glazing 2.1 Cleaning and polishing 2.2 Drawing and glazing 2.3 Main firing (~1260C, 本焼き ) 3. Overglaze 3.1 Overglaze drawing ( う わえ 上絵) 3.2 Overglaze firing (~800C) 3
6. Today’s Talk • Motivation • Sketch Simplification 1.0: Dataset Creation • Sketch Simplification 2.0: Semi-supervised Learning • Sketch Simplification 3.0 aka Inking: Interactive Neural Networks • Reflections 4
7. Motivation
8. Creative Processes? • Professional manga artists are 57.8% full-digital 1 • Professional photographers are 81.7% full digital 2 • Japanese game industry worth 369 billion yen (+12.5% yearly increase) 3 • 13-episode anime costs 250 million yen • Half is for illustration • Mainly illustrated abroad • High costs and low productivity4 1https://mannavi.net/2178/ 2https://www.ntt-ad.co.jp/research_publication/research_ development/report/160525/index.html 3http://gyokai-search.com/3-game.htm 4Emiko Kakiuchi, Kyoshi Takeuchi. Creative industries: Reality and potential in Japan. GRIPS Discussion Papers, 2014. 5
9. Illustration ©David Revoy 6
10. Illustration ©David Revoy 6
12. Sketch Simplification 1.0: Dataset Creation
13. Sketch Simplification Input: Rough Sketch Output: Line Art 8
14. Sketch Simplification Rough Target Rough Target 9
15. Deep Learning • Modern Neural Networks • Computational efficiency with GPU • Large scale datasets 10
16. Deep Learning • Modern Neural Networks • Computational efficiency with GPU • Large scale datasets • Learns input to output mapping 10
17. Fully Convolutional Neural Networks Scale image by stride 1. Flat-convolution 1.1 3 × 3 kernel， 1 × 1 padding， 1 stride 2. Down-convolution 2.1 3 × 3 kernel， 1 × 1 padding， 2 stride 3. Up-convolution 3.1 4 × 4 kernel， 1 × 1 padding， 1/2 stride Down-convolution Flat-convolution Up-convolution stride stride stride 11
18. Basic Sketch Simplification Model • 23 convolutional layers • Output has the same resolution as the input • Encoder-Decoder architecture • Reduces memory usage • Increases spatial resolution Flat-convolution Up-convolution 2 × 2 4 × 4 8 × 8 4 × 4 2 × 2 × × Down-convolution 12
19. Initial Results • Using simple encoder-decoder type model • Create dataset and test 13
20. Initial Results • Using simple encoder-decoder type model • Create dataset and test → blurs! • Analyze training data to see and… 13
21. Inverse Dataset Construction • Data quality is critical • Convert from rough to line and it doesn’t match • Convert back from line to rough and it matches Standard Approach Inverse Approach 14
22. Sketch Dataset • 68 pairs of sketches • Drawn by 5 illustrators • 424 × 424-pixel patches for training ・・・ Extracted patches Sketch dataset ・・・ 15
23. Comparison Input Potrace Adobe Live Trace Ours 16
24. Comparison Input Potrace Adobe Live Trace Ours 16
25. Sketch Simplification 2.0: Semi-supervised Learning
26. Dataset Bias Training pairs Rough sketches “in the wild” 17
27. Dataset Bias Training pairs Rough sketches “in the wild” • Supervised dataset (rough sketch and line drawing pairs): ρx,y • Rough sketch dataset:ρx • Line drawing dataset: ρy 17
28. Generative Adversarial Network (GAN) 18
29. Generative Adversarial Network (GAN) 18
30. Generative Adversarial Network (GAN) 18
31. Generative Adversarial Network (GAN) 18
32. Generative Adversarial Network (GAN) 18
33. Generative Adversarial Network (GAN) 18
34. Generative Adversarial Network (GAN) 18
35. Generative Adversarial Network (GAN) 18
36. Generative Adversarial Network (GAN) 18
37. Generative Adversarial Network (GAN) • D(·): maximize classification prediction max D Ey∗∼ρy Real data log D(y∗) + Ez∼N(0,1) Random log(1 − D(G(z))) 18
38. Generative Adversarial Network (GAN) • D(·): maximize classification prediction • G(·): minimize to fool D(·) min G Ez∼N(0,1) Random log(1 − D(G(z))) 18
39. Generative Adversarial Network (GAN) • D(·): maximize classification prediction • G(·): minimize to fool D(·) • Alternate optimization min G max D Ey∗∼ρy Real data log D(y∗) + Ez∼N(0,1) Random log(1 − D(G(z))) 18
40. Model • S(·): Sketch simplification model • 23 layer fully convolutional neural network [Simo-Serra+ 2016] • D(·): Discriminator model • 6 layer convolutional neural network Flat-convolution Up-convolution 2 × 2 4 × 4 8 × 8 4 × 4 2 × 2 × × Down-convolution 19
41. Proposed framework min S Supervised E(x,y∗)∼ρx , y Standard Loss S(x) − y∗ 2 入力 Standard Loss 20
42. Proposed framework min S max D Supervised E(x,y∗)∼ρx , y    Standard Loss S(x) − y∗ 2 + Adversarial Loss α log D(y∗) + α log(1 − D(S(x)))    入力 Standard Loss +Adversarial 20
43. Proposed framework min S max D Supervised E(x,y∗)∼ρx , y    Standard Loss S(x) − y∗ 2 + Adversarial Loss α log D(y∗) + α log(1 − D(S(x)))    + β Line Ey∼ρy [ log D(y) ] + β Rough Ex∼ρx [ log(1 − D(S(x))) ] Unsupervised Adversarial Loss 入力 Standard Loss +Adversarial +Unsupervised 20
44. Training • Supervised data: standard loss + adversarial loss • Unsupervised data: adversarial loss Supervised Data MSE Adversarial 21
45. Training • Supervised data: standard loss + adversarial loss • Unsupervised data: adversarial loss Line Drawings Rough Sketches Adversarial 21
46. (lack of) Post-processing • MSE loss requires post-processing to avoid blurring • Adversarial loss avoids blurring Input LtS (no PP) LtS (PP) Ours (no PP) 22
47. Sketch Simplification 3.0 aka Inking: Interactive Neural Networks
48. Motivation Input Previous Approach 23
49. Motivation Input Previous Approach User Input Standard Eraser 23
50. Motivation Input Previous Approach User Input Standard Eraser Proposed 23
51. Motivation “1. The inker’s main purpose is to translate the penciller’s graphite pencil lines into reproducible, black, ink lines. 2. The inker must honor the penciller’s original intent while adjusting any obvious mistakes. 3. The inker determines the look of the finished art.” — Gary Martin, The Art of Comic Book Inking [1997] 23
52. Interactive Neural Networks • Feed-forward fully convolutional neural network Input Output Model 24
53. Interactive Neural Networks • Feed-forward fully convolutional neural network • Input rough sketch and user edit are concatenated channel-wise Input Output Model User Edit User + 24
54. Training Framework 1. Line width normalization 2. Simulation of user edits Train User Edit Simulation Smart Inker Training Data Line Normalization 0 0 Dataset 25
55. Training Framework - Line width normalization 1. Line width normalization 2. Simulation of user edits Train User Edit Simulation Smart Inker Training Data Line Normalization 0 0 Dataset 26
56. Training Framework - Line width normalization ベクターデータから生成 学習 26
57. Training Framework - Line width normalization Input No normalization Normalization 26
58. Training Framework - Simulation of user edits 1. Line width normalization 2. Simulation of user edits Train User Edit Simulation Smart Inker Training Data Line Normalization 0 0 Dataset 27
59. Training Framework - Simulation of user edits Input Data Pair Line Drawing Rough Sketch 27
60. Training Framework - Simulation of user edits Input Data Pair Line Drawing Rough Sketch Sampled Regions 27
61. Training Framework - Simulation of user edits Input Data Pair Line Drawing Rough Sketch Sampled Regions Add Edits and Noise 27
63. Results Input Automatic Edit Ours ©Eisaku Kubonouchi 29
64. Results Input Automatic Edit Ours ©Eisaku Kubonouchi 29
65. Limitations Input Edit Ours 30
66. Reflections
67. Reflections • Fundamental Research Approach • Find limitations/problems in existing approaches • Identify potential approaches to solving them • Try approaches knowing that most will fail • Rethink your axioms • Work hard until the end • Research is based on lots of hard work, not genius inspiration • Don’t leave things for the last moment • It is natural to make mistakes, however, it is important to learn from them 31
68. Reflections • Wait what does pottery have to do with research? 31
69. Reflections • Wait what does pottery have to do with research? • Must plan and organize months ahead of time • Focus and dedication are fundamental • Mistakes and failures happen along the way! 31
70. Advice for Success 1. Success is built on mountains of failures. 2. Pick up hobbies while you can. • Help when research goes sour. • Can also give research ideas. 3. Take risks and aim high. 4. Know when to work and when to rest. 5. Be open to new experiences. 32
71. See the World! CVPR2015 Boston ACCV2015 Singapore CVPR2016 Las Vegas ICCV2017 Venecia CGI2018 Bintan SIGGRAPH2018 Vancouver 33
72. Thanks! Questions? https://esslab.jp 34