Learning to Restore Deteriorated Line Drawings

We propose a fully automatic approach to restore aged old line drawings. We decompose the task into two subtasks: the line extraction subtask, which aims to extract line fragments and remove the paper texture background, and the restoration subtask, which fills in possible gaps and deterioration of the lines to produce a clean line drawing. Our approach is based on a convolutional neural network that consists of two sub-networks corresponding to the two subtasks. They are trained as part of a single framework in an end-to-end fashion. We also introduce a new dataset consisting of manually annotated sketches by Leonardo da Vinci which, in combination with a synthetic data generation approach, allows training the network to restore deteriorated line drawings. We evaluate our method on challenging 500-year-old sketches and compare with existing approaches with a user study, in which it is found that our approach is preferred 72.7% of the time.

Transcript

1. Learning to Restore Deteriorated Line Drawings Kazuma Sasaki, Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa 13 June 2018 Waseda University 2
2. Motivation 3
3. Motivation 3
4. Contributions • Model for joint line segmentation extraction and line drawing restoration • Manually annotated dataset based on sketches by Leonardo da Vinci • In-depth evaluation of our approach including a user study 4
5. Related Work • Natural Image Image Restoration • Total variation [Osher+ 2005, Rudin+ 1992] • BM3D [Dabov+ 2007] • Dictionary Learning [Dong+ 2014, Gu+ 2015] • Inpainting [Barnes+ 2009] • Depend on rich image gradients 5
6. Related Work • Natural Image Image Restoration • Total variation [Osher+ 2005, Rudin+ 1992] • BM3D [Dabov+ 2007] • Dictionary Learning [Dong+ 2014, Gu+ 2015] • Inpainting [Barnes+ 2009] • Depend on rich image gradients • Line Drawings • Contour Matching [Han+ 1994] • Sketch Simplification [Favreau+ 2016, Simo-Serra+ 2016] • Line Inpainting [Sasaki+ 2017] 5
7. Approach
8. Model Overview Line Extraction Network Restoration Network ⊕ 2 × 2 4 × 4 8 × 8 4 × 4 2 × 2 × × × Input Output Model output Model output 6
9. Model • x: Input grayscale image • E(·): Line extraction network • R(·, ·): Restoration network • Full model: F(x) = max(0 , min(1 , E(x) + R(x , E(x)))) Input x E(x) R(x , E(x)) F(x) 7
10. Data
11. Training Data • Use three different datasets: 1. Leonardo da Vinci Dataset 2. Random synthetic data 3. “Hard” synthetic data 8
12. Leonardo da Vinci Dataset • Manually annotated 500 year old line drawings by Leonardo da Vinci • 61 images for training • 10 images for testing 9
13. Generating Synthetic Data • Training data is synthesized from vector data • Paper texture and aging is simulated • Two-stage generation allows using separate losses for the line extraction and restoration networks ∗ Input ∗ ① Chip the lines ② Add white noise ③ Lower intensity ④ Synthesize texture 10
14. Texture Synthesis • Old paper textures used as base • Scaled and cropped • Multiplied with the deteriorated line drawing 11
15. Synthetic Data • Random synthetic data • “Hard” synthetic data • Parallel lines • Multiple intersections 12
16. Training Flow Line Extraction Network Restoration Network MSE Loss Input: () ∗ MSE Loss ⊕ () Target: ∗ Training flow (Synthetic data) 13
17. Training Flow Line Extraction Network Restoration Network Input: () MSE Loss ⊕ () Target: ∗ Training flow (Leonard da Vinci’s data) 13
18. Results
19. Results Input [Sasaki+ 2017] Ours 14
20. User Study Input Baseline Ours 15
21. User Study • Baseline consists of [Sasaki+ 2017] retrained on the new dataset. • Evaluation done on test set • 10 users participated in the study Baseline Ours vs Baseline - 27.3 vs Ours 72.7 - 15
22. Additional Results 16
23. Vectorization Post-Processing Input Output Post-Processing [Favreau+ 2016] 17
24. Failure Case Input Output 18
25. Conclusions http://hi.cs.waseda.ac.jp/~iizuka/projects/line_restoration/ • Model for line extraction and restoration of line drawings • Approach for generating synthetic line drawings • Proposed approach outperforms existing methods • Leonardo da Vinci Dataset 19