[CVPR24 Oral] Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation

This is a slide of the following research paper from the University of Tokyo and CyberAgent AI Lab in CVPR 2024.

Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation
Daichi Horita, Naoto Inoue, Kotaro Kikuchi, Kota Yamaguchi, Kiyoharu Aizawa
CVPR, 2024.

Project Website: https://udonda.github.io/RALF/
GitHub Repository: https://github.com/CyberAgentAILab/RALF
Paper: https://arxiv.org/abs/2311.13602

Abstract
Content-aware graphic layout generation aims to automatically arrange visual elements along with a given content, such as an e-commerce product image. In this paper, we argue that the current layout generation approaches suffer from the limited training data for the high-dimensional layout structure. We show that a simple retrieval augmentation can significantly improve the generation quality. Our model, which is named Retrieval-Augmented Layout Transformer (RALF), retrieves nearest neighbor layout examples based on an input image and feeds these results into an autoregressive generator. Our model can apply retrieval augmentation to various controllable generation tasks and yield high-quality layouts within a unified architecture. Our extensive experiments show that RALF successfully generates content-aware layouts in both constrained and unconstrained settings and significantly outperforms the baselines.

Transcript

1. Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation Daichi Horita Naoto Inoue Kotaro Kikuchi Kota Yamaguchi Kiyoharu Aizawa
2. Concept 2 Content-aware layout generation
3. RALF 3 Retrieval-Augmented Layout Transformer 1) Retrieve nearest neighbor layouts based on the input image 2) use them as a reference to augment the generation process.
4. 4 Challenges •Data Scarcity & Training Efficiency •Content-Layout Harmonization •Controllability to User-Specified Constraints
5. 5 Challenges Data Scarcity & Training Efficiency Web Authoring tool
6. 6 Challenges Content-Layout Harmonization -PHP -PHP 5FYU 6OEFSMBZ 5FYU 5FYU .JTBMJHONFOU 0WFSMBQ
7. 7 Challenges Controllability to User-Specified Constraints Category → Size +Position “Logo, Text x2, Underlay” Relationship “Logo top on Text”
8. • Data Scarcity & Training Efficiency •Retrieval augmentation effectively addresses the data scarcity problem • Content-Layout Harmonization •Propose RALF • Controllability to User-Specified Constraints •Show RALF outperforms the baselines on unconditional & conditional tasks 8 Contributions
9. 9 Why Retrieve & Generate? Just retrieve GT Input Top1 Output
10. 10 Why Retrieve & Generate? Retrieve & Generate GT Input Retrieve Output A
11. 11 Preliminaries Representation of layout Z = (bos, c1 , x1 , y1 , w1 , h1 , c2 , x2 , y2 , w2 , h2 , . . . , eos) Sorted by raster scan order ᶃ ᶄ ᶅ ᶆ ᶃ ᶄ
12. • Quantize bbox : • Autoregressive modeling: bi 12 Preliminaries Tokenization of layout B bins I: image, S: saliency map In exp, B = 128
13. 13 Overview of RALF Propose a Retrieval-Augmented Layout Transformer (RALF)
14. 14 Overview of RALF Encodes an input canvas image and a saliency map
15. 15 Overview of RALF Retrieves nearest neighbor layout examples based on the similarity of an input image.
16. 16 Overview of RALF Fuses the features of retrieved layouts with the image feature using cross-attention.
17. 17 Overview of RALF Incorporates user-specified constraints following LayoutFormer++ [Jiang+ CVPR23], which tokenizes constraints.
18. 18 Overview of RALF Autoregressively generates a layout.
19. • A challenge lies in the absence of joint embedding for image–layout retrieval, unlike CLIP for image—text retrieval. Layout Retrieval 19 image—text retrieval Dog image—layout retrieval No joint embedding! joint embedding (CLIP)
20. • We hypothesize that given an image–layout pair ( ), is more likely to be useful when is similar to . ˜ I, ˜ L ˜ L ˜ I I Layout Retrieval 20 GT layout Image (query) I (˜ I1 , ˜ L1 ) (˜ I2 , ˜ L2 ) Retrieve images using similarity, then use paired layout ˜ I ˜ L
21. Layout Retrieval 21
22. Layout Retrieval 22
23. Layout Retrieval 23 Content-layout harmonization DreamSim Random better Layout fidelity
24. 24 Analysis How effective RALF?
25. 25 Analysis How effective RALF? Effective regardless of the training dataset size.
26. 26 Analysis How effective RALF? Not sensitive to the number of retrieved layouts.
27. 27 Analysis How different K affects the output? Compare K=1 with K=16 ɹɹɹSimilar results Output 1 Output 2 Retrieved example(s) ! = 1 ge Output 1 Output 2 Retrieved example(s) ! = 1 Reference Output K=1
28. ! = 16 ! = 1 ! = 16 ! = 1 28 Analysis How different K affects the output? Compare K=1 with K=16 ɹɹɹDiverse and plausible results. Reference K=16 Output
29. 29 Analysis Limitation of FID. K=1 K=16 Same FID 🤔
30. 30 Qualitative Results PKU Dataset Logo Text Underlay CGL Dataset Embellishment Logo Text Underlay ! LayoutDM CGL-GAN DS-GAN Autoreg Baseline RALF (Ours)
31. 31 Quantitative Results Unconstrained generation results on PKU dataset [Hsu+ CVPR23] Train:Test:Val = 7,734:1,000:1,000 Content: an overlap of saliency object and layout
32. 32 Quantitative Results Real data is supposed to be upper bound Validation data
33. 33 Quantitative Results Just top-1 retrieval is the worst in content metrics “Retrieval-augmented” generation is important Top-1 retrieved layout
34. 34 Quantitative Results RALF significantly outperforms the baselines Baseline methods
35. 35 Quantitative Results Baseline methods + Retrieval augmentation
36. 36 Quantitative Results Out-of-domain generalization e.g. Train / DB: CGL dataset, Test: PKU dataset
37. 37 Quantitative Results Category → Size + Position Relationship Constrained generation
38. 38 Conclusion Thank you! Contact: horita@hal.t.u-tokyo.ac.jp @udoooom • Retrieval augmentation effectively addresses the data scarcity problem. • Propose RALF: Retrieval-augmented Layout Transformer • Retrieval augmentation + Autoregressive Transformer. • Show that RALF successfully generates high-quality layouts, significantly outperforming baselines.