LINE Creative Lab : Create Catchy Ads with HTML5 Canvas

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Agenda - Introduction of LINE Creative Lab - Project design - Image editor - Video editor - Our next steps

Slide 3

Slide 3 text

Introduction of LINE Creative Lab

Slide 4

Slide 4 text

What is LINE Creative Lab?

Slide 5

Slide 5 text

How LINE Creative Lab was born Ads on LINE services

Slide 6

Slide 6 text

How LINE Creative Lab was born Before June30 2020 Import creative Designer Ads Hire Advertiser

Slide 7

Slide 7 text

How LINE Creative Lab was born From June30 2020 Export creatives Advertiser Create/Edit template

Slide 8

Slide 8 text

LINE Creative Lab’s main features - Image editor - Video editor - Optimize for ads - For LINE users

Slide 9

Slide 9 text

LINE Creative Lab’s main features Image layer Shape layer Logo layer Text layer Image editor

Slide 10

Slide 10 text

LINE Creative Lab’s main features Scene Transition Sticker Video editor

Slide 11

Slide 11 text

LINE Creative Lab’s main features Designed templates Image designed templates Video designed templates

Slide 12

Slide 12 text

Project design

Slide 13

Slide 13 text

Overview Designer Designed template Editor Advertisement Create User Edit Use Render

Slide 14

Slide 14 text

Editor general architecture Tools Preview Template Edit Save Advertisement Render

Slide 15

Slide 15 text

Image editor

Slide 16

Slide 16 text

Content overview What do we have in image editor section? - Image editor architecture - Layers - Technical challenges

Slide 17

Slide 17 text

Image editor architecture Designer Designed template Image editor User template (Image) Create User Edit Use Save Image advertisement Render

Slide 18

Slide 18 text

Layers Image, Logo Upload logo Add logo to image ads

Slide 19

Slide 19 text

Layers Text Choose font Add text to image ads

Slide 20

Slide 20 text

Layers Shape Choose shape Add shape to image ads

Slide 21

Slide 21 text

Technical challenges Content overview Shape Fonts loading

Slide 22

Slide 22 text

Technical challenges Fonts loading > Large size problem Fonts with large size 15 text layers X 16 MB = 240 MB

Slide 23

Slide 23 text

Technical challenges Fonts loading > Solutions Font server 1st Font subsets 2nd

Slide 24

Slide 24 text

Technical challenges Fonts loading > Font server solution Font server Image of text Send data Generate Draw

Slide 25

Slide 25 text

Technical challenges Fonts loading > Solutions Font server 1st Font subsets 2nd

Slide 26

Slide 26 text

Technical challenges Fonts loading > Font subsets solution Split original font to subsets Register each subset by using font-face

Slide 27

Slide 27 text

Technical challenges Fonts loading > Font subsets solution Split original font to subsets 15 text layers X 50 KB = 750 KB

Slide 28

Slide 28 text

Technical challenges Fonts loading > Pros and Cons > Font server solution Cons Pros

Slide 29

Slide 29 text

Technical challenges Fonts loading > Pros and Cons > Font subsets solution Cons Pros

Slide 30

Slide 30 text

Technical challenges Fonts loading > Conclusion Font server 1st Font subsets 2nd

Slide 31

Slide 31 text

Technical challenges Content overview Shape Fonts loading

Slide 32

Slide 32 text

Technical challenges Shape

Slide 33

Slide 33 text

Technical challenges Shape SVG Mathematical formulas Browser

Slide 34

Slide 34 text

Technical challenges Shape > Problem with SVG SVG shape SVG shape with stroke Wrong shape

Slide 35

Slide 35 text

Technical challenges Shape > Path2D solution Path APIs Path2D Shape Use Path2D – Browser compatibility Draw Canvas

Slide 36

Slide 36 text

Technical challenges Shape > Path2D solution SVG Path2D Canvas Convert to Draw SVG shape with large stroke

Slide 37

Slide 37 text

Video editor

Slide 38

Slide 38 text

Content overview What do we have in video editor section? - Video editor architecture - Layers - Video rendering - Technical challenges

Slide 39

Slide 39 text

Video editor architecture Designer Designed template Video editor User template (Video) Create User Edit Use Save Send to Browser Video advertisement Render Encoding server Video editor architecture Designer Designed template Video editor User template (Video) Create User Edit Use Save Send to Browser Video advertisement Render Encoding server

Slide 40

Slide 40 text

Layers Image, Logo Upload image Add image to video scene

Slide 41

Slide 41 text

Layers Text Text setting Add text to video scene

Slide 42

Slide 42 text

Layers Sticker Choose sticker Add sticker to video scene

Slide 43

Slide 43 text

Video rendering Draw and Encode methods - Canvas: 2d context - Canvas: WebGL context Drawing - MediaRecorder API - FFMPEG - Hardware encoder Encoding

Slide 44

Slide 44 text

Video rendering Drawing > WebGL overview Application WebGL APIs GPU Call Communicate

Slide 45

Slide 45 text

Video rendering Drawing > WebGL overview Application GPU Upload x N Download x N Application GPU Upload large data 1st slow case 2nd slow case

Slide 46

Slide 46 text

Video rendering Drawing > Apply WebGL in CLAB - Text changes are expensive because we have to convert text to image - So text changes should be reduced Our situation Two properties of WebGL - Upload all resource to GPU memory before drawing - All resources must be images

Slide 47

Slide 47 text

Video rendering Encoding > MediaRecorder API Canvas MediaRecorder Video captureStream Record MediaRecorder - Browser compatibility

Slide 48

Slide 48 text

Video rendering Encoding > FFMPEG Raw frames FFMPEG Video Load Encode

Slide 49

Slide 49 text

Video rendering Encoding > Hardware encoder Node.js server N-API binding Low-level SDK (C++) HW encoder

Slide 50

Slide 50 text

Technical challenges Content overview

Slide 51

Slide 51 text

Technical challenges Encoding speed > MediaRecorder API 30s MediaStream 30s Video Play Wait 30s Switch tab = Postpone Close tab = Cancel MediaRecorder

Slide 52

Slide 52 text

Technical challenges Encoding speed > Server-side rendering Node.js Browser JS drawing codebase node-canvas headless-gl Canvas WebGL

Slide 53

Slide 53 text

Technical challenges Encoding speed > FFMPEG encoding flow Video frames GPU memory SSD or RAM SW encoder HW encoder H264 video Or FFMPEG Encode Read Copy video frames from GPU memory to SSD/RAM is a bottleneck and make the encoding process slower Copy Save Node.js server Draw 6 GB 6 GB

Slide 54

Slide 54 text

Technical challenges Encoding speed > HW encoder & low-level SDK Memory area for drawing Memory area for encoding GPU memory HW encoder Video frames H264 video Node.js server Draw Encode Copy uncompressed data by low-level SDK

Slide 55

Slide 55 text

Technical challenges MediaRecorder 30s Node.js + SDK + HW 2s Node.js + FFMPEG 7s Encoding speed > How long does it take to create a 30 seconds video?

Slide 56

Slide 56 text

Technical challenges Encoding speed > Conclusion MediaRecorder 30s Node.js + SDK + HW 2s Node.js + FFMPEG 7s We selected this solution for our users

Slide 57

Slide 57 text

Technical challenges Content overview

Slide 58

Slide 58 text

Technical challenges Difference of text: Client rendering vs Server rendering (MacOS) server Difference

Slide 59

Slide 59 text

Technical challenges Difference of text: Client rendering vs Server rendering (MacOS) (Linux) Difference Headless

Slide 60

Slide 60 text

Technical challenges Content overview

Slide 61

Slide 61 text

Technical challenges Support new effects - Video effects = GLSL code - gl-transition = open source - New effects = increase dev time

Slide 62

Slide 62 text

Our next steps

Slide 63

Slide 63 text

Our next steps In the future, we will - Apply more GLSL effects - Support video layer - Solve difference of text issue - Scale encoding server

Slide 64

Slide 64 text

Thank you