LINE Creative Lab : Create Catchy Ads with HTML5 Canvas

Cong Nguyen
LINE Vietnam / FE Dev 1 / Frontend Engineer

https://linedevday.linecorp.com/2021/ja/sessions/103
https://linedevday.linecorp.com/2021/en/sessions/103
https://linedevday.linecorp.com/2021/ko/sessions/103

Transcript

2. Agenda - Introduction of LINE Creative Lab - Project design - Image editor - Video editor - Our next steps
3. Introduction of LINE Creative Lab
4. What is LINE Creative Lab?
5. How LINE Creative Lab was born Ads on LINE services
6. How LINE Creative Lab was born Before June30 2020 Import creative Designer Ads Hire Advertiser
7. How LINE Creative Lab was born From June30 2020 Export creatives Advertiser Create/Edit template
8. LINE Creative Lab’s main features - Image editor - Video editor - Optimize for ads - For LINE users
9. LINE Creative Lab’s main features Image layer Shape layer Logo layer Text layer Image editor
10. LINE Creative Lab’s main features Scene Transition Sticker Video editor
11. LINE Creative Lab’s main features Designed templates Image designed templates Video designed templates
12. Project design
13. Overview Designer Designed template Editor Advertisement Create User Edit Use Render
14. Editor general architecture Tools Preview Template Edit Save Advertisement Render
15. Image editor
16. Content overview What do we have in image editor section? - Image editor architecture - Layers - Technical challenges
17. Image editor architecture Designer Designed template Image editor User template (Image) Create User Edit Use Save Image advertisement Render
18. Layers Image, Logo Upload logo Add logo to image ads
19. Layers Text Choose font Add text to image ads
20. Layers Shape Choose shape Add shape to image ads
21. Technical challenges Content overview Shape Fonts loading
22. Technical challenges Fonts loading > Large size problem Fonts with large size 15 text layers X 16 MB = 240 MB
23. Technical challenges Fonts loading > Solutions Font server 1st Font subsets 2nd
24. Technical challenges Fonts loading > Font server solution Font server Image of text Send data Generate Draw
25. Technical challenges Fonts loading > Solutions Font server 1st Font subsets 2nd
26. Technical challenges Fonts loading > Font subsets solution Split original font to subsets Register each subset by using font-face
27. Technical challenges Fonts loading > Font subsets solution Split original font to subsets 15 text layers X 50 KB = 750 KB
28. Technical challenges Fonts loading > Pros and Cons > Font server solution Cons Pros
29. Technical challenges Fonts loading > Pros and Cons > Font subsets solution Cons Pros
30. Technical challenges Fonts loading > Conclusion Font server 1st Font subsets 2nd
31. Technical challenges Content overview Shape Fonts loading
32. Technical challenges Shape
33. Technical challenges Shape SVG Mathematical formulas Browser
34. Technical challenges Shape > Problem with SVG SVG shape SVG shape with stroke Wrong shape
35. Technical challenges Shape > Path2D solution Path APIs Path2D Shape Use Path2D – Browser compatibility Draw Canvas
36. Technical challenges Shape > Path2D solution SVG Path2D Canvas Convert to Draw SVG shape with large stroke
37. Video editor
38. Content overview What do we have in video editor section? - Video editor architecture - Layers - Video rendering - Technical challenges
39. Video editor architecture Designer Designed template Video editor User template (Video) Create User Edit Use Save Send to Browser Video advertisement Render Encoding server Video editor architecture Designer Designed template Video editor User template (Video) Create User Edit Use Save Send to Browser Video advertisement Render Encoding server
40. Layers Image, Logo Upload image Add image to video scene
41. Layers Text Text setting Add text to video scene
42. Layers Sticker Choose sticker Add sticker to video scene
43. Video rendering Draw and Encode methods - Canvas: 2d context - Canvas: WebGL context Drawing - MediaRecorder API - FFMPEG - Hardware encoder Encoding
44. Video rendering Drawing > WebGL overview Application WebGL APIs GPU Call Communicate
45. Video rendering Drawing > WebGL overview Application GPU Upload x N Download x N Application GPU Upload large data 1st slow case 2nd slow case
46. Video rendering Drawing > Apply WebGL in CLAB - Text changes are expensive because we have to convert text to image - So text changes should be reduced Our situation Two properties of WebGL - Upload all resource to GPU memory before drawing - All resources must be images
47. Video rendering Encoding > MediaRecorder API Canvas MediaRecorder Video captureStream Record MediaRecorder - Browser compatibility
48. Video rendering Encoding > FFMPEG Raw frames FFMPEG Video Load Encode
49. Video rendering Encoding > Hardware encoder Node.js server N-API binding Low-level SDK (C++) HW encoder
50. Technical challenges Content overview
51. Technical challenges Encoding speed > MediaRecorder API 30s MediaStream 30s Video Play Wait 30s Switch tab = Postpone Close tab = Cancel MediaRecorder
52. Technical challenges Encoding speed > Server-side rendering Node.js Browser JS drawing codebase node-canvas headless-gl Canvas WebGL
53. Technical challenges Encoding speed > FFMPEG encoding flow Video frames GPU memory SSD or RAM SW encoder HW encoder H264 video Or FFMPEG Encode Read Copy video frames from GPU memory to SSD/RAM is a bottleneck and make the encoding process slower Copy Save Node.js server Draw 6 GB 6 GB
54. Technical challenges Encoding speed > HW encoder & low-level SDK Memory area for drawing Memory area for encoding GPU memory HW encoder Video frames H264 video Node.js server Draw Encode Copy uncompressed data by low-level SDK
55. Technical challenges MediaRecorder 30s Node.js + SDK + HW 2s Node.js + FFMPEG 7s Encoding speed > How long does it take to create a 30 seconds video?
56. Technical challenges Encoding speed > Conclusion MediaRecorder 30s Node.js + SDK + HW 2s Node.js + FFMPEG 7s We selected this solution for our users
57. Technical challenges Content overview
58. Technical challenges Difference of text: Client rendering vs Server rendering (MacOS) server Difference
59. Technical challenges Difference of text: Client rendering vs Server rendering (MacOS) (Linux) Difference Headless
60. Technical challenges Content overview
61. Technical challenges Support new effects - Video effects = GLSL code - gl-transition = open source - New effects = increase dev time
62. Our next steps
63. Our next steps In the future, we will - Apply more GLSL effects - Support video layer - Solve difference of text issue - Scale encoding server
64. Thank you