Editable Stickers - Server side rendering in near Real-time

Alex Pedini
LINE Shop TF - Team A Senior Server-Side Engineer
https://linedevday.linecorp.com/2020/ja/sessions/3290
https://linedevday.linecorp.com/2020/en/sessions/3290

Transcript

2. Custom Stickers Product Release 2019/H1 A Personalized set of stickers › Choose one word to make the sticker set your own › e.g. your Name › Anytime you want, change the text on the whole sticker set › Product Story: https://engineering.linecorp.com/ ja/blog/product-story-01-custom- sticker/
3. Custom Stickers Product Release 2019/H1
4. Custom Stickers User Feedback › Unnecessary to change the text on all stickers at the same time › Users want to change the text more easily during the conversation › Users want to add more than a name or single word to the sticker à The personalized content becomes more directly part of the conversation
5. Message Stickers Product Release 2020/H1 › Up to 100 characters per message › Allows for multiple lines of text › Text edit flow integrated into the chat room › Text content is fully part of the conversation › Check out the video! https://www.youtube.com/watch?v=YtP1Xr4SWiY
6. Custom/Message Stickers Sticker Composition Base Image: Downloaded from CDN Text Overlay: Rendered on demand Layered sticker
7. Client-side vs Server-side Rendering Client-side Scenario Client fetches sticker style from server, Then renders locally
8. Client-side vs Server-side Rendering Client-side Scenario Client fetches sticker style from server, Then renders locally PRO › Faster and more scalable as each client renders it’s own sticker
9. Client-side vs Server-side Rendering Client-side Scenario Client fetches sticker style from server, Then renders locally PRO CON › Faster and more scalable as each client renders it’s own sticker › Each client should have all the fonts, or use webfonts on demand › Develop and test on multiple platforms/devices › More complex app release process › support multiple versions of client › Inconsistent quality
10. Client-side vs Server-side Rendering Client-side Server-side Scenario Client fetches sticker style from server, Then renders locally Client sends text content to the server, Server returns PNG PRO CON › Faster and more scalable as each client renders it’s own sticker › Rendering engine is independent from the device › Easier to deploy new feature/fixes/fonts › Each client should have all the fonts, or use webfonts on demand › Develop and test on multiple platforms/devices › More complex app release process › support multiple versions of client › Inconsistent quality
11. Client-side vs Server-side Rendering Client-side Server-side Scenario Client fetches sticker style from server, Then renders locally Client sends text content to the server, Server returns PNG PRO CON › Faster and more scalable as each client renders it’s own sticker › Rendering engine is independent from the device › Easier to deploy new feature/fixes/fonts › Each client should have all the fonts, or use webfonts on demand › Develop and test on multiple platforms/devices › More complex app release process › support multiple versions of client › Inconsistent quality
12. Client-side vs Server-side Rendering Client-side Server-side Scenario Client fetches sticker style from server, Then renders locally Client sends text content to the server, Server returns PNG PRO CON › Faster and more scalable as each client renders it’s own sticker › Rendering engine is independent from the device › Easier to deploy new feature/fixes/fonts › Each client should have all the fonts, or use webfonts on demand › Develop and test on multiple platforms/devices › More complex app release process › support multiple versions of client › Inconsistent quality › Server Capacity planning is difficult › Scale out/in operations › Some limitations in rendering libraries
13. Custom/Message Stickers Base Image: Downloaded from CDN By the client Text Overlay: Rendered on demand By The Server Layered sticker On the client Sticker Composition
14. Message Sticker Secure Rendering › The text content of the rendered image is part of the context of a secure conversation › By context we mean for example who is the sender and receiver/group › The text content should be shared between clients securely › The rendering server doesn’t need the context › Rendering server is user and conversation agnostic › Each client fetches the rendered overlay for itself, opening an encryption session with the server
15. Message Sticker Secure rendering
16. Message Sticker Secure rendering
17. Message Sticker Secure rendering
18. Message Sticker Secure rendering
19. Modeling the Text Overlay Message Sticker An SVG Rect defines the area where the user-given message will be rendered. Other attributes include: Text style, alignment, direction, Maximum Font size, interline and letter spacing Custom Sticker An SVG Path defines the coordinates and transformations where the user-given word will be rendered. Other attributes include: Text style, alignment and direction, character limit
20. SVG Rendering: Which engine? › SVG Format is still evolving › Moving towards CSS Properties rather then attributes › Feature compatibility depends on the interpreter/engine used › Rendering Tools › Java Libraries › Apache Batik › External Tools › Headless Chrome/Firefox › Inkscape / ImageMagick
21. SVG Rendering: Which engine? › Apache Batik is about 10X faster than any other external tool… › Parses SVG Elements into AWT (Abstract Window Toolkit) components › Renders via Java graphics environment › …but it has some drawbacks: › NO browser environment, which means: › Limited CSS/JavaScript inside the SVG file › Alignment of vertical punctuation based on country › Incomplete SVG implementation › https://xmlgraphics.apache.org/batik/status.html JP Top-right-aligned TW center-aligned
22. SVG Composition: Message Stickers › Message sticker elevator pitch: › “The text area will wrap and resize the text automatically, like powerpoint!” › Problem: › SVG does not natively support multi-line text › It only allows for absolute or relative positioning of single-line text elements › Solution: › Implement our own wrapping/resizing and positioning algorithm!
23. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user
24. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。。。たとえば。。。面白い物語 Font Size: 32
25. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。。。面白い物語 Font Size: 32
26. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物語 Font Size: 32
27. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32
28. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。。。たとえば。。。ほんとに長い決して終わらない退屈な物話 Font Size: 32
29. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。。。ほんとに長い決して終わらない退屈な物話 Font Size: 32
30. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。 。。ほんとに長い決して終わらない退屈な物話 Font Size: 32
31. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。 。。ほんとに 長い決して終わらない退屈な物話 Font Size: 32
32. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。 。。ほんとに 長い決して終 わらない退屈な物話 Font Size: 32
33. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。 。。ほんとに 長い決して終 わらない退屈 な物話 Font Size: 32
34. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。 。。ほんとに長 い決して終わら ない退屈な物話 Font Size: 29 Re-execute wrap at lower size!
35. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。 。。ほんとに長 い決して終わら ない退屈な物話 Font Size: 29 Re-execute wrap at lower size! あ。。。たぶん。。。 うーん。。。 分かった！ じゃね！ Font Size: 32
36. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。 。。ほんとに長 い決して終わら ない退屈な物話 Font Size: 29 Re-execute wrap at lower size! あ。。。たぶ ん。。。 うーん。。。 分かった！ じゃね！ Font Size: 32
37. Automatic resizing and wrapping text › Target the largest font size, given the upper limit by the style creator › Prefer wrapping over resizing › Preserve new-line characters given by the user 短いテキスト Font Size: 32 長いテキスト。 。。たとえば。 。。面白い物 語 Font Size: 32 長いテキスト。 。。たとえば。 。。ほんとに長 い決して終わら ない退屈な物話 Font Size: 29 Re-execute wrap at lower size! あ。。。たぶん。。 。 うーん。。。 分かった！ じゃね！ Font Size: 24 Re-execute wrap at lower size! Keep the original new lines!
38. Automatic resizing and wrapping text › Splitting the original text in lines › Calculate the Pixel dimensions of each line using FontMetrics › consider interline-spacing and letter-spacing › If a single line is too long, wrap into a new line › If the summed heights of the lines is too high, reduce the font size › Iterate until it fits both horizontally and vertically
39. Overlay Rendering: Server-side flow
40. Overlay Rendering: Server-side flow
41. Overlay Rendering: Server-side flow
42. Overlay Rendering: Server-side flow
43. Overlay Rendering: Server-side flow
44. Overlay Rendering: Server-side flow
45. Overlay Rendering: Server-side flow
46. Overlay Rendering: Server-side flow
47. Overlay Rendering: Server-side flow
48. Performance Tests: Initial testing data › load test with only quantitative data: › render a simple “text” string using a single font › Promising results: 300 RPS!
49. Performance tests with internal users › Higher Latency ( > 120ms) › Lower RPS (120-150 req/s) › The results themselves were not bad, but less than we were expecting from the initial test › The initial test data was not good enough!
50. Performance Tests: Better testing data › Random characters from a set from each supported language › Strings with random lengths up to a defined maximum › Different font for each language
51. Overlay Rendering: Server-side flow Mostly fast I/O operations CPU Intensive Operations Latency Breakdown (ms) by percentile 50% 90% Max 10.25 24.61 34.03 0.07 0.17 0.57 0.30 0.95 2.11 6.59 37.41 151.71 2.27 3.21 4.50 * Running on a local instance 64.05 117.74 202.75
52. Overlay Rendering: Latency Factors › The number of characters in the input affects the resize/wrap algorithm execution › The font, the language and the complexity of the characters affects: › The drawing of the characters (rendering of SVG into PNG) › The size calculation of each character (resize/wrap algorithm)
53. Capacity Planning › Find the peak rps per server at a tolerable CPU Usage › combined (Message + Custom) 150 rps peak traffic at about 40% CPU usage › Assumptions: › Handle as much traffic for Message Sticker as current custom sticker product › Message Sticker overlay cannot be cached › Due to average group size, 5 rendering calls for each Message sticker sent › …and given a peak of 5K RPS at New Year’s Eve for Custom Sticker, › it could mean 25K RPS for Message Sticker › 200 rendering servers to handle the peak traffic!
54. Product Release - April 2020 › Product Launch with 200 servers › … it was over-provisioned, at least for the season › … scaled down to 100 servers: › Cost effectiveness: it’s not worth it to keep so many servers running the whole year › Most of the peaks are seasonal and/or predictable › More resources for other projects › and… Memory leaks!
55. Production Environment: Memory Leak! › VM Memory 16Gb - JVM Max heap size: 8Gb › Some Font related classes allocate Native Memory for caching purposes, and due to a loop of Hard/SoftReferences (ref: JDK-8209113) , this memory is not freed correctly
56. Production Environment: Memory Leak!
57. Production Environment: Memory Leak! Why? › new Font object at each render request and every resize › Static caching of metrics and glyphs/strikes › Hoping for GC to dispose the objects correctly › The testing data was STILL not stressing the system enough: › limited amount of unique characters › Only 1 font family per language › limited string length, leading to less resize operations › … so less Native memory was used for caching
58. Production Environment: Memory Leak! › Reproducing the issue › Using the full Unicode space of characters, and very long strings of random characters › Using all available fonts › Fixing the Issue › Option 1: Patching the JVM, replacing the SoftReference with a WeakReference › Managing a custom patched JVM is complicated › Increased garbage collection activity › Option 2: caching the Font objects ourselves › limited amount of FontFamilies and allowed Font sizes, so the cache is also limited
59. Production Environment: Scaling Out Database Rendering Server 1 LB Client Rendering Server 2 Rendering Server N
60. Production Environment: Scaling Out Database Rendering Server 1 LB Client Rendering Server 2 Rendering Server N Connection Pool › DB Access Control Layer with IP Whitelisting à slows down an emergency scale out operation › Limited number of connections to DB à limits scaling out capacity › DB access is a minor part of the task of this server, it shouldn’t be a bottle neck for scaling! Access Control Layer
61. Production Environment: Scaling Out Database Rendering Server 1 LB Client Rendering Server 2 Rendering Server N Connection Pool LB DB Api Server 1 DB Api Server 2 DB Api Server M N > M Access Control Layer › DB API servers hold the connection pool to database and use caching for faster access to the data › Scaling out Rendering servers is now independent from ACL › Scaling out Rendering servers is not limited by maximum open connections on DB › DB API servers have a simple task, can be fewer and lower spec VMs
62. Takeaways › A quick prototype can be useful to understand the needs of the business side › Real-world data (or as close to it as possible) is fundamental for stress/load testing › Be aware of the traffic patterns of your service › Keep in mind scalability when thinking about service architecture › Refactoring or architecture changes are not only for legacy code!
63. Thank you