Event Delivery Improvement By Server-side Debouncing

Jeong Yeop Kim
LINE Plus SQ Part Software Engineer
https://linedevday.linecorp.com/2020/ja/sessions/0786
https://linedevday.linecorp.com/2020/en/sessions/0786

Transcript

2. Agenda › Background: Event Delivery Architecture › Problem: Bursting Notifications and Traffics › Throttling: Initial Solution with Limitations › Debouncing: Design and Implementation › Result: Application and Improvements › And More: Room For Further Improvements
3. Event Delivery Architecture that both enables and requires rate-limiting techniques
4. Events as Unit of Updates “Square” Status Updates Mark-As-Read Updates Note & Post Updates Member Status Updates Chat Status Updates Message Notifications
5. Bustling Busy Chats (a.k.a Daily Incidents in LINE OpenChat) members / chatroom 5K updates / sec / chatroom 1M events / sec / chatroom 200
6. Ignorable Events’ History events of the same type, with duplicate target objects in the history
7. Merged Events events of the same type, with duplicate target objects in the history events of the same type, with only the latest state for each object
8. Bursting Notifications necessary evil with heavy traffics history of events (e.g. display name change) event notifications merged events upon an actual fetch (at the blue one) “A” “B” … “Z” “Z”
9. Bursting Notifications members / chatroom 5K updates / sec / chatroom 1M events / sec / chatroom 200
10. Problems worsened by bursting notifications › Hotspots associated with particular busy chats with incessant updates › Wasted notifications immediately followed by round-trips to fetch events › Heavy storage usage to identify users to notify each time
11. Challenges OpenChat Server Talk Server OBS from DevDay 2019: Inside OpenChat
12. throttling a simple rate-limiting technique with limitations
13. Throttling Keeping only the first few within a timeframe
14. Throttling Problem with discarding all the rest latest update, discarded if no more events follow for the next 2 mins… then the client gets no update for 3 mins in total one event per time window of 1 min 2nd window
15. debouncing on the server side
16. › naive client › sends a request per key input › debouncing client › considers the last key input only › removes “bouncing” keystrokes in the middle Example: Search Autocomplete
17. Debouncing Saving the latest one within a timeframe
18. debouncing design & implementation
19. Event Publishing Delegation where redesign for debouncing takes place TaskHandler for chats TaskHandler for messages PublishTask (type, target) Processor … via Kafka Streams Asynchronous Distribution and Delegation of Event Handling › Publish-tasks are asynchronously distributed and delegated to “publish-server” via Kafka Streams › Tasks are divided and distributed by their types and targets, e.g. which chatrooms they are for
20. Overall Flow remains the same TaskHandler for chats TaskHandler for messages PublishTask (type, target) Processor … via Kafka Streams
21. Overall Flow redesign takes place within TaskHandler TaskHandler for chats TaskHandler for messages PublishTask (type, target) Processor … via Kafka Streams
22. Event-Notification Task Handling original TaskHandler without rate-limiting #doHandle (conditionals on throttling) TaskHandler
23. Event-Notification Task Handling revised TaskHandlers with rate-limiting RateLimit Switcher TaskHandler Throttler Debouncer #doHandle
24. 2. Be able to control how many events can be delivered at max for how long 3. Let “urgent” updates bypass rate-limiting, if necessary 1. Notify only the latest updates about all types of entities, at least once in a while Requirements in light of OpenChat’s business logic
25. › Use CentralDogma 2. Be able to control how many events can be delivered at max for how long 3. Let “urgent” updates bypass rate-limiting, if necessary › Override PublishTaskDebouncer<T>#execute with customized handling logic 1. Notify only the latest updates about all types of entities, at least once in a while › Tweak the current implementation › Use Bucket4j and Caffeine Solutions for the requirements
26. rate-limiting Bucket4j dynamic configuration Central Dogma caching Caffeine Req 1 & 2: Rate-Limiting and Configs Inside RateLimiter and Switcher
27. // bucket consumption Single.just(getBucket(task).tryConsumeAndReturnRemaining(1)) .map(probe -> { Cache<K, Target> targets = debounced.get(); ...
 // cache configuration Caffeine. ... .removalListener((key, target, cause) -> { ... doExecute(target.getTask()); ... }) ... Req 1 & 2: Rate-Limiting and Configs Bucket4j and Caffeine for rate-limiting
28. Req 1 & 2: Rate-Limiting and Configs CentralDogma for dynamic configuration "chatStatus": { // configs for each type of events "enabled": true, "dryRun": false, ... "acceptLimit": 20, "acceptDurationSecs": 60, "staticControl": { ... }, "rateLimiterType": "DEBOUNCER" }, …
29. Req 3: Prioritization & Rate-Limit Bypass for urgent message notifications Completable handleImmediates(PublishTask task) { return findMessage(task) .filter(msg -> { ... return isMention || isReply; }) .flatMapCompletable(msg -> { ... return ... doHandleImmediates(task, msg) ...; }); }
30. Req 3: Prioritization & Rate-Limit Bypass for urgent message notifications return new PublishTaskDebouncer<Chatroom>(control, ...) { ... @Override public Single<...> isRateLimited(PublishTask task) { return handleImmediates(task).andThen(super.isRateLimited(task)); } ... @Override public Completable doExecute(PublishTask task) { return doHandleDelayables(task); } ... };
31. results application & improvements
32. Recap: Problems worsened by bursting notifications › Hotspots associated with particular busy chats with incessant updates › Wasted notifications immediately followed by round-trips to fetch events › Heavy storage usage to identify users to notify each time
33. Event Notifications in one chat within ten minutes 21:48 21:49 21:50 21:51 21:52 21:53 21:54 21:55 21:56 21:57 21:58 21:59 rate-limited (1200/5s) unlimited
34. Storage Usages no hotspots and bursts 21:01 21:02 21:03 21:04 21:05 21:06 debounced unlimited - 400%
35. Without Rate-Limiting At Peak notifications bursting wasteful round-trips incessant in storage hotspots
36. notifications no bursting with only a few round-trips consistency in storage no hotspots Summary With Rate-limiting by Debouncing
37. … and more room for further improvements?
38. any new type of events under OpenChat’s event delivery architecture
39. Thank you