Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Event Delivery Improvement By Server-side Debouncing

Event Delivery Improvement By Server-side Debouncing

LINE DevDay 2020

November 25, 2020
Tweet

More Decks by LINE DevDay 2020

Other Decks in Technology

Transcript

  1. Agenda › Background: Event Delivery Architecture › Problem: Bursting Notifications

    and Traffics › Throttling: Initial Solution with Limitations › Debouncing: Design and Implementation › Result: Application and Improvements › And More: Room For Further Improvements
  2. Events as Unit of Updates “Square” Status Updates Mark-As-Read Updates

    Note & Post Updates Member Status Updates Chat Status Updates Message Notifications
  3. Bustling Busy Chats (a.k.a Daily Incidents in LINE OpenChat) members

    / chatroom 5K updates / sec / chatroom 1M events / sec / chatroom 200
  4. Merged Events events of the same type, with duplicate target

    objects in the history events of the same type, with only the latest state for each object
  5. Bursting Notifications necessary evil with heavy traffics history of events

    (e.g. display name change) event notifications merged events upon an actual fetch (at the blue one) “A” “B” … “Z” “Z”
  6. Bursting Notifications members / chatroom 5K updates / sec /

    chatroom 1M events / sec / chatroom 200
  7. Problems worsened by bursting notifications › Hotspots associated with particular

    busy chats with incessant updates › Wasted notifications immediately followed by round-trips to fetch events › Heavy storage usage to identify users to notify each time
  8. Throttling Problem with discarding all the rest latest update, discarded

    if no more events follow for the next 2 mins… then the client gets no update for 3 mins in total one event per time window of 1 min 2nd window
  9. › naive client › sends a request per key input

    › debouncing client › considers the last key input only › removes “bouncing” keystrokes in the middle Example: Search Autocomplete
  10. Event Publishing Delegation where redesign for debouncing takes place TaskHandler

    for chats TaskHandler for messages PublishTask (type, target) Processor … via Kafka Streams Asynchronous Distribution and Delegation of Event Handling › Publish-tasks are asynchronously distributed and delegated to “publish-server” via Kafka Streams › Tasks are divided and distributed by their types and targets, e.g. which chatrooms they are for
  11. Overall Flow remains the same TaskHandler for chats TaskHandler for

    messages PublishTask (type, target) Processor … via Kafka Streams
  12. Overall Flow redesign takes place within TaskHandler TaskHandler for chats

    TaskHandler for messages PublishTask (type, target) Processor … via Kafka Streams
  13. 2. Be able to control how many events can be

    delivered at max for how long 3. Let “urgent” updates bypass rate-limiting, if necessary 1. Notify only the latest updates about all types of entities, at least once in a while Requirements in light of OpenChat’s business logic
  14. › Use CentralDogma 2. Be able to control how many

    events can be delivered at max for how long 3. Let “urgent” updates bypass rate-limiting, if necessary › Override PublishTaskDebouncer<T>#execute with customized handling logic 1. Notify only the latest updates about all types of entities, at least once in a while › Tweak the current implementation › Use Bucket4j and Caffeine Solutions for the requirements
  15. rate-limiting Bucket4j dynamic configuration Central Dogma caching Caffeine Req 1

    & 2: Rate-Limiting and Configs Inside RateLimiter and Switcher
  16. // bucket consumption Single.just(getBucket(task).tryConsumeAndReturnRemaining(1)) .map(probe -> { Cache<K, Target> targets

    = debounced.get(); ...
 // cache configuration Caffeine. ... .removalListener((key, target, cause) -> { ... doExecute(target.getTask()); ... }) ... Req 1 & 2: Rate-Limiting and Configs Bucket4j and Caffeine for rate-limiting
  17. Req 1 & 2: Rate-Limiting and Configs CentralDogma for dynamic

    configuration "chatStatus": { // configs for each type of events "enabled": true, "dryRun": false, ... "acceptLimit": 20, "acceptDurationSecs": 60, "staticControl": { ... }, "rateLimiterType": "DEBOUNCER" }, …
  18. Req 3: Prioritization & Rate-Limit Bypass for urgent message notifications

    Completable handleImmediates(PublishTask task) { return findMessage(task) .filter(msg -> { ... return isMention || isReply; }) .flatMapCompletable(msg -> { ... return ... doHandleImmediates(task, msg) ...; }); }
  19. Req 3: Prioritization & Rate-Limit Bypass for urgent message notifications

    return new PublishTaskDebouncer<Chatroom>(control, ...) { ... @Override public Single<...> isRateLimited(PublishTask task) { return handleImmediates(task).andThen(super.isRateLimited(task)); } ... @Override public Completable doExecute(PublishTask task) { return doHandleDelayables(task); } ... };
  20. Recap: Problems worsened by bursting notifications › Hotspots associated with

    particular busy chats with incessant updates › Wasted notifications immediately followed by round-trips to fetch events › Heavy storage usage to identify users to notify each time
  21. Event Notifications in one chat within ten minutes 21:48 21:49

    21:50 21:51 21:52 21:53 21:54 21:55 21:56 21:57 21:58 21:59 rate-limited (1200/5s) unlimited
  22. Storage Usages no hotspots and bursts 21:01 21:02 21:03 21:04

    21:05 21:06 debounced unlimited - 400%
  23. notifications no bursting with only a few round-trips consistency in

    storage no hotspots Summary With Rate-limiting by Debouncing