Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Event Delivery Improvement By Server-side Debouncing

Event Delivery Improvement By Server-side Debouncing

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 25, 2020
Tweet

Transcript

  1. None
  2. Agenda › Background: Event Delivery Architecture › Problem: Bursting Notifications

    and Traffics › Throttling: Initial Solution with Limitations › Debouncing: Design and Implementation › Result: Application and Improvements › And More: Room For Further Improvements
  3. Event Delivery Architecture that both enables and requires rate-limiting techniques

  4. Events as Unit of Updates “Square” Status Updates Mark-As-Read Updates

    Note & Post Updates Member Status Updates Chat Status Updates Message Notifications
  5. Bustling Busy Chats (a.k.a Daily Incidents in LINE OpenChat) members

    / chatroom 5K updates / sec / chatroom 1M events / sec / chatroom 200
  6. Ignorable Events’ History events of the same type, with duplicate

    target objects in the history
  7. Merged Events events of the same type, with duplicate target

    objects in the history events of the same type, with only the latest state for each object
  8. Bursting Notifications necessary evil with heavy traffics history of events

    (e.g. display name change) event notifications merged events upon an actual fetch (at the blue one) “A” “B” … “Z” “Z”
  9. Bursting Notifications members / chatroom 5K updates / sec /

    chatroom 1M events / sec / chatroom 200
  10. Problems worsened by bursting notifications › Hotspots associated with particular

    busy chats with incessant updates › Wasted notifications immediately followed by round-trips to fetch events › Heavy storage usage to identify users to notify each time
  11. Challenges OpenChat Server Talk Server OBS from DevDay 2019: Inside

    OpenChat
  12. throttling a simple rate-limiting technique with limitations

  13. Throttling Keeping only the first few within a timeframe

  14. Throttling Problem with discarding all the rest latest update, discarded

    if no more events follow for the next 2 mins… then the client gets no update for 3 mins in total one event per time window of 1 min 2nd window
  15. debouncing on the server side

  16. › naive client › sends a request per key input

    › debouncing client › considers the last key input only › removes “bouncing” keystrokes in the middle Example: Search Autocomplete
  17. Debouncing Saving the latest one within a timeframe

  18. debouncing design & implementation

  19. Event Publishing Delegation where redesign for debouncing takes place TaskHandler

    for chats TaskHandler for messages PublishTask (type, target) Processor … via Kafka Streams Asynchronous Distribution and Delegation of Event Handling › Publish-tasks are asynchronously distributed and delegated to “publish-server” via Kafka Streams › Tasks are divided and distributed by their types and targets, e.g. which chatrooms they are for
  20. Overall Flow remains the same TaskHandler for chats TaskHandler for

    messages PublishTask (type, target) Processor … via Kafka Streams
  21. Overall Flow redesign takes place within TaskHandler TaskHandler for chats

    TaskHandler for messages PublishTask (type, target) Processor … via Kafka Streams
  22. Event-Notification Task Handling original TaskHandler without rate-limiting #doHandle (conditionals on

    throttling) TaskHandler
  23. Event-Notification Task Handling revised TaskHandlers with rate-limiting RateLimit Switcher TaskHandler

    Throttler Debouncer #doHandle
  24. 2. Be able to control how many events can be

    delivered at max for how long 3. Let “urgent” updates bypass rate-limiting, if necessary 1. Notify only the latest updates about all types of entities, at least once in a while Requirements in light of OpenChat’s business logic
  25. › Use CentralDogma 2. Be able to control how many

    events can be delivered at max for how long 3. Let “urgent” updates bypass rate-limiting, if necessary › Override PublishTaskDebouncer<T>#execute with customized handling logic 1. Notify only the latest updates about all types of entities, at least once in a while › Tweak the current implementation › Use Bucket4j and Caffeine Solutions for the requirements
  26. rate-limiting Bucket4j dynamic configuration Central Dogma caching Caffeine Req 1

    & 2: Rate-Limiting and Configs Inside RateLimiter and Switcher
  27. // bucket consumption Single.just(getBucket(task).tryConsumeAndReturnRemaining(1)) .map(probe -> { Cache<K, Target> targets

    = debounced.get(); ...
 // cache configuration Caffeine. ... .removalListener((key, target, cause) -> { ... doExecute(target.getTask()); ... }) ... Req 1 & 2: Rate-Limiting and Configs Bucket4j and Caffeine for rate-limiting
  28. Req 1 & 2: Rate-Limiting and Configs CentralDogma for dynamic

    configuration "chatStatus": { // configs for each type of events "enabled": true, "dryRun": false, ... "acceptLimit": 20, "acceptDurationSecs": 60, "staticControl": { ... }, "rateLimiterType": "DEBOUNCER" }, …
  29. Req 3: Prioritization & Rate-Limit Bypass for urgent message notifications

    Completable handleImmediates(PublishTask task) { return findMessage(task) .filter(msg -> { ... return isMention || isReply; }) .flatMapCompletable(msg -> { ... return ... doHandleImmediates(task, msg) ...; }); }
  30. Req 3: Prioritization & Rate-Limit Bypass for urgent message notifications

    return new PublishTaskDebouncer<Chatroom>(control, ...) { ... @Override public Single<...> isRateLimited(PublishTask task) { return handleImmediates(task).andThen(super.isRateLimited(task)); } ... @Override public Completable doExecute(PublishTask task) { return doHandleDelayables(task); } ... };
  31. results application & improvements

  32. Recap: Problems worsened by bursting notifications › Hotspots associated with

    particular busy chats with incessant updates › Wasted notifications immediately followed by round-trips to fetch events › Heavy storage usage to identify users to notify each time
  33. Event Notifications in one chat within ten minutes 21:48 21:49

    21:50 21:51 21:52 21:53 21:54 21:55 21:56 21:57 21:58 21:59 rate-limited (1200/5s) unlimited
  34. Storage Usages no hotspots and bursts 21:01 21:02 21:03 21:04

    21:05 21:06 debounced unlimited - 400%
  35. Without Rate-Limiting At Peak notifications bursting wasteful round-trips incessant in

    storage hotspots
  36. notifications no bursting with only a few round-trips consistency in

    storage no hotspots Summary With Rate-limiting by Debouncing
  37. … and more room for further improvements?

  38. any new type of events under OpenChat’s event delivery architecture

  39. Thank you