Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Armeria at talk-server

Armeria at talk-server

A case study of Armeria at talk-server, which is one of core systems in LINE messaging platform
(presented at Armeria Tokyo workshop 2019)

YoungTae Seok

November 18, 2019
Tweet

Other Decks in Programming

Transcript

  1. Armeria at talk-server YoungTae Seok, LINE

  2. About me • YoungTae Seok (GitHub ID: delegacy) • LINER

    in Korea • Maintain talk-server with excellent colleagues • Feel sorry about a few contributions to Armeria 2
  3. talk-server • talk-server is a API server, providing the following

    LINE’s core functionalities • Various types (text, image) of messages to 1:1, Room, Group • User authentication & registration • Contacts / User setting management • Internal API for other internal systems 3
  4. Armeria at talk-server • Armeria clients to communicate with internal/external

    systems • Armeria server to serve internal/external requests 4
  5. Armeria at talk-server 5

  6. Armeria at talk-server 6

  7. Armeria at talk-server • talk-server uses Armeria’s powerful features as

    well • Circuit breaker • Metrics • Client-side health-check and load-balancing • Client decoration • Retrofit integration • And so much more! 7
  8. Push Notification • talk-server’s push executor threads use Armeria to

    communicate with NPush. 8
  9. 20190101 Push Notification Outage • At that time, talk-server used

    Armeria synchronously in push notification logic. 9
  10. 20190101 Push Notification Outage • When massive sendMessage requests come

    in a very short time just like New Year Greeting, 10
  11. 20190101 Push Notification Outage • Further push notification tasks were

    rejected because the thread pool queue was full. 11
  12. 20190101 Push Notification Outage • talk-server used Armeria to NPush

    synchronously. • talk-server, in some scenarios, should deliver push notification results to other internal systems. private Map<String, Object> invoke(NotificationTarget target, NpushPayload payload) { try { response = npushRetrofitClient.sendPayload(serviceId, queryMap, payload).get(); } catch (Exception e) { … } } WHAT?! 12
  13. 20190101 Push Notification Outage • talk-server uses Armeria to NPush

    asynchronously. • talk-server, however, in minor scenarios, still waits HTTP calls. private CompletableFuture<Integer> requestToNPush(PayloadContext payloadContext, NPushPayload payload, NPushProperty npushProperty, String monitorId, boolean isAsync) { … return client.sendPayload(serviceId, buildQueryMap(npushProperty.getNpushKey()), payload) .handle((response, e) -> { … }; 13
  14. 20190101 Push Notification Outage • talk-server didn’t fully utilize Armeria’s

    connection pool, resulting in unnecessary new connections. <bean id="npushConnectionKR" class="NpushConnectionImpl"> … <constructor-arg name="npushRetrofitClient" ref="npushArmeriaClientKR" /> </bean> <bean id="npushConnectionJP" class="NpushConnectionImpl"> … <constructor-arg name="npushRetrofitClient" ref="npushArmeriaClientJP" /> </bean> <bean id="npushArmeriaClientKR" class="NpushRetrofitClientFactory"> … </bean> <bean id="npushArmeriaClientJP" class="NpushRetrofitClientFactory"> … </bean> $ cat /proc/net/sockstat sockets: used 5576 TCP: inuse 5486 orphan 0 tw 60 alloc 5497 mem 598 UDP: inuse 8 mem 2 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 14
  15. 20190101 Push Notification Outage public class NpushRetrofitClientFactory implements FactoryBean<NpushRetrofitClient> {

    @Override public NpushRetrofitClient getObject() throws Exception { ... final ClientFactory clientFactory = new ClientFactoryBuilder()... .build(); return new ArmeriaRetrofitBuilder(clientFactory) ... .build() .create(NpushRetrofitClient.class); } } 15 final class HttpClientFactory extends AbstractClientFactory { ... private final ConcurrentMap<EventLoop, HttpChannelPool> pools = new MapMaker().weakKeys().makeMap(); ... HttpChannelPool pool(EventLoop eventLoop) { final HttpChannelPool pool = pools.get(eventLoop); if (pool != null) { return pool; } return pools.computeIfAbsent(eventLoop, e -> new HttpChannelPool(this, eventLoop, connectionPoolListener())); } }
  16. 20190101 Push Notification Outage • talk-server uses the existing ClientFactory,

    so that Armeria clients share the connection pool. @Bean NPushConnection npushConnection(NPushRetrofitClient npushArmeriaClient, NPushRetrofitClient npushAsyncArmeriaClient, …) { return new NPushConnectionImpl(npushArmeriaClient, npushAsyncArmeriaClient, …); } @Bean ClientFactory npushAsyncClientFactory(…) { return …; } @Bean NPushRetrofitClient npushAsyncArmeriaClient(ClientFactory npushAsyncClientFactory, …) { return new NPushRetrofitClientBuilder().baseUrl(baseUrl) .clientFactory(npushAsyncClientFactory) … .build(); } $ cat /proc/net/sockstat sockets: used 2994 TCP: inuse 2906 orphan 0 tw 41 alloc 2915 mem 554 UDP: inuse 8 mem 2 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 16
  17. 20190101 Push Notification Outage • talk-server didn’t use a circuit

    breaker, but now it does. @Bean public CircuitBreaker npushCircuitBreaker(…) { return new CircuitBreakerBuilder("npush-circuit-breaker") .counterSlidingWindowMillis(counterWindowMillis) .minimumRequestThreshold(minimumRequest) .failureRateThreshold(failureRate) .circuitOpenWindowMillis(circuitOpenMillis) … .build(); } @Bean NPushRetrofitClient npushAsyncArmeriaClient(…, CircuitBreaker npushCircuitBreaker, …) { return new NPushRetrofitClientBuilder().baseUrl(baseUrl) … .npushCircuitBreaker(npushCircuitBreaker) .defaultCircuitBreakerStrategy(defaultCircuitBreaker Strategy) … .build(); } 17
  18. Control factors for stable push notification • These should be

    under control for stable push notification • # of concurrency • ConcurrencyLimitingClient • # of connections • # of eventLoops per endpoint • Make client connection pooling policy more configurable 18
  19. Thank you