Armeria at talk-server

A case study of Armeria at talk-server, which is one of core systems in LINE messaging platform
(presented at Armeria Tokyo workshop 2019)

Transcript

1. Armeria at talk-server YoungTae Seok, LINE
2. About me • YoungTae Seok (GitHub ID: delegacy) • LINER in Korea • Maintain talk-server with excellent colleagues • Feel sorry about a few contributions to Armeria 2
3. talk-server • talk-server is a API server, providing the following LINE’s core functionalities • Various types (text, image) of messages to 1:1, Room, Group • User authentication & registration • Contacts / User setting management • Internal API for other internal systems 3
4. Armeria at talk-server • Armeria clients to communicate with internal/external systems • Armeria server to serve internal/external requests 4
5. Armeria at talk-server 5
6. Armeria at talk-server 6
7. Armeria at talk-server • talk-server uses Armeria’s powerful features as well • Circuit breaker • Metrics • Client-side health-check and load-balancing • Client decoration • Retrofit integration • And so much more! 7
8. Push Notification • talk-server’s push executor threads use Armeria to communicate with NPush. 8
9. 20190101 Push Notification Outage • At that time, talk-server used Armeria synchronously in push notification logic. 9
10. 20190101 Push Notification Outage • When massive sendMessage requests come in a very short time just like New Year Greeting, 10
11. 20190101 Push Notification Outage • Further push notification tasks were rejected because the thread pool queue was full. 11
12. 20190101 Push Notification Outage • talk-server used Armeria to NPush synchronously. • talk-server, in some scenarios, should deliver push notification results to other internal systems. private Map<String, Object> invoke(NotificationTarget target, NpushPayload payload) { try { response = npushRetrofitClient.sendPayload(serviceId, queryMap, payload).get(); } catch (Exception e) { … } } WHAT?! 12
13. 20190101 Push Notification Outage • talk-server uses Armeria to NPush asynchronously. • talk-server, however, in minor scenarios, still waits HTTP calls. private CompletableFuture<Integer> requestToNPush(PayloadContext payloadContext, NPushPayload payload, NPushProperty npushProperty, String monitorId, boolean isAsync) { … return client.sendPayload(serviceId, buildQueryMap(npushProperty.getNpushKey()), payload) .handle((response, e) -> { … }; 13
14. 20190101 Push Notification Outage • talk-server didn’t fully utilize Armeria’s connection pool, resulting in unnecessary new connections. <bean id="npushConnectionKR" class="NpushConnectionImpl"> … <constructor-arg name="npushRetrofitClient" ref="npushArmeriaClientKR" /> </bean> <bean id="npushConnectionJP" class="NpushConnectionImpl"> … <constructor-arg name="npushRetrofitClient" ref="npushArmeriaClientJP" /> </bean> <bean id="npushArmeriaClientKR" class="NpushRetrofitClientFactory"> … </bean> <bean id="npushArmeriaClientJP" class="NpushRetrofitClientFactory"> … </bean> $ cat /proc/net/sockstat sockets: used 5576 TCP: inuse 5486 orphan 0 tw 60 alloc 5497 mem 598 UDP: inuse 8 mem 2 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 14
15. 20190101 Push Notification Outage public class NpushRetrofitClientFactory implements FactoryBean<NpushRetrofitClient> { @Override public NpushRetrofitClient getObject() throws Exception { ... final ClientFactory clientFactory = new ClientFactoryBuilder()... .build(); return new ArmeriaRetrofitBuilder(clientFactory) ... .build() .create(NpushRetrofitClient.class); } } 15 final class HttpClientFactory extends AbstractClientFactory { ... private final ConcurrentMap<EventLoop, HttpChannelPool> pools = new MapMaker().weakKeys().makeMap(); ... HttpChannelPool pool(EventLoop eventLoop) { final HttpChannelPool pool = pools.get(eventLoop); if (pool != null) { return pool; } return pools.computeIfAbsent(eventLoop, e -> new HttpChannelPool(this, eventLoop, connectionPoolListener())); } }
16. 20190101 Push Notification Outage • talk-server uses the existing ClientFactory, so that Armeria clients share the connection pool. @Bean NPushConnection npushConnection(NPushRetrofitClient npushArmeriaClient, NPushRetrofitClient npushAsyncArmeriaClient, …) { return new NPushConnectionImpl(npushArmeriaClient, npushAsyncArmeriaClient, …); } @Bean ClientFactory npushAsyncClientFactory(…) { return …; } @Bean NPushRetrofitClient npushAsyncArmeriaClient(ClientFactory npushAsyncClientFactory, …) { return new NPushRetrofitClientBuilder().baseUrl(baseUrl) .clientFactory(npushAsyncClientFactory) … .build(); } $ cat /proc/net/sockstat sockets: used 2994 TCP: inuse 2906 orphan 0 tw 41 alloc 2915 mem 554 UDP: inuse 8 mem 2 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 16
17. 20190101 Push Notification Outage • talk-server didn’t use a circuit breaker, but now it does. @Bean public CircuitBreaker npushCircuitBreaker(…) { return new CircuitBreakerBuilder("npush-circuit-breaker") .counterSlidingWindowMillis(counterWindowMillis) .minimumRequestThreshold(minimumRequest) .failureRateThreshold(failureRate) .circuitOpenWindowMillis(circuitOpenMillis) … .build(); } @Bean NPushRetrofitClient npushAsyncArmeriaClient(…, CircuitBreaker npushCircuitBreaker, …) { return new NPushRetrofitClientBuilder().baseUrl(baseUrl) … .npushCircuitBreaker(npushCircuitBreaker) .defaultCircuitBreakerStrategy(defaultCircuitBreaker Strategy) … .build(); } 17
18. Control factors for stable push notification • These should be under control for stable push notification • # of concurrency • ConcurrencyLimitingClient • # of connections • # of eventLoops per endpoint • Make client connection pooling policy more configurable 18
19. Thank you