LINE Corporation June 2018 Building Asynchronous Microservices with Armeria

Why build another microservice framework?

Asynchrony ● Most web frameworks are blocking and synchronous. – 1000 concurrent requests require 1000 threads. – In modern systems, a service sends requests to another backend. – Threads are busy doing nothing but just waiting for the backend. – Request queue is not consumed fast enough. – Result: ● Fragile system that falls apart even on a tiny backend failure ● Inefficient system that takes more memory and CPU than what it actually does

Reactive Streams ● Should we really buffer everything until a response is sent? – Retrieving 100,000 rows from a table ● Why should a client send many requests to reduce the server memory pressure? – Retrieving 1,000 rows 100 times ● What if we want to stream infinitely? – Stock quotes ● Our protocol is not a problem here. – Our ‘blocking’ programming model is blocking ‘us’.

Better RPC support ● RPC has been hardly a 1st-class citizen in web frameworks. – Distinction from RPC-level and HTTP-level failure – Access logs with RPC-level details – Debugging console ● What if … – We build a microservice framework with RPC at its core?

More reasons back in 2015 ● Leverage Java 8 syntax – Lambda expressions – Default interface methods – Static interface methods ● Provide ‘working’ HTTP/2 implementation – ‘H2C’ support (HTTP/2 over ‘C’leartext) – OpenSSL-based faster TLS connection

Better UX for devs ● This is the last yet most important part. ● Nobody wants a framework that’s hard to use. ● Can we design … – a user-friendly asynchronous programming API – … with modest learning curve?

Our answer – Armeria ● HTTP/2 – Both ciphertext and cleartext, a.k.a. h2 and h2c ● Reactive – Implements Reactive Streams API ● Ability to mix different types of services in a single server – gRPC, Thrift, REST services – Legacy webapps via embedded Tomcat or Jetty ● High-performance – Netty with /dev/epoll transport and BoringSSL SSLEngine ● Easy to learn and fun to use (hopefully)

Armeria by examples HTTP / REST

Hello, world! Server server = new ServerBuilder() .http(8080) .https(8443) .tlsSelfSigned() .service("/", (ctx, req) -> HttpResponse.of("Hello, world!")) .build(); server.start();

Hello, world – Parameterized Server server = new ServerBuilder() .http(8080) .service("/hello/:name", (ctx, req) -> HttpResponse.of("Hello, %s!", ctx.pathParam("name"))) .build(); server.start();

Hello, world – Annotated Server server = new ServerBuilder() .http(8080) .annotatedService(new Object() { @Get("/hello/:name") public String hello(@Param String name) { return String.format("Hello, %s!", name); } }) .build(); server.start();

With CompletableFuture Supplier> random = ...; // Takes long! Object randomService = new Object() { @Get("/random") public CompletableFuture nextInt() { // Request a random number. CompletableFuture f1 = random.get(); // Convert the random number into a response. CompletableFuture f2 = f1.thenApply(num -> HttpResponse.of("%d\n", num)) .exceptionally(t -> HttpResponse.of(500)); return f2; } }; new ServerBuilder() .http(8080).annotatedService(randomService).build().start();

Simplification with method chaining // Annotated version: @Get("/random") public CompletableFuture nextIntSimple() { return random.get() .thenApply(num -> HttpResponse.of("%d\n", num)) .exceptionally(t -> HttpResponse.of(500)); } // Non-annotated version: HttpService randomService = (ctx, req) -> HttpResponse.from( random.get() .thenApply(num -> HttpResponse.of("%d\n", num)) .exceptionally(t -> HttpResponse.of(500)));

Streaming a million integers Server server = new ServerBuilder() .http(8080) .service("/count", (ctx, req) -> { HttpResponseWriter res = HttpResponse.streaming(); res.write(HttpHeaders.of(200)); streamNumbers(res, /* start */ 0); return res; }) .build();

Streaming a million integers (cont’d) void streamNumbers(HttpResponseWriter res, int start) { int end = Math.min(start + 1_000, 1_000_000); int i; for (i = start; i < end; i++) { res.write(HttpData.ofUtf8("%d\n", i)); } if (i == 1_000_000) { res.close(); return; } int nextStart = i; res.onDemand(() -> streamNumbers(res, nextStart)); }

There are more! ● Query parameters and cookies ● Request and response converters ● Exception handlers ● Content type negotiation ● Regular-expression path mapping ● Aims to be on-par or better than

Armeria by examples Thrift and gRPC

gRPC & Thrift basics a. Write an IDL file – .proto or .thrift b. A compiler compiles it into .java files. c. Implement interfaces generated at ‘b’. namespace java com.example service HelloService { string hello(1:string name) } syntax = "proto3"; package grpc.hello; option java_package = "com.example"; service HelloService { rpc Hello (HelloRequest) returns (HelloReply) {} } message HelloRequest { string name = 1; } message HelloReply { string message = 1; }

gRPC Server server = new ServerBuilder() .http(8080) .service(new GrpcServiceBuilder().addService(new MyHelloService()) .build()) .build(); class MyHelloService extends HelloServiceGrpc.HelloServiceImplBase { @Override public void hello(HelloRequest req, StreamObserver cb) { HelloReply reply = HelloReply.newBuilder() .setMessage("Hello, " + req.getName() + '!') .build(); cb.onNext(reply); cb.onCompleted(); } }

Thrift Server server = new ServerBuilder() .http(8080) .service("/hello", THttpService.of(new MyHelloService())) .build(); class MyHelloService implements HelloService.AsyncIface { @Override public void hello(String name, AsyncMethodCallback cb) { cb.onComplete("Hello, " + name + '!'); } }

What’s better than the official impls? ● gRPC – Works on HTTP/1·2, TLS on·off – gRPC-Web support, i.e. can call gRPC services from JavaScript frontends ● Thrift – HTTP/2 – TTEXT a.k.a human-readable JSON ● Can mix gRPC, Thrift, REST, Tomcat, Jetty, … ● Can leverage the common Armeria features – Decorators – Web-based dashboard

Decorating services Separation of concerns

Intercepting an HTTP request THttpService.of(new MyHelloService()) .decorate((delegate, ctx, req) -> { logger.info("{} Received a request", ctx); HttpResponse res = delegate.serve(ctx, req); res.completionFuture().thenAccept(unused -> { logger.info("{} Sent a response", ctx); }); return res; }) ● Can’t access RPC information yet. ● Need to decompose THttpService into: – ThriftCallService – RPC layer – THttpService – HTTP layer

Intercepting an RPC request ● Complete access to the RPC call information ● Override the parameters and methods ThriftCallService.of(new MyHelloService()) .decorate((delegate, ctx, req) -> { logger.info("{} Method: {}, Params: {}", ctx, req.method(), req.params()); RpcResponse res = delegate.serve(ctx, req); res.thenAccept(value -> { logger.info("{} Result: {}", ctx, value); }); return res; }) .decorate(THttpService.newDecorator())

Decorator as core extension mechanism ● Request throttling ● Metric collection ● Distributed tracing (Zipkin) ● HTTP content encoding ● HTTP authn/z ● CORS ● Circuit breakers ● Automatic retries ● …

Distributed tracing with Brave·Zipkin import brave.Tracing; import brave.propagation.CurrentTraceContext; import brave.sampler.Sampler; Server server = new ServerBuilder() .http(8080) .service("/my_service", myService.decorate( HttpTracingService.newDecorator( Tracing.newBuilder() .currentTraceContext(CurrentTraceContext.Default.create()) .localServiceName("myService") .spanReporter(spanReporter) .sampler(Sampler.create(/* 5% */ 0.05f)) .build()))) .build();

Playing better with RPC Armeria documentation service

Armeria documentation service ● Enabled by adding DocService to ServerBuilder ● Browse and invoke RPC services in an Armeria server ● Supports both gRPC and Thrift ● We have a plan to add: – Support for plain HTTP, like Swagger – Metric monitoring console – Runtime configuration editor, e.g. logger level

No content

No content

● Share the URL to reproduce a call.

All the pieces fit together The ultimate all-in-one micro(?) service

ServerBuilder sb = new ServerBuilder(); // Thrift services sb.service("/thrift/foo", THttpService.of(myThriftFooService)); sb.service("/thrift/bar", THttpService.of(myThriftBarService)); // gRPC services sb.service(new GrpcServiceBuilder().addService(myGrpcFooService) .addService(myGrpcBarService) .build()); // Static files sb.service("prefix:/files", HttpFileService.forFileSystem("htdocs")); // Legacy webapps sb.service("prefix:/webapp/foo", TomcatService.forFileSystem("foo.war")); sb.service("prefix:/webapp/bar", TomcatService.forFileSystem("bar.war")); // Documentation service sb.service("prefix:/internal/docs", new DocService()); // L7 health check service sb.service("/internal/healthcheck", new HttpHealthCheckService()); // Prometheus exposition sb.service("/internal/prometheus", new PrometheusExpositionService(...)); // Enable logging for all services. sb.decorator(LoggingService.newDecorator()); // Enable distributed tracing for all services. sb.decorator(HttpTracingService.newDecorator(...)); // Send all access logs to Kafka. sb.decorator(KafkaStructuredLoggingService.newDecorator(...));

What about client-side?

Armeria client API ● Similar experience to the server-side API – e.g. Common type, Decorators, RPC-friendliness ● Client-specific stuff – Circuit breakers – Automatic retries – Client-side load balancing – Retrofit integration

// Send DNS queries periodically to discover service hosts. DnsServiceEndpointGroup group = DnsServiceEndpointGroup.of("my-service.cluster.local"); // Register the group into the registry. EndpointGroupRegistry.register("myService", group, EndpointSelectionStrategy.WEIGHTED_ROUND_ROBIN); // Create a new HTTP client that connects to the group, with retries and logging. HttpClient client = new HttpClientBuilder("http://group:myService") .decorator(RetryingHttpClient.newDecorator( RetryStrategy.onServerErrorStatus())) .decorator(LoggingClient.newDecorator()) .build(); // Send a request and print the response content. client.get("/greet").aggregate() .thenAccept(res -> System.out.println(res.content().toStringUtf8())) .exceptionally(cause -> { /* Handle an error. */ });

Got interested?

Let’s build Armeria together! ● Use it. ● Ask questions. ● Request new features. ● Tell us what rocks and sucks. ● Consider joining the effort.

Meet us at GitHub and Slack ● https://github.com/line/armeria ● https://line-slacknow.herokuapp.com/line-armeria/