Jakub Marchwicki <@kubem> from sync to async the swampy grounds of handling http requests Jakub Marchwicki <@kubem> 

Jakub Marchwicki <@kubem> Tomek Nurkiewicz http://nurkiewicz.github.io/talks/2018/reactive-lessons/ 

Jakub Marchwicki <@kubem> Tomek Nurkiewicz http://nurkiewicz.github.io/talks/2018/reactive-lessons/ 

Jakub Marchwicki <@kubem> Tomek Nurkiewicz http://nurkiewicz.github.io/talks/2018/reactive-lessons/ 

Jakub Marchwicki <@kubem> Tomek Nurkiewicz http://nurkiewicz.github.io/talks/2018/reactive-lessons/ The Point 

Jakub Marchwicki <@kubem> What is The Point 

Software engineer Consultant Trainer Chief Mob Officer Jakub Marchwicki <@kubem> http://jakub.marchwicki.pl 

Jakub Marchwicki <@kubem> What is The Point a traffic which justifies the Netflix architecture 

Jakub Marchwicki <@kubem> What is The Point a traffic which justifies the Netflix a reactive architecture 

Jakub Marchwicki <@kubem> What is The Point a traffic which justifies the Netflix a reactive architecture way of solving business goals 

Jakub Marchwicki <@kubem> traditional synchronous bleeding edge reactive architecture are not toggles architecture are sliders 

Jakub Marchwicki <@kubem> https://dzone.com/articles/spring-boot-20-webflux-reactive-performance-test (...) better performance than synchronous code on high concurrency scenarios 

Jakub Marchwicki <@kubem> https://dzone.com/articles/spring-boot-20-webflux-reactive-performance-test (...) better performance than synchronous code on high concurrency scenarios public Flux getUsersAsync() { return Flux .fromIterable(userList) .delaySubscription( Duration.ofMillis(delay) ); } 

Jakub Marchwicki <@kubem> let’s decompose 

Jakub Marchwicki <@kubem> synchronous, multithreaded, blocking each http request in handled exclusively by a single thread (end-2-end) number of threads (therefore nb of concurrent connections) is limited to the pool size processing time for each requests determines the throughput 

Jakub Marchwicki <@kubem> This leads us to question... how many requests can we handle 

Jakub Marchwicki <@kubem> ~450ms/request with up to 200 threads 

Jakub Marchwicki <@kubem> ~450ms/request with up to 200 threads 

Jakub Marchwicki <@kubem> ~450ms/request with up to 200 threads ~444 requests / second 

Jakub Marchwicki <@kubem> ~450 ms/request with up to 200 threads ~444 requests / second on a long-term average 

Jakub Marchwicki <@kubem> ~450 ms/request with up to 200 threads ~444 requests / second on a long-term average no. items in the queue average time spent in the queue arrival rate https://www.process.st/littles-law/ 

Jakub Marchwicki <@kubem> L = λ * W customers arrive at the rate of 10 per hour customer stay an average of 0.5 hour average number of customers in the store at any time to be 5 customers arrive at the rate of 20 per hour average number of customers in the store at any time to be ?? customer stay an average of ?? hour customer stay an average of 0.5 hour (stays the same) average number of customers in the store at any time to be 10 

Jakub Marchwicki <@kubem> in theory practice meets theory 

Jakub Marchwicki <@kubem> @GetMapping("/loans/{loanId}") public LoanInformation loanDetails(@PathVariable("loanId") UUID loanId) { var riskInformation = riskClient.getRiskAssessmentDetails(loanId); var loanDetails = loansClient.getLoanDetails(loanId); var member = membersClient.getMemberDetails(loanDetails.getMemberId()); return LoanInformation .fromLoanDetails(loanDetails) .requestId(UUID.randomUUID()) .member(member) .riskAssessment(riskInformation) .build(); } 

Jakub Marchwicki <@kubem> Ideal world 

Jakub Marchwicki <@kubem> Tomcat 

Jakub Marchwicki <@kubem> Jetty 

Jakub Marchwicki <@kubem> Undertow 

Jakub Marchwicki <@kubem> Upon startup, Tomcat will create threads based on the value set for minSpareThreads (10) and increase that number based on demand, up to the number of maxThreads (200). If the maximum number of threads is reached, and all threads are busy, incoming requests are placed in a queue (acceptCount - 100) to wait for the next available thread. The server will only continue to accept a certain number of concurrent connections (as determined by maxConnections - 200). https://www.datadoghq.com/blog/tomcat-architecture-and-performance/ 

Jakub Marchwicki <@kubem> Undertow uses XNIO as the default connector. XNIO (...) default configuration (...) is I/O threads initialized to the number of your logical threads and the worker thread equal to 8 * CPU cores. So on typical 4 cores Intel CPU with hyper-threading you will end up with 8 I/O threads and 64 working threads. https://jmnarloch.wordpress.com/2016/04/26/spring-boot-tuning-your-undertow-application-for-throughput/ 

Jakub Marchwicki <@kubem> Undertow uses XNIO as the default connector. XNIO (...) default configuration (...) is I/O threads initialized to the number of your logical threads and t`he worker thread equal to 8 * CPU cores. So on typical 4 cores Intel CPU with hyper-threading you will end up with 8 I/O threads and 64 working threads. https://jmnarloch.wordpress.com/2016/04/26/spring-boot-tuning-your-undertow-application-for-throughput/ var ioThreads = Math.max( Runtime.getRuntime().availableProcessors(), 2 ); var workerThreads = ioThreads * 8; number of processors available to the JVM number of logical cores: Core i7 w. HyperThreading: 8 number of logical cores: Q6700: 4 number of logical cores: docker --cpus=1 on a quad core: 8 Compare it to 100 or 1000, defaults at Tomcat or Jetty number of logical cores: docker --cpus=1 on a quad core: 1 (JDK10) number of logical cores: docker --cpuset-cpus=0,1 on a quad core: 2 

TAKEAWAY Full stack developer doesn’t mean same technology on frontend and backend. Seniority comes from understanding layers beyond the code you craft. 

Jakub Marchwicki <@kubem> not all threads were created equal requests 

Jakub Marchwicki <@kubem> @GetMapping("/loans/{loanId}") public LoanInformation loanDetails(@PathVariable("loanId") UUID loanId) { var riskInformation = riskClient.getRiskAssessmentDetails(loanId); var loanDetails = loansClient.getLoanDetails(loanId); var member = membersClient.getMemberDetails(loanDetails.getMemberId()); return LoanInformation .fromLoanDetails(loanDetails) .requestId(UUID.randomUUID()) .member(member) .riskAssessment(riskInformation) .build(); } on demand computation, take time (~600ms) these two are relatively fast, direct lookups (~150 - 300 ms) 

Jakub Marchwicki <@kubem> requests thread pool lookup member (~300ms) search members (~600ms) lookup loan (~150ms) 

Jakub Marchwicki <@kubem> requests thread pool lookup member (~300ms) search members (~600ms) lookup loan (~150ms) 

Jakub Marchwicki <@kubem> requests thread pool lookup member (~300ms) search members (~600ms) lookup loan (~150ms) 

Jakub Marchwicki <@kubem> requests thread pool lookup member (~300ms) search members (~600ms) lookup loan (~150ms) 

Jakub Marchwicki <@kubem> 

Jakub Marchwicki <@kubem> 

Jakub Marchwicki <@kubem> asynchronous, multithreaded, blocking 

Jakub Marchwicki <@kubem> operations are done by choosing a worker thread from a thread pool the io thread is returned to the pool to run other requests, and process the upstream response asynchronously too the worker thread notifies the request thread when its work is complete. to offset these risks of backend latency, throttling mechanisms and circuit breakers help keep the blocking systems stable and resilient. 

Jakub Marchwicki <@kubem> @GetMapping("/loans/{loanId}") public CompletableFuture loanDetails(@PathVariable("loanId") UUID loanId) { return supplyAsync(() -> loansClient.getLoanDetails(loanId), executor) .thenApply(l -> { Member memberDetails = membersClient.getMemberDetails(l.getMemberId()); return Tuple.of(l, memberDetails); }) .thenCombine(supplyAsync(() -> riskClient.getRiskAssessmentDetails(loanId), executor), (loanDetailsMember, riskInformation) -> LoanInformation .fromLoanDetails(loanDetailsMember.getLeft()) .requestId(UUID.randomUUID()) .member(loanDetailsMember.getRight()) .riskAssessment(riskInformation) .build()); } 

Jakub Marchwicki <@kubem> requests thread pool lookup member (~300ms) search members (~600ms) lookup loan (~150ms) service thread pool 

Jakub Marchwicki <@kubem> requests thread pool lookup member (~300ms) search members (~600ms) lookup loan (~150ms) service thread pool ? 

TAKEAWAY Thread pool tuning is tied to what the application needs Understand the nature of the traffic 

Jakub Marchwicki <@kubem> Worst case scenario?? 

Jakub Marchwicki <@kubem> Worst case scenario?? https://www.nurkiewicz.com/2011/03/tenfold-increase-in-server-throughput.html 

Jakub Marchwicki <@kubem> asynchronous, single-threaded, non-blocking 

Jakub Marchwicki <@kubem> @GetMapping("/loans/{loanId}") public Observable loanDetails(@PathVariable("loanId") UUID loanId) { Single loanDetails = loansClient.getLoanDetailsSingle(loanId).cache(); Single member = loanDetails .flatMap(l -> membersClient.getMemberDetailsSingle(l.getMemberId())); Single riskInformation = riskClient.getRiskAssessmentDetailsSingle(loanId); return Single.zip( loanDetails, member, riskInformation, (l, m, r) -> LoanInformation .fromLoanDetails(l) .requestId(UUID.randomUUID()) .member(m) .riskAssessment(r) .build() ).toObservable(); } 

Jakub Marchwicki <@kubem> reactive programming does not build a reactive system 

Jakub Marchwicki <@kubem> responsive handle requests in a reasonable time resilient stay responsive in the face of failures elastic scale up and down, be able to handle the load with minimal resources. message driven interactions using asynchronous message passing. a reactive system promise 

Jakub Marchwicki <@kubem> a reactive programming - in technical terms Handling huge volumes of data in multi-userness environment Efficiency gains: data stays on the same CPU, use of CPU level caches, fewer context switches 25% increase in throughput corresponding with a 25% reduction in CPU utilization 

Jakub Marchwicki <@kubem> a reactive programming - in technical terms Handling huge volumes of data in multi-userness environment Efficiency gains: data stays on the same CPU, use of CPU level caches, fewer context switches 25% increase in throughput corresponding with a 25% reduction in CPU utilization MOAR TRAFFIC! 

Jakub Marchwicki <@kubem> the reactive promise - but... Blocking systems are easy to grok and debug a thread is always doing a single operation The event loop’s stack trace is meaningless when trying to follow a request Unhandled exceptions create dangling resources (exception swallow) 

Jakub Marchwicki <@kubem> at what cost? 

Jakub Marchwicki <@kubem> final List results = getQueries().stream() //there're 6 db queries .map(query -> db.apply(query)) .sorted(naturalOrder()) .collect(Collectors.toList()); final List results = Observable.from(getQueries()) // there're 6 db queries .flatMap(query -> Async.start(() -> db.apply(query), scheduler)) .toSortedList() .toBlocking() .single(); final List results = new ArrayList<>(); for (Query q: getQueries()) { String result = db.apply(q); results.add(result); } results.sort(naturalOrder()); 

Jakub Marchwicki <@kubem> [EL Warning]: 2009-08-29 12:53:13.718--Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.1.2.v20090612-r4475): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.BatchUpdateException: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'SQL090829125312890' defined on 'REMINDISSUE'. Error Code: 20000 Exception in thread "main" javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.1.2.v20090612-r4475): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.BatchUpdateException: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'SQL090829125312890' defined on 'REMINDISSUE'. Error Code: 20000 at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commitIntern at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(Entit at com.example.dao.jpa.JpaDAO.commit(JpaDAO.java:99) at com.example.dao.jpa.JpaDAO.persist(JpaDAO.java:41) at com.example.dao.jpa.JpaDAO.persist(JpaDAO.java:1) at com.example.test.TestDAO.main(TestDAO.java:44) 

Jakub Marchwicki <@kubem> public static void main(String[] args) { Observable.empty() .observeOn(Schedulers.io()) .toBlocking() .first(); } Exception in thread "main" java.util.NoSuchElementException: Sequence contains no el at rx.internal.operators.OperatorSingle$ParentSubscriber.onCompleted(OperatorSin at rx.internal.operators.OperatorTake$1.onCompleted(OperatorTake.java:53) at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.pollQueue(Operato at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber$1.call(OperatorOb at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$2 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sche at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:114 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:61 at java.lang.Thread.run(Thread.java:745) 

TAKEAWAY Both WebMVC / Servlet / synchronous and WebFlux / RxJava / reactive have a reason to exist. 

TAKEAWAY Know your clients users and their flows Know your expectations what you optimizing for Know your domain Know your opportunity costs 

Jakub Marchwicki <@kubem> Thank you! https://speakerdeck.com/kubamarchwicki/sync-or-async Jakub Marchwicki <@kubem>