sync or async

We live in exciting times. People say servlets - as we know them - are dead. The future is bright and asynchronous. Bright and event loop'ed. Bright and reactive.
People judge - it's not async / reactive / nodejs'ed, it's a dead end. But the world of sync and async is not black and white. It's shady, swampy and full of surprises. In this talk, we will explore some of these shades of grey.

In this talk, we will try to give an opinionated answer if reactive all the things is a way to go, at least in the context of web applications. We will explore the spectrum between traditionally synchronous servlets and a fully blown async reactive approach. With the support of numbers, measurements and experiments, we will investigate the simple and traditional approach to web requests. See where it works and where it fails - based on some typical web applications use cases. Having such unoptimal resource utilization as a baseline, we will look for some other approaches and evaluate them from different perspectives: performence (an obvoius one) but also readability, familiarity and a general common sense.

Transcript

1. Jakub Marchwicki <@kubem>
   from sync to async
   the swampy grounds of handling http requests
   Jakub Marchwicki <@kubem>
   

   View Slide

2. Jakub Marchwicki <@kubem> Tomek Nurkiewicz http://nurkiewicz.github.io/talks/2018/reactive-lessons/
   

   View Slide

3. Jakub Marchwicki <@kubem> Tomek Nurkiewicz http://nurkiewicz.github.io/talks/2018/reactive-lessons/
   

   View Slide

4. Jakub Marchwicki <@kubem> Tomek Nurkiewicz http://nurkiewicz.github.io/talks/2018/reactive-lessons/
   

   View Slide

5. Jakub Marchwicki <@kubem> Tomek Nurkiewicz http://nurkiewicz.github.io/talks/2018/reactive-lessons/
   The Point
   

   View Slide

6. Jakub Marchwicki <@kubem>
   What is The Point
   

   View Slide

7. Software engineer
   Consultant
   Trainer
   Chief Mob Officer
   Jakub Marchwicki <@kubem>
   http://jakub.marchwicki.pl
   

   View Slide

8. Jakub Marchwicki <@kubem>
   What is The Point
   a traffic which justifies
   the Netflix
   architecture
   

   View Slide

9. Jakub Marchwicki <@kubem>
   What is The Point
   a traffic which justifies
   the Netflix a reactive
   architecture
   

   View Slide

10. Jakub Marchwicki <@kubem>
    What is The Point
    a traffic which justifies
    the Netflix a reactive
    architecture
    way of solving business goals
    

    View Slide

11. Jakub Marchwicki <@kubem>
    traditional
    synchronous
    bleeding edge
    reactive
    architecture are not toggles
    architecture are sliders
    

    View Slide

12. Jakub Marchwicki <@kubem> https://dzone.com/articles/spring-boot-20-webflux-reactive-performance-test
    (...) better performance than
    synchronous code on high
    concurrency scenarios
    

    View Slide

13. Jakub Marchwicki <@kubem> https://dzone.com/articles/spring-boot-20-webflux-reactive-performance-test
    (...) better performance than
    synchronous code on high
    concurrency scenarios
    public Flux getUsersAsync() {
    return Flux
    .fromIterable(userList)
    .delaySubscription(
    Duration.ofMillis(delay)
    );
    }
    

    View Slide

14. Jakub Marchwicki <@kubem>
    let’s decompose
    

    View Slide

15. Jakub Marchwicki <@kubem>
    synchronous, multithreaded, blocking
    each http request in
    handled exclusively by a
    single thread (end-2-end)
    number of threads
    (therefore nb of concurrent
    connections) is limited to
    the pool size
    processing time for each
    requests determines the
    throughput
    

    View Slide

16. Jakub Marchwicki <@kubem>
    This leads us to question...
    how many requests can we handle
    

    View Slide

17. Jakub Marchwicki <@kubem>
    ~450ms/request with up to 200 threads
    

    View Slide

18. Jakub Marchwicki <@kubem>
    ~450ms/request with up to 200 threads
    

    View Slide

19. Jakub Marchwicki <@kubem>
    ~450ms/request with up to 200 threads
    ~444 requests / second
    

    View Slide

20. Jakub Marchwicki <@kubem>
    ~450 ms/request with up to 200 threads
    ~444 requests / second
    on a long-term average
    

    View Slide

21. Jakub Marchwicki <@kubem>
    ~450 ms/request with up to 200 threads
    ~444 requests / second
    on a long-term average
    no. items in the queue
    average time spent in
    the queue
    arrival rate
    https://www.process.st/littles-law/
    

    View Slide

22. Jakub Marchwicki <@kubem>
    L = λ * W
    customers arrive at
    the rate of 10 per hour
    customer stay an
    average of 0.5 hour
    average number of
    customers in the store
    at any time to be 5
    customers arrive at
    the rate of 20 per hour
    average number of
    customers in the store
    at any time to be ??
    customer stay an
    average of ?? hour
    customer stay an
    average of 0.5 hour
    (stays the same)
    average number of
    customers in the store
    at any time to be 10
    

    View Slide

23. Jakub Marchwicki <@kubem>
    in theory practice meets theory
    

    View Slide

24. Jakub Marchwicki <@kubem>
    @GetMapping("/loans/{loanId}")
    public LoanInformation loanDetails(@PathVariable("loanId") UUID loanId) {
    var riskInformation = riskClient.getRiskAssessmentDetails(loanId);
    var loanDetails = loansClient.getLoanDetails(loanId);
    var member = membersClient.getMemberDetails(loanDetails.getMemberId());
    return LoanInformation
    .fromLoanDetails(loanDetails)
    .requestId(UUID.randomUUID())
    .member(member)
    .riskAssessment(riskInformation)
    .build();
    }
    

    View Slide

25. Jakub Marchwicki <@kubem>
    Ideal world
    

    View Slide

26. Jakub Marchwicki <@kubem>
    Tomcat
    

    View Slide

27. Jakub Marchwicki <@kubem>
    Jetty
    

    View Slide

28. Jakub Marchwicki <@kubem>
    Undertow
    

    View Slide

29. Jakub Marchwicki <@kubem>
    Upon startup, Tomcat will create threads based on the value set for
    minSpareThreads (10) and increase that number based on demand, up to the
    number of maxThreads (200).
    If the maximum number of threads is reached, and all threads are busy, incoming
    requests are placed in a queue (acceptCount - 100) to wait for the next
    available thread.
    The server will only continue to accept a certain number of concurrent
    connections (as determined by maxConnections - 200).
    https://www.datadoghq.com/blog/tomcat-architecture-and-performance/
    

    View Slide

30. Jakub Marchwicki <@kubem>
    Undertow uses XNIO as the default connector. XNIO (...) default configuration (...)
    is I/O threads initialized to the number of your logical threads and the worker
    thread equal to 8 * CPU cores. So on typical 4 cores Intel CPU with
    hyper-threading you will end up with 8 I/O threads and 64 working threads.
    https://jmnarloch.wordpress.com/2016/04/26/spring-boot-tuning-your-undertow-application-for-throughput/
    

    View Slide

31. Jakub Marchwicki <@kubem>
    Undertow uses XNIO as the default connector. XNIO (...) default configuration (...)
    is I/O threads initialized to the number of your logical threads and t`he worker
    thread equal to 8 * CPU cores. So on typical 4 cores Intel CPU with
    hyper-threading you will end up with 8 I/O threads and 64 working threads.
    https://jmnarloch.wordpress.com/2016/04/26/spring-boot-tuning-your-undertow-application-for-throughput/
    var ioThreads = Math.max(
    Runtime.getRuntime().availableProcessors(), 2
    );
    var workerThreads = ioThreads * 8;
    number of processors
    available to the JVM
    number of logical cores:
    Core i7 w. HyperThreading: 8
    number of logical cores:
    Q6700: 4
    number of logical cores:
    docker --cpus=1
    on a quad core: 8
    Compare it to 100 or 1000,
    defaults at Tomcat or Jetty
    number of logical cores:
    docker --cpus=1
    on a quad core: 1 (JDK10)
    number of logical cores:
    docker --cpuset-cpus=0,1
    on a quad core: 2
    

    View Slide

32. TAKEAWAY
    Full stack developer doesn’t mean same
    technology on frontend and backend.
    Seniority comes from understanding layers
    beyond the code you craft.
    

    View Slide

33. Jakub Marchwicki <@kubem>
    not all threads were created equal
    requests
    

    View Slide

34. Jakub Marchwicki <@kubem>
    @GetMapping("/loans/{loanId}")
    public LoanInformation loanDetails(@PathVariable("loanId") UUID loanId) {
    var riskInformation = riskClient.getRiskAssessmentDetails(loanId);
    var loanDetails = loansClient.getLoanDetails(loanId);
    var member = membersClient.getMemberDetails(loanDetails.getMemberId());
    return LoanInformation
    .fromLoanDetails(loanDetails)
    .requestId(UUID.randomUUID())
    .member(member)
    .riskAssessment(riskInformation)
    .build();
    }
    on demand computation,
    take time (~600ms)
    these two are relatively
    fast, direct lookups (~150
    - 300 ms)
    

    View Slide

35. Jakub Marchwicki <@kubem>
    requests
    thread pool
    lookup member
    (~300ms)
    search members
    (~600ms)
    lookup loan
    (~150ms)
    

    View Slide

36. Jakub Marchwicki <@kubem>
    requests
    thread pool
    lookup member
    (~300ms)
    search members
    (~600ms)
    lookup loan
    (~150ms)
    

    View Slide

37. Jakub Marchwicki <@kubem>
    requests
    thread pool
    lookup member
    (~300ms)
    search members
    (~600ms)
    lookup loan
    (~150ms)
    

    View Slide

38. Jakub Marchwicki <@kubem>
    requests
    thread pool
    lookup member
    (~300ms)
    search members
    (~600ms)
    lookup loan
    (~150ms)
    

    View Slide

39. Jakub Marchwicki <@kubem>
    

    View Slide

40. Jakub Marchwicki <@kubem>
    

    View Slide

41. Jakub Marchwicki <@kubem>
    asynchronous, multithreaded, blocking
    

    View Slide

42. Jakub Marchwicki <@kubem>
    operations are done by choosing a worker thread from a thread pool
    the io thread is returned to the pool to run other requests, and process the
    upstream response asynchronously too
    the worker thread notifies the request thread when its work is complete.
    to offset these risks of backend latency, throttling mechanisms and circuit
    breakers help keep the blocking systems stable and resilient.
    

    View Slide

43. Jakub Marchwicki <@kubem>
    @GetMapping("/loans/{loanId}")
    public CompletableFuture loanDetails(@PathVariable("loanId") UUID loanId) {
    return supplyAsync(() -> loansClient.getLoanDetails(loanId), executor)
    .thenApply(l -> {
    Member memberDetails = membersClient.getMemberDetails(l.getMemberId());
    return Tuple.of(l, memberDetails);
    })
    .thenCombine(supplyAsync(() -> riskClient.getRiskAssessmentDetails(loanId), executor),
    (loanDetailsMember, riskInformation) -> LoanInformation
    .fromLoanDetails(loanDetailsMember.getLeft())
    .requestId(UUID.randomUUID())
    .member(loanDetailsMember.getRight())
    .riskAssessment(riskInformation)
    .build());
    }
    

    View Slide

44. Jakub Marchwicki <@kubem>
    requests
    thread pool
    lookup member
    (~300ms)
    search members
    (~600ms)
    lookup loan
    (~150ms)
    service
    thread pool
    

    View Slide

45. Jakub Marchwicki <@kubem>
    requests
    thread pool
    lookup member
    (~300ms)
    search members
    (~600ms)
    lookup loan
    (~150ms)
    service
    thread pool
    ?
    

    View Slide

46. TAKEAWAY
    Thread pool tuning is tied to what the
    application needs
    Understand the nature of the traffic
    

    View Slide

47. Jakub Marchwicki <@kubem>
    Worst case scenario??
    

    View Slide

48. Jakub Marchwicki <@kubem>
    Worst case scenario??
    https://www.nurkiewicz.com/2011/03/tenfold-increase-in-server-throughput.html
    

    View Slide

49. Jakub Marchwicki <@kubem>
    asynchronous, single-threaded, non-blocking
    

    View Slide

50. Jakub Marchwicki <@kubem>
    @GetMapping("/loans/{loanId}")
    public Observable loanDetails(@PathVariable("loanId") UUID loanId) {
    Single loanDetails = loansClient.getLoanDetailsSingle(loanId).cache();
    Single member = loanDetails
    .flatMap(l -> membersClient.getMemberDetailsSingle(l.getMemberId()));
    Single riskInformation = riskClient.getRiskAssessmentDetailsSingle(loanId);
    return Single.zip(
    loanDetails, member, riskInformation,
    (l, m, r) -> LoanInformation
    .fromLoanDetails(l)
    .requestId(UUID.randomUUID())
    .member(m)
    .riskAssessment(r)
    .build()
    ).toObservable();
    }
    

    View Slide

51. Jakub Marchwicki <@kubem>
    reactive programming
    does not build a reactive system
    

    View Slide

52. Jakub Marchwicki <@kubem>
    responsive handle requests in a reasonable time
    resilient stay responsive in the face of failures
    elastic scale up and down,
    be able to handle the load with minimal resources.
    message driven
    interactions using asynchronous message passing.
    a reactive system promise
    

    View Slide

53. Jakub Marchwicki <@kubem>
    a reactive programming - in technical terms
    Handling huge volumes of data in multi-userness environment
    Efficiency gains: data stays on the same CPU, use of CPU level caches,
    fewer context switches
    25% increase in throughput corresponding with a 25% reduction in CPU
    utilization
    

    View Slide

54. Jakub Marchwicki <@kubem>
    a reactive programming - in technical terms
    Handling huge volumes of data in multi-userness environment
    Efficiency gains: data stays on the same CPU, use of CPU level caches,
    fewer context switches
    25% increase in throughput corresponding with a 25% reduction in CPU
    utilization
    MOAR TRAFFIC!
    

    View Slide

55. Jakub Marchwicki <@kubem>
    the reactive promise - but...
    Blocking systems are easy to grok and debug
    a thread is always doing a single operation
    The event loop’s stack trace is meaningless when trying to follow a request
    Unhandled exceptions create dangling resources (exception swallow)
    

    View Slide

56. Jakub Marchwicki <@kubem>
    at what cost?
    

    View Slide

57. Jakub Marchwicki <@kubem>
    final List results = getQueries().stream() //there're 6 db queries
    .map(query -> db.apply(query))
    .sorted(naturalOrder())
    .collect(Collectors.toList());
    final List results = Observable.from(getQueries()) // there're 6 db queries
    .flatMap(query -> Async.start(() -> db.apply(query), scheduler))
    .toSortedList()
    .toBlocking()
    .single();
    final List results = new ArrayList<>();
    for (Query q: getQueries()) {
    String result = db.apply(q);
    results.add(result);
    }
    results.sort(naturalOrder());
    

    View Slide

58. Jakub Marchwicki <@kubem>
    [EL Warning]: 2009-08-29 12:53:13.718--Exception [EclipseLink-4002] (Eclipse Persistence
    Services - 1.1.2.v20090612-r4475): org.eclipse.persistence.exceptions.DatabaseException
    Internal Exception: java.sql.BatchUpdateException: The statement was aborted because it
    would have caused a duplicate key value in a unique or primary key constraint or unique
    index identified by 'SQL090829125312890' defined on 'REMINDISSUE'.
    Error Code: 20000
    Exception in thread "main" javax.persistence.RollbackException: Exception
    [EclipseLink-4002] (Eclipse Persistence Services - 1.1.2.v20090612-r4475):
    org.eclipse.persistence.exceptions.DatabaseException
    Internal Exception: java.sql.BatchUpdateException: The statement was aborted because it
    would have caused a duplicate key value in a unique or primary key constraint or unique
    index identified by 'SQL090829125312890' defined on 'REMINDISSUE'.
    Error Code: 20000
    at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commitIntern
    at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(Entit
    at com.example.dao.jpa.JpaDAO.commit(JpaDAO.java:99)
    at com.example.dao.jpa.JpaDAO.persist(JpaDAO.java:41)
    at com.example.dao.jpa.JpaDAO.persist(JpaDAO.java:1)
    at com.example.test.TestDAO.main(TestDAO.java:44)
    

    View Slide

59. Jakub Marchwicki <@kubem>
    public static void main(String[] args) {
    Observable.empty()
    .observeOn(Schedulers.io())
    .toBlocking()
    .first();
    }
    Exception in thread "main" java.util.NoSuchElementException: Sequence contains no el
    at rx.internal.operators.OperatorSingle$ParentSubscriber.onCompleted(OperatorSin
    at rx.internal.operators.OperatorTake$1.onCompleted(OperatorTake.java:53)
    at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.pollQueue(Operato
    at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber$1.call(OperatorOb
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$2
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sche
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:114
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:61
    at java.lang.Thread.run(Thread.java:745)
    

    View Slide

60. TAKEAWAY
    Both WebMVC / Servlet / synchronous
    and WebFlux / RxJava / reactive
    have a reason to exist.
    

    View Slide

61. TAKEAWAY
    Know your clients users and their flows
    Know your expectations what you optimizing for
    Know your domain
    Know your opportunity costs
    

    View Slide

62. Jakub Marchwicki <@kubem>
    Thank you!
    https://speakerdeck.com/kubamarchwicki/sync-or-async
    Jakub Marchwicki <@kubem>
    

    View Slide