Practical non-blocking microservices in Java 8 - JDD

Talk at #JDDKrakow2016 with @MichalBalinski about non-blocking microservices

Transcript

1. Practical non-blocking microservices in Java 8 Michał Baliński System Architect Oleksandr Goldobin System Architect
2. Cloud of microservices for secure IoT gateway backing service core service gateway gateway core service backing service
3. The demand and characteristics of our domain Crowded IoT environment Slow, long lived connections Big amount of concurrent connections Scalability and resilience Rather I/O intensive than CPU intensive
4. OTA Gateway – Use Case TSM Trusted Service Manager
5. OTA Gateway – Use Case TSM Trusted Service Manager Security Module
6. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB
7. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts
8. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts
9. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts
10. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 4. submition response 3. store scripts
11. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 4. submition response HTTP 5. poll scripts 3. store scripts
12. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts 4. submition response HTTP 5. poll scripts 6. search scripts
13. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts 4. submition response HTTP 5. poll scripts 6. search scripts 7. response with scripts
14. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP HTTP HTTP TCP
15. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O Monitoring System HTTP
16. Let’s test blocking approach!
17. OTA Gateway Blocking - technologies 17 TSM DB Log Storage HTTP TCP STDOUT OTA Gateway JAX-RS Logback appender Security Module JAX-RS Jedis JAX-RS Client
18. OTA Gateway – Blocking - test OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms expected latency < 450 ms verified with throughput over 7k req/s
19. Blocking – 1k threads OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms max 1000 connections max 1000 threads expected latency < 450 ms
20. Blocking – 1k threads 500 req/s 1000 req/s 1500 req/s 2000 req/s
21. The drawbacks of classic synchronous I/O One thread per connection Threads waiting instead of running Context switches Resource wasting (~1 MB per thread - 64bit)
22. Let’s switch from blocking to non-blocking
23. OTA Gateway Non-blocking - technologies 23 TSM DB Log Storage HTTP TCP STDOUT OTA Gateway JAX-RS 2.0 Logback async appender Async Http Client 2 (Netty) Security Module JAX-RS 2.0 Lettuce (Netty)
24. Non-blocking – 16 threads OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms no limit for connections max 16 threads expected latency < 450 ms
25. Non-blocking – 16 threads 1k req/s 2k req/s 2.5k req/s 3k req/s
26. Blocking Non-blocking 1000 threads 56 threads 1.2 GB 0.5 GB ~1.2k r/s 2.5k r/s
27. Let’s talk about challenges
28. OTA Gateway Blocking – sequence diagram OTA Gateway TSM Security Module DB Logs loop 1. submit scripts 2. encrypt script 3a. store script 4. submition response 3b. count scripts
29. OTA Gateway Non-blocking – sequence diagram OTA Gateway TSM Security Module DB Logs loop 1. submit scripts 2. encrypt script 3a. store script 4. submition response 3b. count scripts
30. OTA Non-blocking – reality OTA Gateway TSM Security Module DB Logs „looped processing chain” 1. submit scripts 4. submition response 3b. count scripts HTTP Server 2. encrypt script 3a. store script Logging DB Client Security Client
31. Code. Bird view 31 10 October 2016 package org.demo.ota.blocking.rest; @Path("se") public class ScriptSubmissionResource extends Application { private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission"); private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance(); @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) { MDC.put("flow", "submission"); MDC.put("se", seId); return METRICS.instrument(() -> { log.debug("Processing {} scripts submission", scripts.size(), seId); for (int i = 0; i < scripts.size(); i++) { final Script script = scripts.get(i); log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient.encrypt(seId, script.getPayload()); script.setPayload(encryptedPayload); log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); } long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); log.debug("Request processed", seId); return numberOfScripts; }); } @Override public Set<Object> getSingletons() { return Collections.singleton(this); } } package org.demo.ota.nonblocking.rest; @Path("se") public class ScriptSubmissionResource extends Application { private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission"); private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance(); @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); METRICS.instrumentStage(() -> { log.debug("{} Processing {} scripts submission", diagnosticContext, scripts.size()); return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) ); }) .whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); } }); } private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts ) { CompletionStage<Void> stage = null; // <- non final field, potential concurrent access bug! for (int i = 0; i < scripts.size(); i++) { final int scriptIndex = i; final Script script = scripts.get(scriptIndex); if (stage == null) { stage = encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script)); } } return stage; } private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script ) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex); return secureModuleClient .encrypt(seId, script.getPayload()) .thenCompose( encryptedPayload -> { log.debug("{} Storing encrypted script {}", diagnosticContext, scriptIndex); return scriptStorageClient.storeScript(seId, new Script(encryptedPayload)); } ); } @Override public Set<Object> getSingletons() { return new HashSet<>(Collections.singletonList(this)); } }
32. Code. Blocking. Submission 1 10 October 2016 32 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) { MDC.put("flow", "submission"); // ß Setting diagnostic context MDC.put("se", seId); return METRICS.instrument(() -> { // ß Instrumenting with metrics //...
33. Code. Blocking. Submission 2 10 October 2016 33 log.debug("Processing {} scripts submission", scripts.size(), seId); for (int i = 0; i < scripts.size(); i++) { ß Cycle through the scripts final Script script = scripts.get(i); log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient .encrypt(seId, script.getPayload()); ß Encrypting the script script.setPayload(encryptedPayload); log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); ß Saving the script into DB } long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); ß Getting current number of scripts in DB log.debug("Request processed", seId); return numberOfScripts;
34. Code. Non-blocking. Submission 1 10 October 2016 34 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); ß Creating diagnostic context METRICS.instrumentStage(() -> { ß Instrumenting with metrics
35. Code. Non-blocking. Submission 1 10 October 2016 35 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); ß Creating diagnostic context METRICS.instrumentStage(() -> { ß Instrumenting with metrics
36. Code. Non-blocking. Submission 2 10 October 2016 36 METRICS.instrumentStage(() -> { log.debug( "{} Processing {} scripts submission", diagnosticContext, scripts.size()); return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) ); }) .whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); } });
37. Code. Non-blocking. Submission 3 10 October 2016 37 private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts ) { CompletionStage<Void> stage = null; // <- non final field, potential // concurrent access bug! for (int i = 0; i < scripts.size(); i++) { ß Cycle through the scripts final int scriptIndex = i; final Script script = scripts.get(scriptIndex); if (stage == null) { stage = encryptAndStoreSingleScript( diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript( diagnosticContext, seId, scriptIndex, script)); } } return stage; }
38. Code. Non-blocking. Submission 4 10 October 2016 38 private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script ) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex); return secureModuleClient .encrypt(seId, script.getPayload()) ß Encrypting the script .thenCompose( encryptedPayload -> { log.debug( "{} Storing encrypted script {}", diagnosticContext, scriptIndex); return scriptStorageClient.storeScript( ß Saving the script into seId, the DB new Script(encryptedPayload)); } ); }
39. Code. Blocking. Integration 10 October 2016 39 private final JedisPool pool; public void storeScript(String seId, Script script) { try (Jedis jedis = pool.getResource()) { jedis.rpush(seId, script.getPayload()); } } public long numberOfScriptsForSe(String seId) { try (Jedis jedis = pool.getResource()) { return jedis.llen(seId); } } public Optional<Script> nextScript(String seId) { try (Jedis jedis = pool.getResource()) { return Optional.ofNullable(jedis.lpop(seId)).map(Script::new); } } public String encrypt(String keyDiversifier, String payload) { // ... }
40. Code. Non-Blocking. Integration 10 October 2016 40 private final RedisAsyncCommands<String, String> commands; public CompletionStage<Void> storeScript(String seId, Script script) { return commands .rpush(seId, script.getPayload()) .thenApply(ignore -> null); } public CompletionStage<Long> numberOfScriptsForSe(String seId) { return commands.llen(seId); } public CompletionStage<Optional<String>> nextScript(String seId) { return commands.lpop(seId).thenApply(Optional::ofNullable); } public CompletionStage<String> encrypt(String keyDiversifier, String payload) { // ... }
41. Diagnostics in non-blocking systems No clear stack traces, need for good logs Name your threads properly MDC becomes useless (thread locals) Explicitly pass debug context to trace flows Use metrics Be prepared for debugging non-obvious errors
42. NIO technology landscape JDK 1.4 NIO JDK 1.7 NIO.2 JAX-RS 2.x Servlet API 3.x
43. Lessons learned, part 1 Vanila Java 8 for NIO µ-services Netty best for custom protocols in NIO Unit tests should be synchronous Load/stress testing is a must Make bulkheading and plan your resources
44. Lessons learned, part 2 Functional programming patterns for readability Immutability as 1-st class citizen Scala may be good choice ;-)
45. Conclusion Non-blocking processing can really save your resources (== money) But! Weight all pros and cons and use non-blocking processing only if you really need it.
46. Thank you. Michał Baliński m.balinski@oberthur.com Oleksandr Goldobin o.goldobin@oberthur.com goldobin @goldobin balonus @MichalBalinski Readings: • https://github.com/balonus/blocking-vs-nonblocking-demo • C10k Problem, C10M Problem, Asynchronous I/O • Boost application performance using asynchronous I/O (M. Tim Jones, 2006) • Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking Systems (Netflix, 2016) • Thousands of Threads and Blocking I/O: The Old Way to Write Java Servers Is New Again (and Way Better) (Paul Tyma, 2008) • Why Non-Blocking? (Bozhidar Bozhanov, 2011)