Reactive Java for The Realtime

The demos are developed by IPT (http://iproduct.org/) for the BGOUG 2018 conference.

Sensor data is streamed in realtime from Arduino + accelerometeres, gyroscopes & compass 3D, ultrasound distance sensor, etc. using UDP protocol. The data processing is done with reactive Java alterantive implementations: callbacks, CompletableFutures and using Spring 5 Reactor library. The web 3D visualization with Three.js is streamed using Server Sent Events (SSE).

A video for the IoT demo is available @YouTube: https://www.youtube.com/watch?v=AB3AWAfcy9U

All source code of the demo is freely available @GitHub: https://github.com/iproduct/reactive-demos-iot

There are more reactive Java demos in the same repository - callbacks, CompletableFuture, realtime event streaming. Soon I'll add a description how to build the device and upload Arduino sketch, as well as describe CompletableFuture and Reactor demos and 3D web visualization part with Three.js. Please stay tuned :)

Transcript

1. November 16, 2018 BGOUG Autumn Conference Reactive Java for The Realtime Trayan Iliev tiliev@iproduct.org http://iproduct.org Copyright © 2003-2018 IPT - Intellectual Products & Technologies
2. 2 Reactive Javafor The Realtime  Java data streaming for the (soft) realtime applicatins  Async programming alternatives  Introduction to FRP, Reactive Streams spec  RxJava Project Reactor  REST services with JAX-RS and Reactor  End-to-end non-blocking reactive SOA with Netty  Realtime event streaming to JS clients using SSE  IoT event streaming demo
3. 3 Reactive Javafor The Realtime  Java data streaming for the (soft) realtime applicatins  Async programming alternatives  Introduction to FRP, Reactive Streams spec  RxJava Project Reactor  REST services with JAX-RS and Reactor  End-to-end non-blocking reactive SOA with Netty  Realtime event streaming to JS clients using SSE  IoT event streaming demo Best Explained in Code
4. About me 4 Trayan Iliev – CEO of IPT – Intellectual Products & Technologies (http://iproduct.org/) – Oracle® certified programmer 15+ Y – end-to-end reactive fullstack apps with Java, ES6/7, TypeScript, Angular, React and Vue.js – 12+ years IT trainer – Voxxed Days, jPrime, jProfessionals, BGOUG, BGJUG, DEV.BG speaker – Organizer RoboLearn hackathons and IoT enthusiast (http://robolearn.org)
5. 5 Since 2003: IT Education Evolved. Courses:  Java SE/Web/EE, JPA / Hibernate, Spring 5  Reactive event stream processing with Reactor / RxJava / RxJS  Node.js + Express + React + Redux + GraphQL  Angular + TypeScript + GraphQL  SOA & REST HATEOAS  DDD & Reactive Microservices IPT - Intellectual Products & Technologies http://www.iproduct.org
6. Where to Find the Demo Code? 6 Reactive IoT demos available @ GitHub: https://github.com/iproduct/reactive-demos-iot YouTube Video for the IoT demo: https://www.youtube.com/watch?v=AB3AWAfcy9U
7. Data / Event / Message Streams 7 “Conceptually, a stream is a (potentially never-ending) flow of data records, and a transformation is an operation that takes one or more streams as input, and produces one or more output streams as a result.” Apache Flink: Dataflow Programming Model
8. Data Stream Programming 8 The idea of abstracting logic from execution is hardly new -- it was the dream of SOA. And the recent emergence of microservices and containers shows that the dream still lives on. For developers, the question is whether they want to learn yet one more layer of abstraction to their coding. On one hand, there's the elusive promise of a common API to streaming engines that in theory should let you mix and match, or swap in and swap out. Tony Baer (Ovum) @ ZDNet - Apache Beam and Spark: New coopetition for squashing the Lambda Architecture?
9. Realtime Event Processing 9 Distributed realtime event processing becomes a hot topic:  IoT,  Service/process monitoring,  Realtime analytics, fraud detection  Click stream analytics  Stock-trading analysis  Supply chain and transportation alerts  ...
10. Lambda Architecture - I 10 https://commons.wikimedia.org/w/index.php?curid=34963986, By Textractor - Own work, CC BY-SA 4
11. Lambda Architecture - II 11 https://commons.wikimedia.org/w/index.php?curid=34963987, By Textractor - Own work, CC BY-SA 4
12. Lambda Architecture - III 12  Data-processing architecture designed to handle massive quantities of data by using both batch- and stream-processing methods  Balances latency, throughput, fault-tolerance, big data, real-time analytics, mitigates the latencies of map-reduce  Data model with an append-only, immutable data source that serves as a system of record  Ingesting and processing timestamped events that are appended to existing events. State is determined from the natural time-based ordering of the data.
13. Lambda Architecture: Projects - I 13  Apache Spark is an open-source cluster-computing framework. Spark Streaming, Spark Mllib  Apache Storm is a distributed stream processing – streams DAG  Apache Apex™ unified stream and batch processing engine.
14. Lambda Architecture: Projects - II  Apache Flink - open source stream processing framework – Java, Scala  Apache Kafka - open-source stream processing (Kafka Streams), real- time, low-latency, high-throughput, massively scalable pub/sub  Apache Beam – unified batch and streaming, portable, extensible
15. Direct Acyclic Graphs - DAG 15
16. Synchronous vs. Asynchronous IO 16 DB Synchronous A A B B DB Asynchronous A B C D A B C D
17. Example: Internet of Things (IoT) 17 CC BY 2.0, Source: https://www.flickr.com/photos/wilgengebroed/8249565455/ Radar, GPS, lidar for navigation and obstacle avoidance ( 2007 DARPA Urban Challenge )
18. IoT Services Architecture 18 Devices: Hardware + Embedded Software + Firmware UART/ I2C/ 2G/ 3G/ LTE/ ZigBee/ 6LowPan/ BLE Aggregation/ Bus: ESB, Message Broker Device Gateway: Local Coordination and Event Aggregation M2M: HTTP(/2) / WS / MQTT / CoAP Management: TR-069 / OMA-DM / OMA LWM2M HTTP, AMQP Cloud (Micro)Service Mng. Docker, Kubernetes/ Apache Brooklyn Web/ Mobile Portal PaaS Dashboard PaaS API: Event Processing Services, Analytics
19. 19  Performance is about 2 things (Martin Thompson – http://www.infoq.com/articles/low-latency-vp ): – Throughput – units per second, and – Latency – response time  Real-time – time constraint from input to response regardless of system load.  Hard real-time system if this constraint is not honored then a total system failure can occur.  Soft real-time system – low latency response with little deviation in response time  100 nano-seconds to 100 milli-seconds. [Peter Lawrey] What's High Performance?
20. 20  Callbacks – asynchronous methods do not have a return value but take an extra callback parameter (a lambda or anonymous class) that gets called when the result is available. Ex.: Swing’s EventListener  Futures, Promises – asynchronous methods return a (Completable)Future<T> immediately. The value is not immediately available, and the object can be polled Ex.: Callable<T> task  Reactive Streams (functional, non-blocking) – Observable (RxJava), Flowable (RxJava2), Flux & Mono (Project Reactor):  Composability and readability  Data as a flow manipulated with a rich vocabulary of operators  Lazy evaluation – nothing happens until you subscribe ()  Backpressure – consumer can signal to producer that the rate is high  High level but high value abstraction that is concurrency-agnostic How to Do Async Programming? Source: https://projectreactor.io/docs/core/release/reference
21. Futures in Java 8 - I 21  Future (implemented by FutureTask) – represents the result of an cancelable asynchronous computation. Methods are provided to check if the computation is complete, to wait for its completion, and to retrieve the result of the computation (blocking till its ready).  RunnableFuture – a Future that is Runnable. Successful execution of the run method causes Future completion, and allows access to its results.  ScheduledFuture – delayed cancelable action that returns result. Usually a scheduled future is the result of scheduling a task with a ScheduledExecutorService
22. Future Use Example 22 Future<String> future = executor.submit( new Callable<String>() { public String call() { return searchService.findByTags(tags); } } ); DoSomethingOther(); try { showResult(future.get()); // use future result } catch (ExecutionException ex) { cleanup(); }
23. Futures in Java 8 - II 23  CompletableFuture – a Future that may be explicitly completed (by setting its value and status), and may be used as a CompletionStage, supporting dependent functions and actions that trigger upon its completion.  CompletionStage – a stage of possibly asynchronous computation, that is triggered by completion of previous stage or stages (CompletionStages form Direct Acyclic Graph – DAG). A stage performs an action or computes value and completes upon termination of its computation, which in turn triggers next dependent stages. Computation may be Function (apply), Consumer (accept), or Runnable (run).
24. CompletableFuture Example - I 24 private CompletableFuture<String> longCompletableFutureTask(int i, Executor executor) { return CompletableFuture.supplyAsync(() -> { try { Thread.sleep(1000); // long computation :) } catch (InterruptedException e) { e.printStackTrace(); } return i + "-" + "test"; }, executor); }
25. CompletableFuture Example - II 25 ExecutorService executor = ForkJoinPool.commonPool(); //ExecutorService executor = Executors.newCachedThreadPool(); public void testlCompletableFutureSequence() { List<CompletableFuture<String>> futuresList = IntStream.range(0, 20).boxed() .map(i -> longCompletableFutureTask(i, executor) .exceptionally(t -> t.getMessage())) .collect(Collectors.toList()); CompletableFuture<List<String>> results = CompletableFuture.allOf( futuresList.toArray(new CompletableFuture[0])) .thenApply(v -> futuresList.stream() .map(CompletableFuture::join) .collect(Collectors.toList()) );
26. CompletableFuture Example - III 26 try { System.out.println(results.get(10, TimeUnit.SECONDS)); } catch (ExecutionException | TimeoutException | InterruptedException e) { e.printStackTrace(); } executor.shutdown(); } // OR just: System.out.println(results.join()); executor.shutdown(); Which is better?
27. CompletionStage 27  Computation may be Function (apply), Consumer (accept), or Runnable (run) – e.g.: completionStage.thenApply( x -> x * x ) .thenAccept(System.out::print ) .thenRun( System.out::println )  Stage computation can be triggered by completion of 1 (then), 2 (combine), or either 1 of 2 (either)  Functional composition can be applied to stages themselves instead to their results using compose  handle & whenComplete – support unconditional computation – both normal or exceptional triggering
28. CompletionStages Composition 28 public void testlCompletableFutureComposition() throws InterruptedException, ExecutionException { Double priceInEuro = CompletableFuture.supplyAsync(() -> getStockPrice("GOOGL")) .thenCombine(CompletableFuture.supplyAsync(() -> getExchangeRate(USD, EUR)), this::convertPrice) .exceptionally(throwable -> { System.out.println("Error: " + throwable.getMessage()); return -1d; }).get(); System.out.println("GOOGL stock price in Euro: " + priceInEuro ); }
29. More Demos ... 29 CompletableFuture, Flow & RxJava2 @ GitHub: https://github.com/iproduct/reactive-demos-java-9  completable-future-demo – composition, delayed, ...  flow-demo – custom Flow implementations using CFs  rxjava2-demo – RxJava2 intro to reactive composition  completable-future-jaxrs-cdi-cxf – async observers, ...  completable-future-jaxrs-cdi-jersey  completable-future-jaxrs-cdi-jersey-client
30. Ex.1: Async CDI Events with CF 30 @Inject @CpuProfiling private Event<CpuLoad> event; ... IntervalPublisher.getDefaultIntervalPublisher( 500, TimeUnit.MILLISECONDS) // Custom CF Flow Publisher .subscribe(new Subscriber<Integer>() { @Override public void onComplete() {} @Override public void onError(Throwable t) {} @Override public void onNext(Integer i) { event.fireAsync(new CpuLoad( System.currentTimeMillis(), getJavaCPULoad(), areProcessesChanged())) .thenAccept(event -> { logger.info("CPU load event fired: " + event); }); } //firing CDI async event returns CF @Override public void onSubscribe(Subscription subscription) {subscription.request(Long.MAX_VALUE);} });
31. Ex.2: Reactive JAX-RS Client - CF 31 CompletionStage<List<ProcessInfo>> processesStage = processes.request().rx() .get(new GenericType<List<ProcessInfo>>() {}) .exceptionally(throwable -> { logger.error("Error: " + throwable.getMessage()); return Collections.emptyList(); }); CompletionStage<Void> printProcessesStage = processesStage.thenApply(proc -> { System.out.println("Active JAVA Processes: " + proc); return null; });
32. Ex.2: Reactive JAX-RS Client - CF 32 (- continues -) printProcessesStage.thenRun( () -> { try (SseEventSource source = SseEventSource.target(stats).build()) { source.register(System.out::println); source.open(); Thread.sleep(20000); // Consume events for 20 sec } catch (InterruptedException e) { logger.info("SSE consumer interrupted: " + e); } }) .thenRun(() -> {System.exit(0);});
33. Listing Favs or Suggestions - Callbacks 33 userService.getFavorites(userId, new Callback<List<String>>() { public void onSuccess(List<String> list) { if (list.isEmpty()) { suggestionService.getSuggestions(new Callback<List<Favorite>>() { public void onSuccess(List<Favorite> list) { UiUtils.submitOnUiThread(() -> { list.stream().limit(5).forEach(uiList::show);}); } public void onError(Throwable error) { UiUtils.errorPopup(error); } }); } else { list.stream().limit(5) .forEach(favId -> favoriteService.getDetails(favId, new Callback<Favorite>() { public void onSuccess(Favorite details) { UiUtils.submitOnUiThread(() -> uiList.show(details)); } public void onError(Throwable error) { UiUtils.errorPopup(error);} } )); }} public void onError(Throwable error) { UiUtils.errorPopup(error); } } https://projectreactor.io/docs/core/release/reference/, Apache Software License 2.0
34. Listing Favs or Suggestions - Reactor 34 userService.getFavorites(userId) .timeout(Duration.ofMillis(800)) .onErrorResume(cacheService.cachedFavoritesFor(userId)) .flatMap(favoriteService::getDetails) .switchIfEmpty(suggestionService.getSuggestions()) .take(5) .publishOn(UiUtils.uiThreadScheduler()) .subscribe(uiList::show, UiUtils::errorPopup); }); https://projectreactor.io/docs/core/release/reference/, Apache Software License 2.0
35. Comb. Names & Stats – CompletableFuture 35 CompletableFuture<List<String>> ids = findIds(); CompletableFuture<List<String>> result = ids.thenComposeAsync(l -> { Stream<CompletableFuture<String>> zip = l.stream().map(i -> { CompletableFuture<String> nameTask = findName(i); CompletableFuture<Integer> statTask = findStat(i); return nameTask.thenCombineAsync(statTask, (name, stat) -> "Name " + name + " has stats " + stat); }); List<CompletableFuture<String>> combineList = zip.collect(Collectors.toList()); CompletableFuture<String>[] combineArray = combineList.toArray(new CompletableFuture[combineList.size()]); CompletableFuture<Void> allDone = CompletableFuture.allOf(combineArray); return allDone.thenApply(v -> combineList.stream() .map(CompletableFuture::join) .collect(Collectors.toList())); }); List<String> results = result.join(); https://projectreactor.io/docs/core/release/reference/, Apache Software License 2.0
36. Combined Names & Stats – Reactor 36 Flux<String> ids = findIds(); Flux<String> combinations = ids.flatMap(id -> { Mono<String> nameTask = findName(id); Mono<Integer> statTask = findStat(id); return nameTask.zipWith(statTask, (name, stat) -> "Name " + name + " has stats " + stat); }); Mono<List<String>> result = combinations.collectList(); List<String> results = result.block(); } https://projectreactor.io/docs/core/release/reference/, Apache Software License 2.0
37. Imperative and Reactive 37 We live in a Connected Universe ... there is hypothesis that all the things in the Universe are intimately connected, and you can not change a bit without changing all. Action – Reaction principle is the essence of how Universe behaves.
38. Imperative and Reactive  Reactive Programming: using static or dynamic data flows and propagation of change Example: a := b + c  Functional Programming: evaluation of mathematical functions, ➢ Avoids changing-state and mutable data, declarative programming ➢ Side effects free => much easier to understand and predict the program behavior. Example: books.stream().filter(book -> book.getYear() > 2010) .forEach( System.out::println )
39. Functional Reactive (FRP) 39 According to Connal Elliot's (ground-breaking paper @ Conference on Functional Programming, 1997), FRP is: (a) Denotative (b) Temporally continuous
40. Reactive Programming 40  Microsoft® opens source polyglot project ReactiveX (Reactive Extensions) [http://reactivex.io]: Rx = Observables + LINQ + Schedulers :) Java: RxJava, JavaScript: RxJS, C#: Rx.NET, Scala: RxScala, Clojure: RxClojure, C++: RxCpp, Ruby: Rx.rb, Python: RxPY, Groovy: RxGroovy, JRuby: RxJRuby, Kotlin: RxKotlin ...  Reactive Streams Specification [http://www.reactive-streams.org/] used by:  (Spring) Project Reactor [http://projectreactor.io/]  Actor Model – Akka (Java, Scala) [http://akka.io/]
41. Reactive Streams Spec. 41  Reactive Streams – provides standard for asynchronous stream processing with non-blocking back pressure.  Minimal set of interfaces, methods and protocols for asynchronous data streams  April 30, 2015: has been released version 1.0.0 of Reactive Streams for the JVM (Java API, Specification, TCK and implementation examples)  Java 9+: java.util.concurrent.Flow
42. Reactive Streams Spec. 42  Publisher – provider of potentially unbounded number of sequenced elements, according to Subscriber(s) demand. Publisher.subscribe(Subscriber) => onSubscribe onNext* (onError | onComplete)?  Subscriber – calls Subscription.request(long) to receive notifications  Subscription – one-to-one Subscriber ↔ Publisher, request data and cancel demand (allow cleanup).  Processor = Subscriber + Publisher
43. FRP = Async Data Streams 43  FRP is asynchronous data-flow programming using the building blocks of functional programming (e.g. map, reduce, filter) and explicitly modeling time  Used for GUIs, robotics, and music. Example (RxJava): Observable.from( new String[]{"Reactive", "Extensions", "Java"}) .take(2).map(s -> s + " : on " + new Date()) .subscribe(s -> System.out.println(s)); Result: Reactive : on Wed Jun 17 21:54:02 GMT+02:00 2015 Extensions : on Wed Jun 17 21:54:02 GMT+02:00 2015
44. Project Reactor 44  Reactor project allows building high-performance (low latency high throughput) non-blocking asynchronous applications on JVM.  Reactor is designed to be extraordinarily fast and can sustain throughput rates on order of 10's of millions of operations per second.  Reactor has powerful API for declaring data transformations and functional composition.  Makes use of the concept of Mechanical Sympathy built on top of Disruptor / RingBuffer.
45. Reactor Projects 45 https://github.com/reactor/reactor, Apache Software License 2.0 IPC – Netty, Kafka, Aeron
46. Reactor Flux 46 https://github.com/reactor/reactor-core, Apache Software License 2.0
47. Example: Flux.combineLatest() 47 https://projectreactor.io/core/docs/api/, Apache Software License 2.0
48. Source: RxJava 2 API documentation, http://reactivex.io/RxJava/2.x/javadoc/ Redux == Rx Scan Opearator
49. Hot and Cold Event Streams 49  PULL-based (Cold Event Streams) – Cold streams (e.g. RxJava Observable / Flowable or Reactor Flow / Mono) are streams that run their sequence when and if they are subscribed to. They present the sequence from the start to each subscriber.  PUSH-based (Hot Event Streams) – emit values independent of individual subscriptions. They have their own timeline and events occur whether someone is listening or not. When subscription is made observer receives current events as they happen. Example: mouse events
50. Converting Cold to Hot Stream 50 Source: RxJava 2 API documentation, http://reactivex.io/RxJava/2.x/javadoc/
51. Hot Stream Example - Reactor 51 EmitterProcessor<String> emitter = EmitterProcessor.create(); FluxSink<String> sink = emitter.sink(); emitter.publishOn(Schedulers.single()) .map(String::toUpperCase) .filter(s -> s.startsWith("HELLO")) .delayElements(Duration.of(1000, MILLIS)) .subscribe(System.out::println); sink.next("Hello World!"); // emit - non blocking sink.next("Goodbye World!"); sink.next("Hello Trayan!"); Thread.sleep(3000);
52. Reactor: Best Expalined in Code 52 Lets see some Reactive IoT demos @ GitHub: https://github.com/iproduct/reactive-demos-iot YouTube Video for the IoT demo: https://www.youtube.com/watch?v=AB3AWAfcy9U
53. Druid Distributed Data Store (Java) 53 https://commons.wikimedia.org/w/index.php?curid=33899448 By Fangjin Yang - sent to me personally, GFDL https://en.wikipedia.org/wiki/File:Flight_dynamics_with_text.png
54. Druid Distributed Data Store (Java) 54 https://commons.wikimedia.org/w/index.php?curid=33899448 By Fangjin Yang - sent to me personally, GFDL https://en.wikipedia.org/wiki/File:Centrale- intertielle_missile_S3_Musee_du_Bourget_P1010652.JPG
55. 55 Now much lighter and smaller - data is available in realtime thanks to reactive JAVA
56. Example: IPTPI - RPi + Ardunio Robot 56  Raspberry Pi 2 (quad-core ARMv7 @ 900MHz) + Arduino Leonardo cloneA-Star 32U4 Micro  Optical encoders (custom), IR optical array, 3D accelerometers, gyros, and compass MinIMU-9 v2  IPTPI is programmed in Java using Pi4J, Reactor, RxJava, Akka  More information about IPTPI: http://robolearn.org/iptpi-robot/
57. IPTPI Hot Event Streams Example 57 Encoder Readings ArduinoData Flux Arduino SerialData Position Flux Robot Positions Command Movement Subscriber RobotWSService (using Reactor) Angular 2 / TypeScript MovementCommands
58. Interested?-> Welcome to IPT Spring 5 & Reactor Course 58
59. Thank’s for Your Attention! 59 Trayan Iliev CEO of IPT – Intellectual Products & Technologies http://iproduct.org/ http://robolearn.org/ https://github.com/iproduct https://twitter.com/trayaniliev https://www.facebook.com/IPT.EACAD https://plus.google.com/+IproductOrg