Slide 1
Slide 1 text
November 16, 2018
BGOUG Autumn Conference
Reactive Java
for The Realtime
Trayan Iliev
[email protected]
http://iproduct.org
Copyright © 2003-2018 IPT - Intellectual
Products & Technologies
Slide 2
Slide 2 text
2
Reactive Javafor The Realtime
Java data streaming for the (soft) realtime applicatins
Async programming alternatives
Introduction to FRP, Reactive Streams spec
RxJava Project Reactor
REST services with JAX-RS and Reactor
End-to-end non-blocking reactive SOA with Netty
Realtime event streaming to JS clients using SSE
IoT event streaming demo
Slide 3
Slide 3 text
3
Reactive Javafor The Realtime
Java data streaming for the (soft) realtime applicatins
Async programming alternatives
Introduction to FRP, Reactive Streams spec
RxJava Project Reactor
REST services with JAX-RS and Reactor
End-to-end non-blocking reactive SOA with Netty
Realtime event streaming to JS clients using SSE
IoT event streaming demo
Best Explained in Code
Slide 4
Slide 4 text
About me
4
Trayan Iliev
– CEO of IPT – Intellectual Products &
Technologies (http://iproduct.org/)
– Oracle® certified programmer 15+ Y
– end-to-end reactive fullstack apps with Java,
ES6/7, TypeScript, Angular, React and Vue.js
– 12+ years IT trainer
– Voxxed Days, jPrime, jProfessionals,
BGOUG, BGJUG, DEV.BG speaker
– Organizer RoboLearn hackathons and IoT
enthusiast (http://robolearn.org)
Slide 5
Slide 5 text
5
Since 2003: IT Education Evolved. Courses:
Java SE/Web/EE, JPA / Hibernate, Spring 5
Reactive event stream processing with Reactor /
RxJava / RxJS
Node.js + Express + React + Redux + GraphQL
Angular + TypeScript + GraphQL
SOA & REST HATEOAS
DDD & Reactive Microservices
IPT - Intellectual Products & Technologies
http://www.iproduct.org
Slide 6
Slide 6 text
Where to Find the Demo Code?
6
Reactive IoT demos available @ GitHub:
https://github.com/iproduct/reactive-demos-iot
YouTube Video for the IoT demo:
https://www.youtube.com/watch?v=AB3AWAfcy9U
Slide 7
Slide 7 text
Data / Event / Message Streams
7
“Conceptually, a stream is a (potentially never-ending)
flow of data records, and a transformation is an
operation that takes one or more streams as input,
and produces one or more output streams as a
result.”
Apache Flink: Dataflow Programming Model
Slide 8
Slide 8 text
Data Stream Programming
8
The idea of abstracting logic from execution is hardly
new -- it was the dream of SOA. And the recent
emergence of microservices and containers shows that
the dream still lives on.
For developers, the question is whether they want to
learn yet one more layer of abstraction to their coding.
On one hand, there's the elusive promise of a common
API to streaming engines that in theory should let you
mix and match, or swap in and swap out.
Tony Baer (Ovum) @ ZDNet - Apache Beam and Spark:
New coopetition for squashing the Lambda Architecture?
Slide 9
Slide 9 text
Realtime Event Processing
9
Distributed realtime event processing becomes a hot
topic:
IoT,
Service/process monitoring,
Realtime analytics, fraud detection
Click stream analytics
Stock-trading analysis
Supply chain and transportation alerts
...
Slide 10
Slide 10 text
Lambda Architecture - I
10
https://commons.wikimedia.org/w/index.php?curid=34963986, By Textractor - Own work, CC BY-SA 4
Slide 11
Slide 11 text
Lambda Architecture - II
11
https://commons.wikimedia.org/w/index.php?curid=34963987, By Textractor - Own work, CC BY-SA 4
Slide 12
Slide 12 text
Lambda Architecture - III
12
Data-processing architecture designed to handle
massive quantities of data by using both batch- and
stream-processing methods
Balances latency, throughput, fault-tolerance, big
data, real-time analytics, mitigates the latencies of
map-reduce
Data model with an append-only, immutable data
source that serves as a system of record
Ingesting and processing timestamped events that are
appended to existing events. State is determined from
the natural time-based ordering of the data.
Slide 13
Slide 13 text
Lambda Architecture: Projects - I
13
Apache Spark is an open-source
cluster-computing framework. Spark
Streaming, Spark Mllib
Apache Storm is a distributed
stream processing – streams DAG
Apache Apex™ unified stream and
batch processing engine.
Slide 14
Slide 14 text
Lambda Architecture: Projects - II
Apache Flink - open source stream
processing framework – Java, Scala
Apache Kafka - open-source stream
processing (Kafka Streams), real-
time, low-latency, high-throughput,
massively scalable pub/sub
Apache Beam – unified batch and
streaming, portable, extensible
Slide 15
Slide 15 text
Direct Acyclic Graphs - DAG
15
Slide 16
Slide 16 text
Synchronous vs. Asynchronous IO
16
DB
Synchronous
A
A
B
B
DB
Asynchronous
A
B
C
D
A
B
C
D
Slide 17
Slide 17 text
Example: Internet of Things (IoT)
17
CC BY 2.0, Source:
https://www.flickr.com/photos/wilgengebroed/8249565455/
Radar, GPS, lidar for navigation and obstacle
avoidance ( 2007 DARPA Urban Challenge )
Slide 18
Slide 18 text
IoT Services Architecture
18
Devices: Hardware + Embedded Software + Firmware
UART/ I2C/ 2G/ 3G/ LTE/ ZigBee/ 6LowPan/ BLE
Aggregation/ Bus: ESB, Message Broker
Device Gateway: Local Coordination and Event Aggregation
M2M: HTTP(/2) / WS / MQTT / CoAP
Management: TR-069 / OMA-DM / OMA LWM2M
HTTP, AMQP
Cloud (Micro)Service Mng.
Docker, Kubernetes/
Apache Brooklyn
Web/ Mobile
Portal
PaaS
Dashboard
PaaS API: Event Processing Services, Analytics
Slide 19
Slide 19 text
19
Performance is about 2 things (Martin Thompson –
http://www.infoq.com/articles/low-latency-vp ):
– Throughput – units per second, and
– Latency – response time
Real-time – time constraint from input to response
regardless of system load.
Hard real-time system if this constraint is not honored then
a total system failure can occur.
Soft real-time system – low latency response with little
deviation in response time
100 nano-seconds to 100 milli-seconds. [Peter Lawrey]
What's High Performance?
Slide 20
Slide 20 text
20
Callbacks – asynchronous methods do not have a return value but take
an extra callback parameter (a lambda or anonymous class) that gets
called when the result is available. Ex.: Swing’s EventListener
Futures, Promises – asynchronous methods return a
(Completable)Future immediately. The value is not immediately
available, and the object can be polled Ex.: Callable task
Reactive Streams (functional, non-blocking) – Observable (RxJava),
Flowable (RxJava2), Flux & Mono (Project Reactor):
Composability and readability
Data as a flow manipulated with a rich vocabulary of operators
Lazy evaluation – nothing happens until you subscribe ()
Backpressure – consumer can signal to producer that the rate is high
High level but high value abstraction that is concurrency-agnostic
How to Do Async Programming?
Source: https://projectreactor.io/docs/core/release/reference
Slide 21
Slide 21 text
Futures in Java 8 - I
21
Future (implemented by FutureTask) – represents the
result of an cancelable asynchronous computation.
Methods are provided to check if the computation is
complete, to wait for its completion, and to retrieve the
result of the computation (blocking till its ready).
RunnableFuture – a Future that is Runnable.
Successful execution of the run method causes Future
completion, and allows access to its results.
ScheduledFuture – delayed cancelable action that
returns result. Usually a scheduled future is the result
of scheduling a task with a ScheduledExecutorService
Slide 22
Slide 22 text
Future Use Example
22
Future future = executor.submit(
new Callable() {
public String call() {
return searchService.findByTags(tags);
}
}
);
DoSomethingOther();
try {
showResult(future.get()); // use future result
} catch (ExecutionException ex) { cleanup(); }
Slide 23
Slide 23 text
Futures in Java 8 - II
23
CompletableFuture – a Future that may be explicitly
completed (by setting its value and status), and may
be used as a CompletionStage, supporting dependent
functions and actions that trigger upon its completion.
CompletionStage – a stage of possibly asynchronous
computation, that is triggered by completion of
previous stage or stages (CompletionStages form
Direct Acyclic Graph – DAG). A stage performs an
action or computes value and completes upon
termination of its computation, which in turn triggers
next dependent stages. Computation may be Function
(apply), Consumer (accept), or Runnable (run).
Slide 24
Slide 24 text
CompletableFuture Example - I
24
private CompletableFuture
longCompletableFutureTask(int i, Executor executor) {
return CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(1000); // long computation :)
} catch (InterruptedException e) {
e.printStackTrace();
}
return i + "-" + "test";
}, executor);
}
Slide 25
Slide 25 text
CompletableFuture Example - II
25
ExecutorService executor = ForkJoinPool.commonPool();
//ExecutorService executor = Executors.newCachedThreadPool();
public void testlCompletableFutureSequence() {
List> futuresList =
IntStream.range(0, 20).boxed()
.map(i -> longCompletableFutureTask(i, executor)
.exceptionally(t -> t.getMessage()))
.collect(Collectors.toList());
CompletableFuture> results =
CompletableFuture.allOf(
futuresList.toArray(new CompletableFuture[0]))
.thenApply(v -> futuresList.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList())
);
Slide 26
Slide 26 text
CompletableFuture Example - III
26
try {
System.out.println(results.get(10, TimeUnit.SECONDS));
} catch (ExecutionException | TimeoutException
| InterruptedException e) {
e.printStackTrace();
}
executor.shutdown();
}
// OR just:
System.out.println(results.join());
executor.shutdown();
Which is better?
Slide 27
Slide 27 text
CompletionStage
27
Computation may be Function (apply), Consumer
(accept), or Runnable (run) – e.g.:
completionStage.thenApply( x -> x * x )
.thenAccept(System.out::print )
.thenRun( System.out::println )
Stage computation can be triggered by completion of
1 (then), 2 (combine), or either 1 of 2 (either)
Functional composition can be applied to stages
themselves instead to their results using compose
handle & whenComplete – support unconditional
computation – both normal or exceptional triggering
Slide 28
Slide 28 text
CompletionStages Composition
28
public void testlCompletableFutureComposition() throws
InterruptedException, ExecutionException {
Double priceInEuro = CompletableFuture.supplyAsync(()
-> getStockPrice("GOOGL"))
.thenCombine(CompletableFuture.supplyAsync(() ->
getExchangeRate(USD, EUR)), this::convertPrice)
.exceptionally(throwable -> {
System.out.println("Error: " +
throwable.getMessage());
return -1d;
}).get();
System.out.println("GOOGL stock price in Euro: " +
priceInEuro );
}
Slide 29
Slide 29 text
More Demos ...
29
CompletableFuture, Flow & RxJava2 @ GitHub:
https://github.com/iproduct/reactive-demos-java-9
completable-future-demo – composition, delayed, ...
flow-demo – custom Flow implementations using CFs
rxjava2-demo – RxJava2 intro to reactive composition
completable-future-jaxrs-cdi-cxf – async observers, ...
completable-future-jaxrs-cdi-jersey
completable-future-jaxrs-cdi-jersey-client
Slide 30
Slide 30 text
Ex.1: Async CDI Events with CF
30
@Inject @CpuProfiling private Event event; ...
IntervalPublisher.getDefaultIntervalPublisher(
500, TimeUnit.MILLISECONDS) // Custom CF Flow Publisher
.subscribe(new Subscriber() {
@Override public void onComplete() {}
@Override public void onError(Throwable t) {}
@Override public void onNext(Integer i) {
event.fireAsync(new CpuLoad(
System.currentTimeMillis(), getJavaCPULoad(),
areProcessesChanged()))
.thenAccept(event -> {
logger.info("CPU load event fired: " + event);
}); } //firing CDI async event returns CF
@Override public void onSubscribe(Subscription
subscription) {subscription.request(Long.MAX_VALUE);} });
Slide 31
Slide 31 text
Ex.2: Reactive JAX-RS Client - CF
31
CompletionStage> processesStage =
processes.request().rx()
.get(new GenericType>() {})
.exceptionally(throwable -> {
logger.error("Error: " + throwable.getMessage());
return Collections.emptyList();
});
CompletionStage printProcessesStage =
processesStage.thenApply(proc -> {
System.out.println("Active JAVA Processes: " + proc);
return null;
});
Slide 32
Slide 32 text
Ex.2: Reactive JAX-RS Client - CF
32
(- continues -)
printProcessesStage.thenRun( () -> {
try (SseEventSource source =
SseEventSource.target(stats).build()) {
source.register(System.out::println);
source.open();
Thread.sleep(20000); // Consume events for 20 sec
} catch (InterruptedException e) {
logger.info("SSE consumer interrupted: " + e);
}
})
.thenRun(() -> {System.exit(0);});
Slide 33
Slide 33 text
Listing Favs or Suggestions - Callbacks
33
userService.getFavorites(userId, new Callback>() {
public void onSuccess(List list) {
if (list.isEmpty()) {
suggestionService.getSuggestions(new Callback>() {
public void onSuccess(List list) {
UiUtils.submitOnUiThread(() -> {
list.stream().limit(5).forEach(uiList::show);}); }
public void onError(Throwable error) { UiUtils.errorPopup(error); }
});
} else {
list.stream().limit(5)
.forEach(favId -> favoriteService.getDetails(favId,
new Callback() {
public void onSuccess(Favorite details) {
UiUtils.submitOnUiThread(() -> uiList.show(details)); }
public void onError(Throwable error) {
UiUtils.errorPopup(error);}
}
));
}}
public void onError(Throwable error) { UiUtils.errorPopup(error); }
}
https://projectreactor.io/docs/core/release/reference/, Apache Software License 2.0
Slide 34
Slide 34 text
Listing Favs or Suggestions - Reactor
34
userService.getFavorites(userId)
.timeout(Duration.ofMillis(800))
.onErrorResume(cacheService.cachedFavoritesFor(userId))
.flatMap(favoriteService::getDetails)
.switchIfEmpty(suggestionService.getSuggestions())
.take(5)
.publishOn(UiUtils.uiThreadScheduler())
.subscribe(uiList::show, UiUtils::errorPopup);
});
https://projectreactor.io/docs/core/release/reference/, Apache Software License 2.0
Slide 35
Slide 35 text
Comb. Names & Stats – CompletableFuture
35
CompletableFuture> ids = findIds();
CompletableFuture> result = ids.thenComposeAsync(l -> {
Stream> zip =
l.stream().map(i -> {
CompletableFuture nameTask = findName(i);
CompletableFuture statTask = findStat(i);
return nameTask.thenCombineAsync(statTask,
(name, stat) -> "Name " + name + " has stats " + stat); });
List> combineList =
zip.collect(Collectors.toList());
CompletableFuture[] combineArray =
combineList.toArray(new CompletableFuture[combineList.size()]);
CompletableFuture allDone =
CompletableFuture.allOf(combineArray);
return allDone.thenApply(v -> combineList.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList()));
});
List results = result.join();
https://projectreactor.io/docs/core/release/reference/, Apache Software License 2.0
Slide 36
Slide 36 text
Combined Names & Stats – Reactor
36
Flux ids = findIds();
Flux combinations =
ids.flatMap(id -> {
Mono nameTask = findName(id);
Mono statTask = findStat(id);
return nameTask.zipWith(statTask,
(name, stat) -> "Name " + name + " has stats " + stat);
});
Mono> result = combinations.collectList();
List results = result.block();
}
https://projectreactor.io/docs/core/release/reference/, Apache Software License 2.0
Slide 37
Slide 37 text
Imperative and Reactive
37
We live in a Connected Universe
... there is hypothesis that all
the things in the Universe are
intimately connected, and you
can not change a bit without
changing all.
Action – Reaction principle is
the essence of how Universe
behaves.
Slide 38
Slide 38 text
Imperative and Reactive
Reactive Programming: using static or dynamic data
flows and propagation of change
Example: a := b + c
Functional Programming: evaluation of mathematical
functions,
➢ Avoids changing-state and mutable data, declarative
programming
➢ Side effects free => much easier to understand and
predict the program behavior.
Example: books.stream().filter(book -> book.getYear() > 2010)
.forEach( System.out::println )
Slide 39
Slide 39 text
Functional Reactive (FRP)
39
According to Connal Elliot's (ground-breaking paper @
Conference on Functional Programming, 1997), FRP is:
(a) Denotative
(b) Temporally continuous
Slide 40
Slide 40 text
Reactive Programming
40
Microsoft® opens source polyglot project ReactiveX
(Reactive Extensions) [http://reactivex.io]:
Rx = Observables + LINQ + Schedulers :)
Java: RxJava, JavaScript: RxJS, C#: Rx.NET, Scala: RxScala,
Clojure: RxClojure, C++: RxCpp, Ruby: Rx.rb, Python: RxPY,
Groovy: RxGroovy, JRuby: RxJRuby, Kotlin: RxKotlin ...
Reactive Streams Specification
[http://www.reactive-streams.org/] used by:
(Spring) Project Reactor [http://projectreactor.io/]
Actor Model – Akka (Java, Scala) [http://akka.io/]
Slide 41
Slide 41 text
Reactive Streams Spec.
41
Reactive Streams – provides standard for
asynchronous stream processing with non-blocking
back pressure.
Minimal set of interfaces, methods and protocols for
asynchronous data streams
April 30, 2015: has been released version 1.0.0 of
Reactive Streams for the JVM (Java API,
Specification, TCK and implementation examples)
Java 9+: java.util.concurrent.Flow
Slide 42
Slide 42 text
Reactive Streams Spec.
42
Publisher – provider of potentially unbounded number
of sequenced elements, according to Subscriber(s)
demand.
Publisher.subscribe(Subscriber) => onSubscribe onNext*
(onError | onComplete)?
Subscriber – calls Subscription.request(long) to
receive notifications
Subscription – one-to-one Subscriber ↔ Publisher,
request data and cancel demand (allow cleanup).
Processor = Subscriber + Publisher
Slide 43
Slide 43 text
FRP = Async Data Streams
43
FRP is asynchronous data-flow programming using the
building blocks of functional programming (e.g. map,
reduce, filter) and explicitly modeling time
Used for GUIs, robotics, and music. Example (RxJava):
Observable.from(
new String[]{"Reactive", "Extensions", "Java"})
.take(2).map(s -> s + " : on " + new Date())
.subscribe(s -> System.out.println(s));
Result:
Reactive : on Wed Jun 17 21:54:02 GMT+02:00 2015
Extensions : on Wed Jun 17 21:54:02 GMT+02:00 2015
Slide 44
Slide 44 text
Project Reactor
44
Reactor project allows building high-performance (low
latency high throughput) non-blocking asynchronous
applications on JVM.
Reactor is designed to be extraordinarily fast and can
sustain throughput rates on order of 10's of millions of
operations per second.
Reactor has powerful API for declaring data
transformations and functional composition.
Makes use of the concept of Mechanical Sympathy
built on top of Disruptor / RingBuffer.
Slide 45
Slide 45 text
Reactor Projects
45
https://github.com/reactor/reactor, Apache Software License 2.0
IPC – Netty, Kafka, Aeron
Slide 46
Slide 46 text
Reactor Flux
46
https://github.com/reactor/reactor-core, Apache Software License 2.0
Slide 47
Slide 47 text
Example: Flux.combineLatest()
47
https://projectreactor.io/core/docs/api/, Apache Software License 2.0
Slide 48
Slide 48 text
Source: RxJava 2 API documentation, http://reactivex.io/RxJava/2.x/javadoc/
Redux == Rx Scan Opearator
Slide 49
Slide 49 text
Hot and Cold Event Streams
49
PULL-based (Cold Event Streams) – Cold streams (e.g.
RxJava Observable / Flowable or Reactor Flow / Mono)
are streams that run their sequence when and if they
are subscribed to. They present the sequence from the
start to each subscriber.
PUSH-based (Hot Event Streams) – emit values
independent of individual subscriptions. They have their
own timeline and events occur whether someone is
listening or not. When subscription is made observer
receives current events as they happen.
Example: mouse events
Slide 50
Slide 50 text
Converting Cold to Hot Stream
50
Source: RxJava 2 API documentation, http://reactivex.io/RxJava/2.x/javadoc/
Slide 51
Slide 51 text
Hot Stream Example - Reactor
51
EmitterProcessor emitter =
EmitterProcessor.create();
FluxSink sink = emitter.sink();
emitter.publishOn(Schedulers.single())
.map(String::toUpperCase)
.filter(s -> s.startsWith("HELLO"))
.delayElements(Duration.of(1000, MILLIS))
.subscribe(System.out::println);
sink.next("Hello World!"); // emit - non blocking
sink.next("Goodbye World!");
sink.next("Hello Trayan!");
Thread.sleep(3000);
Slide 52
Slide 52 text
Reactor: Best Expalined in Code
52
Lets see some Reactive IoT demos @ GitHub:
https://github.com/iproduct/reactive-demos-iot
YouTube Video for the IoT demo:
https://www.youtube.com/watch?v=AB3AWAfcy9U
Slide 53
Slide 53 text
Druid Distributed Data Store (Java)
53
https://commons.wikimedia.org/w/index.php?curid=33899448 By Fangjin Yang - sent to me personally, GFDL
https://en.wikipedia.org/wiki/File:Flight_dynamics_with_text.png
Slide 54
Slide 54 text
Druid Distributed Data Store (Java)
54
https://commons.wikimedia.org/w/index.php?curid=33899448 By Fangjin Yang - sent to me personally, GFDL
https://en.wikipedia.org/wiki/File:Centrale-
intertielle_missile_S3_Musee_du_Bourget_P1010652.JPG
Slide 55
Slide 55 text
55
Now much lighter and smaller -
data is available in realtime thanks
to reactive JAVA
Slide 56
Slide 56 text
Example: IPTPI - RPi + Ardunio Robot
56
Raspberry Pi 2 (quad-core ARMv7
@ 900MHz) + Arduino Leonardo
cloneA-Star 32U4 Micro
Optical encoders (custom), IR
optical array, 3D accelerometers,
gyros, and compass MinIMU-9 v2
IPTPI is programmed in Java
using Pi4J, Reactor, RxJava, Akka
More information about IPTPI:
http://robolearn.org/iptpi-robot/
Slide 57
Slide 57 text
IPTPI Hot Event Streams Example
57
Encoder
Readings
ArduinoData
Flux
Arduino
SerialData
Position
Flux
Robot
Positions
Command
Movement
Subscriber
RobotWSService
(using Reactor)
Angular 2 /
TypeScript
MovementCommands
Slide 58
Slide 58 text
Interested?-> Welcome to IPT Spring 5 &
Reactor Course
58
Slide 59
Slide 59 text
Thank’s for Your Attention!
59
Trayan Iliev
CEO of IPT – Intellectual Products
& Technologies
http://iproduct.org/
http://robolearn.org/
https://github.com/iproduct
https://twitter.com/trayaniliev
https://www.facebook.com/IPT.EACAD
https://plus.google.com/+IproductOrg