Eclipse Vert.x for Dj fun and for profit!

Talk on Vert.x given at the JUG Summer Camp 2017 / La Rochelle, France.

Transcript

1. Eclipse Vert.x for Dj fun and for profit! Speaker : Julien Ponge - @jponge
2. Julien Ponge Maitre de Conférences “Delegated consultant to Red Hat” on Vert.x Eclipse Golo + extensive F/OSS background ! https://julien.ponge.org/ " @jponge # @jponge  https://www.mixcloud.com/hclcast/
3. Outline ✓ Reactive? Vert.x? ✓ Vert.x core ✓ Reactive programming with Vert.x / RxJava ✓ BoilerVroom
4. Reactive? Vert.x?
6. Application
7. Software Messages Requests Metrics Availability
8. Reactive systems Reactive streams Reactive programming Reactive “Responding to stimuli” Manifesto, Actor, Messages Resilience, Elasticity, Scalability, Asynchronous, non-blocking Data flow Back-pressure Non-blocking Data flow Events, Observable Spreadsheets Akka, Vert.x Akka Streams, Rx v2, Reactor, Vert.x Reactor, Reactive Spring, RxJava, Vert.x
9. Eclipse Vert.x Open source project started in 2012 Created by Tim Fox Eclipse / Apache licensing A toolkit for building reactive applications for the JVM Built on top of Netty ! https://vertx.io " @vertx_project
11. (demo) “Hello world in action”
12. Vert.x core — 101
13. Outline ✓ Vert.x concurrency model ✓ Message passing on the event bus ✓ Dealing with asynchronous events ✓ Failover demo
14. Vert.x Concurrency Model
15. while (isRunning) { String line = bufferedReader.readLine(); switch (line.substring(0, 4)) { case "ECHO": bufferedWriter.write(line); break // ... // other cases ( ...) // ... default: bufferedWriter.write("UNKW Unknown command"); } }
16. x 1000 = %
17. C1 “When you have a line of text, call C2” Something else with no blocking call either C2
18. Events Thread Event Loop
19.  2 event-loops per CPU core by default
20.  Verticles ' ' ' public class SomeVerticle extends AbstractVerticle { @Override public void start() throws Exception { } @Override public void stop() throws Exception { } } class SomeVerticle : AbstractVerticle() { override fun start() { } override fun stop() { } } exports.vertxStart = function() { } exports.vertxStop = function() { }
21. ' ( Configuration ' ' Network events come from acceptor threads Verticle Deploy Deploy
22. ' Worker thread pool Blocking task ) executeBlocking Result
23. ' “Regular verticle” (on event-loop) ) ' Worker verticle (1 thread) Multi-thread worker verticle ' Worker pool
24. Message passing on the event bus
25. ' ' Http server verticle Database client verticle  ?
26. ' , ' Http server verticle Database client verticle  Event Bus - “Details for user 1234?” Send to “user.db”
27. ' , ' Http server verticle Database client verticle  Event Bus - “Details for user 1234?” Send to “user.db” Consume from “user.db”
28. ' , ' Http server verticle Database client verticle  Event Bus - , “Details for user 1234?” “{data}”
29. - Asynchronous messaging “foo.bar”, “foo-bar”, “foo/bar”, … Point to point (with possible response back) Publish / subscribe
30. - Distributed across Vert.x nodes Hazelcast, Ignite, Infinispan, … TCP bridge interface Go, Python, C, JavaScript, Swift, C#, … SockJS bridge Seamless frontend / backend messaging
31. , Headers DeliveryOptions (e.g., timeouts) Body Address Reply address
32. , “Primitive” types String, int, double, … JSON Object/Array Polyglot applications, clean boundaries Custom codecs For advanced usages
33. EventBus eb = vertx.eventBus(); eb.consumer("ping-address", message -> { System.out.println("Received message: " + message.body()); message.reply("pong!"); }); EventBus eb = vertx.eventBus(); vertx.setPeriodic(1000, v -> { eb.send("ping-address", "ping!", reply -> { if (reply.succeeded()) { System.out.println("Received reply " + reply.result().body()); } else { System.out.println("No reply"); } }); });
34. switch (message.headers().get("action")) { case "all-pages": fetchAllPages(message); break; case "get-page": fetchPage(message); break; case "create-page": createPage(message); break; case "save-page": savePage(message); break; case "delete-page": deletePage(message); break; default: message.fail(ErrorCodes.BAD_ACTION.ordinal(), "Bad action: " + action); }
35. private void deletePage(Message<JsonObject> message) { dbClient.getConnection(car -> { if (car.succeeded()) { SQLConnection connection = car.result(); JsonArray data = new JsonArray().add(message.body().getString("id")); connection.updateWithParams(sqlQueries.get(SqlQuery.DELETE_PAGE), data, res -> { connection.close(); if (res.succeeded()) { message.reply(new JsonObject().put("result", "ok")); } else { reportQueryError(message, res.cause()); } }); } else { reportQueryError(message, car.cause()); } }); }
36. switch (message.headers().get("action")) { case "all-pages": fetchAllPages(message); break; case "get-page": fetchPage(message); break; case "create-page": createPage(message); break; case "save-page": savePage(message); break; case "delete-page": deletePage(message); break; default: message.fail(ErrorCodes.BAD_ACTION.ordinal(), "Bad action: " + action); } . If lots of actions…
37. @ProxyGen public interface WikiDatabaseService { // ( ...) @Fluent WikiDatabaseService savePage(int id, String markdown, Handler<AsyncResult<Void >> resultHandler); @Fluent WikiDatabaseService deletePage(int id, Handler<AsyncResult<Void >> resultHandler); static WikiDatabaseService createProxy(Vertx vertx, String address) { return new WikiDatabaseServiceVertxEBProxy(vertx, address); } // ( ...) } Proxy + handler source code will be generated Parameters from a JSON document Handlers for replies Generated proxy
38. dbService = WikiDatabaseService.createProxy(vertx, "wikidb.queue"); private void pageDeletionHandler(RoutingContext context) { dbService.deletePage(Integer.valueOf(context.request().getParam("id")), reply -> { if (reply.succeeded()) { context.response().setStatusCode(303); context.response().putHeader("Location", "/"); context.response().end(); } else { context.fail(reply.cause()); } }); }
39. Dealing with asynchronous events
40. Callbacks RxJava 1 + 2 Quasar (vertx-sync) Kotlin coroutines (core) (codegen) (codegen) (codegen) (codegen)
41. foo.a(1, res1 -> { if (res1.succeeded()) { bar.b("abc", 1, res2 -> { if (res.succeeded()) { baz.c(res3 -> { dosomething(res1, res2, res3, res4 -> { // (...) }); }); } }); } }); “Callback hell”
42. jdbc.rxGetConnection().flatMap(conn -> { // Now chain some statements using flatmap composition Single<ResultSet> resa = conn.rxUpdate("CREATE TABLE test(col VARCHAR(20))") .flatMap(result -> conn.rxUpdate("INSERT INTO test (col) VALUES ('val1')")) .flatMap(result -> conn.rxUpdate("INSERT INTO test (col) VALUES ('val2')")) .flatMap(result -> conn.rxQuery("SELECT * FROM test")); return resa.doAfterTerminate(conn ::close); }).subscribe(resultSet -> { // Subscribe to the final result System.out.println("Results : " + resultSet.getRows()); }, err -> { System.out.println("Database problem"); err.printStackTrace(); });
43. try (SQLConnection conn = awaitResult(jdbc ::getConnection)) { // Create a table Void v = awaitResult(h -> conn.execute("CREATE TABLE test(col VARCHAR(20))", h)); // Insert some stuff for (int i = 0; i < 10; i ++) { int ii = i; UpdateResult res = awaitResult(h -> conn.update("INSERT INTO test (col) VALUES ('val" + ii + "')", h)); System.out.println("Rows updated: " + res.getUpdated()); } // Select the results ResultSet res = awaitResult(h -> conn.query("SELECT * FROM test", h)); System.out.println("Selected " + res.getNumRows() + " results"); res.getResults().forEach(System.out ::println); }
44. val consumer = vertx.eventBus().localConsumer<String>("a.b.c") consumer.handler { message -> println("Consumer received: ${message.body()}") message.reply("pong") } // Wait until the consumer has properly registered asyncResult<Void> { h -> consumer.completionHandler(h) } // Send a message and wait for a reply val reply = asyncResult<Message<String >> { h -> vertx.eventBus().send("a.b.c", "ping", h) } println("Reply received: ${reply.body()}")
45. Reactive Programming with Vert.x and RxJava
46. RxJava Data and events flows Organising transformation of data and coordination of events Makes most sense with many sources of events
47. Motivation Future<List<T >> is not appropriate Dealing with latencies Functional programming influence
48. Push based subscribe ➊ ➋push Observer Observable
49. Iterable / Observable try {
 for (String item : it) { ➊
 } ➌
 } catch (Throwable e) { ➋
 } observable.subscribe(item -> {
 ➊ // onNext
 }, error -> {
 ➋ // onError
 }, () -> {
 ➌ // onCompleted
 });
50. 0 1 0..n Reactive Completable Single<T> Observable<T> Interactive void T Iterable<T>
55. Rxified APIs Each API type (annotated with @VertxGen) has its prefix io.vertx replaced by io.vertx.rxjava io.vertx.core.Vertx / io.vertx.rxjava.Vertx etc…
56. Rxified Handler<AsyncResult> void listen(int port, Handler<AsyncResult<HttpServer>> ar)
 
 Single<HttpServer> rxListen(int port);
57. (demo) RxJava
58. RxJava 2 Better performances Based on reactive-streams Null values forbidden More reactive types Observable / Flowable Single / Maybe / Completable Upcoming in 3.5.0
59. Boiler Vroom 0
60. Boiler Vroom, unplugged 1
61. MIDI Signals Channel control Note on / off Start / Stop Pitchbend Clock (…) https://github.com/cotejp/webmidi https://webaudio.github.io/web-midi-api/ https://github.com/jponge/webmidi-sequencer # https://github.com/jponge/boiler-vroom 24 clock messages per bar
62. MIDI In / Out mappings
63. 2 - WebMidi JSON / Event Bus
64. 3 4 5 6 IceCast VLC 2 Chrome / DJ Booth app Clients 7 NuProcess MIDI Event Bus Event Bus
65. 3 4 5 6 IceCast VLC Clients 7 OGG/Vorbis OGG/Vorbis MP3 MP3 HTTP 1.0 HTTP 1.1 HTTP Client Event bus → Chunked HTTP
67. Future<Void> icecastFuture = Future.future(); icecastProcess = Process.create(vertx, "icecast", asList("-c", “etc/icecast.xml")); icecastProcess.start(process -> icecastFuture.complete()); icecastProcess.exitHandler(exitCode -> { String message = "icecast exited with status code " + exitCode; if (!icecastFuture.isComplete()) { icecastFuture.fail(message); } logger.warn(message); }); icecastProcess.stdout().handler(logger ::info); icecastProcess.stderr().handler(logger ::warn); Reactive processes
68. HttpClientOptions options = new HttpClientOptions() .setDefaultHost("localhost") .setDefaultPort(8001); vertx.createHttpClient(options) .getNow("/stream.mp3", response -> { eventBus.publish(ANNOUNCE_DESTINATION, CONNECTED_MESSAGE); logger.info("Connected to the VLC transcoder"); response.handler(buffer -> eventBus.publish("boilervroom.audiostream", buffer)); response.exceptionHandler(t -> { logger.error(t); eventBus.publish(ANNOUNCE_DESTINATION, DISCONNECTED_MESSAGE); }); response.endHandler(v -> { logger.warn("Connection closed to the VLC transcoder"); eventBus.publish(ANNOUNCE_DESTINATION, DISCONNECTED_MESSAGE); }); }); Icecast to subscribers
69. router.get("/audiostream").handler(context -> { logger.info("New streaming client: " + context.request().remoteAddress()); HttpServerResponse response = context.response(); response.setStatusCode(200); response.setChunked(true); response.putHeader("Content-Type", "audio/mpeg"); MessageConsumer<Buffer> consumer = eventBus.consumer("boilervroom.audiostream"); consumer.bodyStream().handler(buffer -> { if (!response.writeQueueFull()) { response.write(buffer); } }); ( ...) Back-pressure
70. SockJSHandler sockJSHandler = SockJSHandler.create(vertx); PermittedOptions permittedOptions = new PermittedOptions() .setAddressRegex(“boilervroom \\ ..+"); BridgeOptions bridgeOptions = new BridgeOptions() .addInboundPermitted(permittedOptions) .addOutboundPermitted(permittedOptions); sockJSHandler.bridge(bridgeOptions); router.route("/eventbus /*").handler(sockJSHandler); SockJS event bus bridge: “killer feature” SockJS Event Bus
71. const eventBus = new EventBus("/eventbus") traktorIn.addListener("controlchange", 5, (event) => { eventBus.publish("boilervroom.vu-meter", { type: "volume-level", value: event.value }) }) eventBus.publish("boilervroom.fromtraktor", { type: "filter", number: 1, value: (event.value !== 0) })
72. ITopic<Object> topic = Hazelcast.newHazelcastInstance().getTopic("vu-meter"); vertx.eventBus().consumer("boilervroom.vu-meter", event -> { JsonObject body = (JsonObject) event.body(); topic.publish(body.getInteger("value")); }); Publish audio levels using Hazelcast
74. Outro
75. Vert.x Awesome Vert.x Stack Vert.x Core
77. Unified end-to-end reactive model + ecosystem (not just APIs…) For all kinds of distributed applications (even the small-scale ones) Flexible toolkit, not a framework (your needs, your call)
78. https: //youtu.be/ZkWsilpiSqw 8 Applications réactives avec Eclipse Vert.x 9 Building Reactive Microservices in Java https: //goo.gl/ep6yB9 https: //youtu.be/ApGNp4uHKdY 8 Microservices réactifs avec Eclipse Vert.x et Kubernetes 9 Guide to async programming with Vert.x for Java developers https: //goo.gl/AcWW3A