Reactive Patterns and Distributed Systems - OUGN 2018

Slide 1

Slide 1 text

reactive patterns and distributed systems arturo.viveros@sysco.no jorge.quilcate@sysco.no

Slide 2

Slide 2 text

agenda ● Intro to Reactive Systems ● Back-Pressure ● Stream-Processing ● Observability

Slide 3

Slide 3 text

Reactive Systems

Slide 4

Slide 4 text

data on the outside vs data on the inside “Going [from monolithic architecture] to SOA is like going from Newton’s physics to Einstein’s physics. Newton’s time marched forward uniformly with instant knowledge at a distance. Before SOA, distributed computing strove to make many systems look like one with RPC, 2PC, etc [...]

Slide 5

Slide 5 text

data on the outside vs data on the inside [...] In Einstein’s universe, everything is relative to one’s perspective. SOA has “now” inside and the “past” arriving in messages.” - Pat Helland

Slide 6

Slide 6 text

data on the outside vs data on the inside “perhaps we should rename the “extract microservice” refactoring operation to “change model of time and space” ;).” - Adrian Colyer

Slide 7

Slide 7 text

distributed systems & complexity

Slide 8

Slide 8 text

“we want more, of everything, and we want it now” - YOUR CUSTOMERS

Slide 9

Slide 9 text

Reactive Manifesto Value Form Means

Slide 10

Slide 10 text

Adopting Reactive Patterns (i.e. Ivy Pattern) -Why Reactive?, O’Reilly, Konrad Malawski

Slide 11

Slide 11 text

Back-Pressure

Slide 12

Slide 12 text

What is Back Pressure? • Systems should be designed and implemented to react assertively under sustained load • This is often neglected although such conditions are usually expected and highly predictable • Negotiating back pressure usually requires some sort of Flow Control • Which in simpler terms is the ability to say “hey, slow down!!”, or even “stop it!!”

Slide 13

Slide 13 text

There are Several types of Back Pressure • Fast publisher / slow subscriber • Insufficient buffer • Accumulation without discharge • Packet size variation • Clogged channel • Etc.

Slide 14

Slide 14 text

Dealing with Back Pressure within Integrations is quite Important… Failing to account for it can have a “domino effect”, overloading and eventually crippling many of the participating systems.

Slide 15

Slide 15 text

Meterpoint Data Collection IoT Devices Customer Management Work Order Management Delivery Experience Web Data Warehouse System AQ REST / FTP SOAP Event Streams REST CQRS DSO Central Authorities Grid Companies REST Mobile The more Complex the Integration, the more Important (and challenging) it is to Handle Back Pressure

Slide 16

Slide 16 text

There are Several ways to Handle Back Pressure… • Dynamic push / pull • Static / Dynamic Quotas • Buffers • Ack / Nack • Throttling • Caching • Circuit breaker • Scaling • Some / all of the above • Etc…

Slide 17

Slide 17 text

But make sure you are doing something to handle it “Simply put, an unchecked Kafka consumer is a DoS attack on your application waiting to happen” -https://medium.com/@petermelias/kafka-consumer-patterns-and-gotchas-1bfc04cd643b “Unbounded queues are fundamentally broken because it puts the correctness of your system in someone else's hand” -@ztellman

Slide 18

Slide 18 text

And Always Take into Account System’s Reactions to Back Pressure… are highly contagious (Newton’s 3rd law) can propagate exponentially (butterfly effect)

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

So, a Robust Solution for Back Pressure / Flow Control should always be non-blocking requires some complex service choreography

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Backpressure with Akka Streams + Kafka • Akka Streams ○ Ideal for Fast Data ○ Seamless Integration ○ Abstraction to configure end-to-end Backpressure ○ Fully Asynchronous & non-blocking • Kafka ○ Dynamic durable buffer ○ Horizontally scalable ○ Load distribution ○ No data loss settings (Producer, Consumer, Broker) ○ Quotas

Slide 23

Slide 23 text

“With the use of asynchronous and back-pressured APIs, we’re able to push our systems to their limits, but not beyond them” -Why Reactive?, O’Reilly, Konrad Malawski

Slide 24

Slide 24 text

Demo: Back-Pressure -Fast Producer, Fast Consumers -Fast Producer, Fast and Slow Consumers -Fast Producer, Fast and Slow Consumers with buffering: OverflowStrategy.dropNew -Fast Producer, Fast and Slow Consumers with buffering: OverflowStrategy.backpressure

Slide 25

Slide 25 text

Stream Processing and Event-Driven Services

Slide 26

Slide 26 text

“MAKING SURE **ALL DATA** ENDS UP IN THE **RIGHT PLACES**” data integration

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

data input # ETL * Batches of records (Bounded sequence) * Time inside records * Denormalized View * Stage Tables, Files # EAI * Individual Messages * Batches are discouraged * MQ, Web Services * Normalized View # Stream Processing * Streams of Events (Unbounded sequence) * Event Log

Slide 33

Slide 33 text

data processing (i.e. vetro) # ETL * Transforming operational model into target schema. * Denormalization # EAI * Stateless processing * Remote Lookups (e.g. DB queries) * Cache for performance (e.g. Coherence) * Imperative approach / Side effects # Streaming Platform * Stateless and *Stateful* processing * Local Materialized views / Avoid remote Lookups * Functional approach / No side effects

Slide 34

Slide 34 text

data output # ETL * Data Warehouse # EAI * Operational back-ends # Streaming Platform * Both (Lambda Architecture)

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

# Event-Driven Services * embedded library in any application * event-log + your application * makes stream processing accessible to any use case (very low latency) Stream Processing visions # Big data, Real-Time Map-Reduce * central CLUSTER * custom packaging, deployment & monitoring * suitable for analytics-type use-cases (+/- low latency)

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

One more reason to start with event-driven stream processing

Slide 40

Slide 40 text

state and events: “the future is a function of the past”

Slide 41

Slide 41 text

immutability changes everything “transaction logs records ALL THE CHANGES made to the database [...] THE TRUTH IS IN THE LOG. the database is a cache of a subset of log.” Immutability changes everything - Pat Helland http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf

Slide 42

Slide 42 text

From Caches to Materialized Views r = cache.get(key) if (!r) { r = db.get(key) cache.put(key, r) } return k App Cache Database

Slide 43

Slide 43 text

From Caches to Materialized Views * cache invalidation? * race conditions? * consistency issues? * cold start/bootstrapping? App Cache Database

Slide 44

Slide 44 text

From Caches to Materialized Views Traditional Views: * *procrastinating* approach create view example from select foo from bar …; => rewrite result at query time create materialized view example from select foo from bar …; Materialized Views: * *proactive* approach

Slide 45

Slide 45 text

materialized views in an *unbounded database* Turning the Database Inside Out - Martin Kleppmann https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/

Slide 46

Slide 46 text

Kafka Streams: Streams-Relational Processing Platform Build Services on a Backbone of Events - Ben Stopford https://www.confluent.io/blog/build-services-backbone-events/

Slide 47

Slide 47 text

Demo: Comparing a EAI Integration with Stream Processing

Slide 48

Slide 48 text

observability

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

Observability is for **unknown unknowns** Is about making a system more: * Debuggable: tracking down failures and bugs * Understandable: answer questions, trends A Superset of: * Monitoring: how to operate a system * Instrumentation: how to develop a system to be monitoriable Observability for Emerging Infra: What Got You Here Won't Get You There - Charity Majors https://www.youtube.com/watch?v=1wjovFSCGhE