apidays Australia 2023 - The Playful Bond Between REST And Data Streams, Warren Vella, Confluent

Slide 1

Slide 1 text

The Playful Bond Between REST And Data Streams Warren Vella | Solutions Engineer, Melbourne | Confluent

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

About me.

Slide 4

Slide 4 text

“Hey, I am a microservice, come and talk to me.” REST API

Slide 5

Slide 5 text

“I need to share my data with an external application.” REST API

Slide 6

Slide 6 text

“I need to modernise my database with a facade.” REST API

Slide 7

Slide 7 text

API ≠ HTTP-based RESTful APIs

Slide 8

Slide 8 text

Operational Databases Analytic Systems Azure Synapse How do we integrate these systems together? SaaS Applications Applications Microservices Monoliths LEGACY INTEGRATION REST APIs only work here

Slide 9

Slide 9 text

REST APIs 8 1. …have proliferated as a way to integrate microservices. 2.…are often the default when exchanging data between organisations. 3.…don’t meet all our integration needs.

Slide 10

Slide 10 text

Lets go on a journey… The Stream Bean Cafe ☕

Slide 11

Slide 11 text

Early 2000s

Slide 12

Slide 12 text

Intra-Process Communication

Slide 13

Slide 13 text

2011ish

Slide 14

Slide 14 text

Microservices. Trend away from monolithic applications.

Slide 15

Slide 15 text

Distributed application architecture POS Payment Gateway DataWarehouse Realtime Analytics CRM

Slide 16

Slide 16 text

How many lines to integrate 4 systems?

Slide 17

Slide 17 text

n2-n

Slide 18

Slide 18 text

App 2 App 3 App 4 App 1 42-4 = 12 lines

Slide 19

Slide 19 text

How many lines to integrate 2,000 systems?

Slide 20

Slide 20 text

n2-n

Slide 21

Slide 21 text

20002-2000 = 3,998,000 lines

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

The microservices death star…

Slide 24

Slide 24 text

1. Physical 2. Data Link 3. Network 4. Transport 5. Session 6. Presentation 7. Application REST API POS DataWarehouse

Slide 25

Slide 25 text

Synchronous request-response REST APIS are addictive. Like an addiction, they give us a false sense of comfort of being always-available. The more we build, the more we want but we often overlook the exponential costs associated with building failure mechanisms for never ending point- to-point integrations. We end up relapsing to REST APIs.

Slide 26

Slide 26 text

Payment Gateway Request Response POS DataWarehouse Realtime Analytics CRM Asynchronous Synchronous

Slide 27

Slide 27 text

REST APIs 26 1. …have proliferated as a way to integrate microservices. 2.…are often the default when exchanging data between organisations. 3.…don’t meet all our integration needs. But we have ended up with a death star. But are not suited for high-throughput. So we have lots of different integration tools.

Slide 28

Slide 28 text

Let’s meet

Slide 29

Slide 29 text

Govern Process Connect Store Kafka decouples the producers and consumers Data Topic

Slide 30

Slide 30 text

Side note… Why is Kafka not a HTTP protocol like REST?

Slide 31

Slide 31 text

Old World New World Event Processor Stream Processor REST API

Slide 32

Slide 32 text

REST API • Great for Synchronous Responses • Tight-coupling • No delivery guarantees • Receiving systems can be overwhelmed • Emphasis on Requests/Responses • No Replay • True Decoupling • Stronger delivery guarantees. Kafka is a buffer. • Receiving systems process at a rate they can handle. • Emphasis on events. • Rebuild state by replay.

Slide 33

Slide 33 text

Payment Gateway Data Topic Can REST and Kafka co-exist? DataWarehouse Realtime Analytics CRM Inside Boundary POS Microservices Outside Boundary Request / Response

Slide 34

Slide 34 text

1. Physical 2. Data Link 3. Network 4. Transport 5. Session 6. Presentation 7. Application POS Realtime Analytics

Slide 35

Slide 35 text

Confluent Stream Sharing

Slide 36

Slide 36 text

Connect with Confluent

Slide 37

Slide 37 text

Operational Databases SaaS Applications Analytic Systems Azure Synaps e | DATA STREAMING PLATFORM Today, a fourth estate is emerging that provides a new foundation for your data strategy

Slide 38

Slide 38 text

“Hey, I am a microservice, come and talk to me.” REST API

Slide 39

Slide 39 text

“I need to share my data with an external application.” REST API

Slide 40

Slide 40 text

“I need to modernise my database with a facade.”

Slide 41

Slide 41 text

The future of integration.

Slide 42

Slide 42 text

Key Takeaways 41 1. ✅ Reduce the amount of lines between systems with Kafka. 2.✅ REST APIs are great for Request- Response but also consider Kafka as a pattern for exchanging data between organisations. 3.✅ A Data Streaming Platform integrates both application and data estates.