Microservices as an evolutionary architecture: lessons learned

Slide 1

Slide 1 text

Microservices as an Evolutionary Architecture: Lessons learned Claudio Eduardo de Oliveira & Luram Archanjo

Slide 2

Slide 2 text

Who am I? ● Software Engineer @Sensedia ● MBA in Java projects ● Java and Microservice enthusiast

Slide 3

Slide 3 text

Who am I? ● Lead Solutions Architect @Sensedia ● Book Author ● Spring, Java and Cloud Native enthusiast

Slide 4

Slide 4 text

Agenda ● Use Case & As Is ● Microservices ● Evolutionary Architecture ● Domain Driven Design ● Communication Patterns & Reactive Programming ● Observability & Database Migration ● Questions

Slide 5

Slide 5 text

Use case

Slide 6

Slide 6 text

Use case Token Service Provider Tokenization reduces the value of stored payment credentials by removing consumers’ primary account numbers (PANs) from the transaction process. It does this by replacing them with a unique identifier called a payment token. This reduces the appeal of stealing the credentials as they would be largely useless to hackers. Source: https://www.rambus.com/blogs/what-is-a-token-service-provider/

Slide 7

Slide 7 text

Use case

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

In the beginning

Slide 11

Slide 11 text

As is ● Single communication pattern ● 50-60% of our commits during the sprint were in the same repository ● Service with different workload requirements (Crypto and Business) ● Traffic east-west in the API Gateway ● Java & Spring everywhere ● Synchronous process everywhere

Slide 12

Slide 12 text

We were in the wrong way to adopt microservices architecture!

Slide 13

Slide 13 text

Let's recap what is the microservices architecture!

Slide 14

Slide 14 text

Moving to Microservices Feature A Feature B Feature C Monolith Microservice Microservice Microservices Microservice

Slide 15

Slide 15 text

Technological Heterogeneity Microservice Microservice Microservice DB DB DB

Slide 16

Slide 16 text

Evolutionary Architecture

Slide 17

Slide 17 text

Evolutionary Architecture An evolutionary architecture supports guided, incremental change as a first principle across multiple dimensions. Source: https://www.thoughtworks.com/books/building-evolutionary-architectures

Slide 18

Slide 18 text

Response time Microservices as an Evolutionary Architecture: Lessons learned Throughput

Slide 19

Slide 19 text

Before make a architectural decision, we need to be conﬁdent! How?

Slide 20

Slide 20 text

Regression Tests

Slide 21

Slide 21 text

Regression Tests REGRESSION TEST SUITE REGRESSION SCRIPTS AUTOMATED REGRESSION TESTING NEW FEATURES SCRIPT UPDATES

Slide 22

Slide 22 text

We have tests coverage and now?

Slide 23

Slide 23 text

Domain Driven Design (DDD)

Slide 24

Slide 24 text

Domain Driven Design(DDD) Domain-driven design (DDD) is an approach to developing software for complex needs by deeply connecting the implementation to an evolving model of the core business concepts. ● Place the project’s primary focus on the core domain and domain logic ● Base complex designs on a model ● Initiate a creative collaboration between technical and domain experts to iteratively cut ever closer to the conceptual heart of the problem. Source: https://dddcommunity.org/learning-ddd/what_is_ddd/

Slide 25

Slide 25 text

We are trying to ﬁgure out the technique to divide services, what part of DDD can help us?

Slide 26

Slide 26 text

Domain Driven Design - Bounded Context Bounded Context is a central pattern in Domain-Driven Design. It is the focus of DDD's strategic design section which is all about dealing with large models and teams. DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships. Source: https://martinfowler.com/bliki/BoundedContext.html

Slide 27

Slide 27 text

We separated the responsibilities and now?

Slide 28

Slide 28 text

Communication Patterns

Slide 29

Slide 29 text

REST API REST API, EVERYWHERE

Slide 30

Slide 30 text

Advanced Message Queuing Protocol (AMQP)

Slide 31

Slide 31 text

Advanced Message Queuing Protocol (AMQP) ● Introducing Message-Driven Programming with Low Level Events ● Fire & Forget principle ● Durable Messages (avoid Circuit Breakers & Retries) ● Better Scale strategies & Application Decoupling ● Fits well for 3rd partners integrations

Slide 32

Slide 32 text

Advanced Message Queuing Protocol (AMQP) - Thoughts ● Keep track of events otherwise you will have problems ● We used retries to put message in the queue ● Take care of the data into the message (security concerns)

Slide 33

Slide 33 text

HTTP / gRPC

Slide 34

Slide 34 text

HTTP / gRPC ● HTTP Connections are stateful (avoid open & close) ● Static Typed & Well Defined Contracts ● Reduce serialization complexity & improve efficiency ● Easy Peasy to integrate (there is code generation based in .proto) ● Traffic is binary reduce bandwidth usage

Slide 35

Slide 35 text

HTTP / gRPC - Thoughts ● Load Balancing and Client Load Balancing ● We used in the high throughput solution ● By definition in general we use for internal communication (a.k.a East-West) ● Not for humans, designed by machine communications

Slide 36

Slide 36 text

HTTP / REST

Slide 37

Slide 37 text

HTTP / REST ● Well documented Style ● “De-Facto” Pattern for Microservices Architecture ● “Toolability” ● Security

Slide 38

Slide 38 text

HTTP / REST - Thoughts ● Well documented Style ● For Humans, fits very well for External APIs ( API Strategy ) ● API Management provided by Sensedia offers interesting features to troubleshoot

Slide 39

Slide 39 text

We have tests, responsibilities and communication! It is possible to improve more?

Slide 40

Slide 40 text

Reactive Programming

Slide 41

Slide 41 text

Reactive Programming Reactive programming is an asynchronous programming paradigm concerned with data streams and the propagation of change. ● Easier to read, maintain and evolute ● Low memory footprint ● Scalability Source: https://en.wikipedia.org/wiki/Reactive_programming

Slide 42

Slide 42 text

Reactive Programming Source: https://craftsmen.nl/memory-usage-6-popular-rest-server-frameworks-compared/

Slide 43

Slide 43 text

Language

Slide 44

Slide 44 text

Language Kotlin is fully compatible with all Java-based frameworks, which lets you stay on your familiar technology stack while reaping the benefits of a more modern language. ● 40% less code = Easier to read, maintain and evolute ● Fail fast principle = Time to Market Source: https://kotlinlang.org/docs/reference/server-overview.html

Slide 45

Slide 45 text

Everything have been possible, because we formed a great Team Mindset

Slide 46

Slide 46 text

Team Mindset In a fixed mindset, people believe their basic qualities, like their intelligence or talent, are simply fixed traits. Contrarily, in a growth mindset, people believe that their most basic abilities can be developed through dedication and hard work. Source: https://mindsetonline.com/whatisit/about/

Slide 47

Slide 47 text

Team Mindset - Thoughts ● Hands-on ● Tech talks ● Lightning talks ● Events ○ The Developer’s Conference (TDC) ○ APIX ○ Meetups

Slide 48

Slide 48 text

We achieved the challenge! And now?

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

sensedia.com Monitoring & Observability

Slide 51

Slide 51 text

Monitoring is the practice of collecting signals, telemetry, traces, etc and aggregating them, and matching them against some pre-deﬁned criteria of system states we should carefully watch. When we ﬁnd that one of our signals has crossed a threshold and may be heading toward a known bad state, we take action to remedy the system - Christian Posta

Slide 52

Slide 52 text

Monitoring is a subset of Observability

Slide 53

Slide 53 text

Observability on the other hand supposes up front that our systems are highly unpredictable and we cannot know all of the possible failure modes up front

Slide 54

Slide 54 text

We need to collect much more data, even high-cardinality data like userIDs, requestIDs, source IPs, etc where the entire set could be exponentially large

Slide 55

Slide 55 text

With observability, we need ﬁne-grained data

Slide 56

Slide 56 text

A user goes to pay for the items in their cart and experiences a 10s delay choosing a payment option. All of the pre-deﬁned metric thresholds (disk usage, queue depth, machine health, etc) might be at acceptable levels

Slide 57

Slide 57 text

sensedia.com

Slide 58

Slide 58 text

Database Migration

Slide 59

Slide 59 text

Database Migration SQL databases scale vertically, meaning you’ll need to increase the capacity of a single server (increasing CPU, RAM, or SSD) to scale your database. NoSQL databases scale horizontally, meaning you can add more servers to power your growing database. Source: https://www.thorntech.com/2019/03/sql-vs-nosql/

Slide 60

Slide 60 text

Database Migration Microservice DB DB Synchronous, legacy Asynchronous, new Event

Slide 61

Slide 61 text

Before & After

Slide 62

Slide 62 text

Before

Slide 63

Slide 63 text

After

Slide 64

Slide 64 text

Summary 2º Place 1º Place 3º Place Last responsible moment ● Decisions ● Experience ● Maturity Team mindset ● Commitment ● Opportunity to grow and learn ● Feedbacks Technologies ● Understand the technologies concepts ● Business value

Slide 65

Slide 65 text

Thanks a million! Questions? /larchanjo /luram-archanjo /claudioed /claudioed