Tweet
Tweet

Slide 1

Slide 1 text

EXPLORING MICROSERVICE ARCHITECTURE THROUGH GRAPH THEORY Nicki Watt - CTO @techiewatt

Slide 2

Slide 2 text

@techiewatt Hands-on software delivery Consulting provides services is CTO at Cloud 
 Native / 
 Microservices Architectures Data
 Engineering Platforms Co-author of with expertise in lead projects on includes experience gained building

Slide 3

Slide 3 text

AGENDA OVERVIEW ● INTRO & HYPOTHESIS
 ● A BIT OF THEORY
 ● MICROSERVICE EXPLORING
 ● SUMMARY @techiewatt

Slide 4

Slide 4 text

INTRODUCTION @techiewatt

Slide 5

Slide 5 text

Graph theory is already being used to drive efficiencies in, and produce more reliable software systems INTRODUCTION @techiewatt

Slide 6

Slide 6 text

INTRODUCTION @techiewatt Code: Analysing software package & license dependencies

Slide 7

Slide 7 text

INTRODUCTION @techiewatt Infrastructure: Reliably managing and updating cloud and other infrastructure-as-code resources Credit: Paul Hinze (HashiConf 2016)
 Applying Graph Theory to Infrastructure-as-code

Slide 8

Slide 8 text

INTRODUCTION @techiewatt Ops & Monitoring: Troubleshooting, Distributed Tracing & Latency analysis

Slide 9

Slide 9 text

Can we also use it to help inspect and drive us towards improvements in our microservices architecture? INTRODUCTION @techiewatt

Slide 10

Slide 10 text

HYPOTHESIS @techiewatt

Slide 11

Slide 11 text

HYPOTHESIS @techiewatt General Hypothesis:
 Data Driven Architectural Improvement
 
 You can extract metrics and KPIs from a microservices architecture using graph theory AND use these to gain insight into the structure and characteristics of your microservices architecture

Slide 12

Slide 12 text

HYPOTHESIS @techiewatt For this talk … 
 Can we use these metrics to detect bad microservice architectural smells and anti-patterns like a tightly coupled architecture (distributed monolith)

Slide 13

Slide 13 text

A QUICK BIT OF THEORY @techiewatt

Slide 14

Slide 14 text

GRAPH THEORY 101 Nodes / 
 Vertices Relationships / 
 Edges @techiewatt A graph is a way to formally represent a network, or collection of related objects, in a mathematical way

Slide 15

Slide 15 text

● Graph Analytics ○ An Action Performed: The act of analysing connected data - using any appropriate graph-based approach or tools (visualisations, queries, statistics, algorithms)
 ● Graph Algorithms ○ Programmable process or set of rules: Leverages the mathematical properties of graphs to explore, classify and interpret connected data. A subset of tooling used to do graph analytics. @techiewatt GRAPH INSIGHTS 101

Slide 16

Slide 16 text

GRAPH INSIGHTS 101 @techiewatt New academic field, circa 21st century includes study of: ● Real-world representations in order to understand the universal properties of networks (Biological, social, transport) ● Large, complex networks Roots date back to 1786, includes study of: ● Abstract theoretical graph forms (e.g. random graphs, trees, directed graphs) ● Graphs of any size NETWORK SCIENCE GRAPH THEORY

Slide 17

Slide 17 text

“Based on the mathematics of graph theory, graph algorithms use the relationships between nodes to infer the organization and dynamics of complex systems. Network scientists use these algorithms to uncover hidden information, test hypotheses, and make predictions about behavior”.
 - Graph Algorithms by M Needham, A Hodler @techiewatt GRAPH INSIGHTS 101

Slide 18

Slide 18 text

Show me some of this graph theory / network science
 stuff ... @techiewatt

Slide 19

Slide 19 text

@techiewatt Path 
 Finding Centrality Community Detection HIGH LEVEL ALGORITHM TYPES

Slide 20

Slide 20 text

@techiewatt Path 
 Finding Centrality Community Detection HIGH LEVEL ALGORITHM TYPES

Slide 21

Slide 21 text

DEGREE How connected is a specific node? @techiewatt 1

Slide 22

Slide 22 text

DEGREE How connected is a specific node? A is more highly connected than 
 B and C 6 2 2 @techiewatt 1

Slide 23

Slide 23 text

CLUSTER COEFFICIENT How tightly is a group clustered, compared to how tightly it could be clustered? @techiewatt 2

Slide 24

Slide 24 text

CLUSTER COEFFICIENT @techiewatt 2

Slide 25

Slide 25 text

CLUSTER COEFFICIENT @techiewatt 2

Slide 26

Slide 26 text

CLUSTER COEFFICIENT @techiewatt 2

Slide 27

Slide 27 text

CLUSTER COEFFICIENT @techiewatt 0.66 2

Slide 28

Slide 28 text

CLUSTER COEFFICIENT @techiewatt 0.66 2

Slide 29

Slide 29 text

CLUSTER COEFFICIENT @techiewatt 0.66 1.0 2

Slide 30

Slide 30 text

COMMUNITY DETECTION Used to find related communities, uncover groupings, and quantify the quality of groupings @techiewatt 3

Slide 31

Slide 31 text

DIVE IN! EXPLORING MICROSERVICES (AS A NETWORK) @techiewatt

Slide 32

Slide 32 text

#1 A “Microservice” Architecture 
 (v1.0) @techiewatt

Slide 33

Slide 33 text

@techiewatt V1.0

Slide 34

Slide 34 text

@techiewatt V1.0 - NETWORK STATS Density: 0.19 Number of nodes: 20 Number of edges: 36 Average degree: 3.6000 Average Clustering Co-eff: 0.30

Slide 35

Slide 35 text

@techiewatt V1.0 - DEGREE

Slide 36

Slide 36 text

@techiewatt V1.0 - CLUSTER COEFFICIENT

Slide 37

Slide 37 text

@techiewatt We’ve been directed to look a bit closer at a few potential problem services … Where do they fit in architecturally?

Slide 38

Slide 38 text

V1.0 - ARCHITECTURE DIAGRAM @techiewatt Front End Service Back End Service External Integration Adapter Service Database API API API API API API

Slide 39

Slide 39 text

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C v1 - degree & cluster coefficient degree cluster coefficient V1.0 ANALYSIS @techiewatt Front End Back End External Integration

Slide 40

Slide 40 text

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C v1 - degree & cluster coefficient degree cluster coefficient V1.0 ANALYSIS @techiewatt Front End Back End External Integration

Slide 41

Slide 41 text

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C v1 - degree & cluster coefficient degree cluster coefficient V1.0 ANALYSIS @techiewatt Average degree: 3.6 Average Clustering 
 Coefficient: 0.3 Front End Back End External Integration

Slide 42

Slide 42 text

V1.0 INVESTIGATION @techiewatt - Entity Services 
 Anti-Pattern - Distributed 
 Monolith Front End Back End External Integration

Slide 43

Slide 43 text

V1.0 INVESTIGATION @techiewatt - Entity Services 
 Anti-Pattern - Distributed 
 Monolith Front End Back End External Integration Order User Product

Slide 44

Slide 44 text

V1.0 INVESTIGATION @techiewatt - Entity Services 
 Anti-Pattern - Distributed 
 Monolith Front End Back End External Integration

Slide 45

Slide 45 text

V1.0 - COMMUNITY DETECTION @techiewatt

Slide 46

Slide 46 text

V1.0 - OBSERVATIONS CONTINUED @techiewatt - Help detect tightly coupled areas - Not always DDD 
 aligned

Slide 47

Slide 47 text

COMMUNITIES NOT ALWAYS DDD ALIGNED @techiewatt Organisational Credit: http://bonkersworld.net/organizational-charts Conway’s Law ?

Slide 48

Slide 48 text

COMMUNITIES NOT ALWAYS DDD ALIGNED @techiewatt Process & Data Criteria Organisation Operations Erich Eichinger Blog: “Heuristics for Identifying Service Boundaries” https://opencredo.com/blogs/identify-service-boundary-heuristics/

Slide 49

Slide 49 text

#2
 The Revised Microservice Arch 
 User Domain Decoupling
 (v2.0) @techiewatt

Slide 50

Slide 50 text

Recap … V1.0 @techiewatt User Front End Back End External Integration Database

Slide 51

Slide 51 text

V2.0 @techiewatt User V1.0 —> V2.0 Front End Back End External Integration Database

Slide 52

Slide 52 text

V2.0 @techiewatt Front End Back End External Integration Database

Slide 53

Slide 53 text

V2.0 - NETWORK STATS @techiewatt Density: 0.11 Number of nodes: 29 Number of edges: 46 Average degree: 3.1724 Average Clustering Co-eff: 0.20

Slide 54

Slide 54 text

V2.0 - COMBINED STATS @techiewatt

Slide 55

Slide 55 text

V2.0 - COMBINED STATS @techiewatt

Slide 56

Slide 56 text

V2.0 - COMBINED STATS @techiewatt

Slide 57

Slide 57 text

V2.0 - COMMUNITY DETECTION @techiewatt

Slide 58

Slide 58 text

COMPARING V1 and V2 @techiewatt

Slide 59

Slide 59 text

@techiewatt COMPARING Degree Cluster Coefficient 0 0.2 0.4 0.6 0.8 1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient

Slide 60

Slide 60 text

0 0.2 0.4 0.6 0.8 1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient Degree Cluster Coefficient @techiewatt COMPARING V1 User Service

Slide 61

Slide 61 text

0 0.2 0.4 0.6 0.8 1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient Degree Cluster Coefficient @techiewatt COMPARING V1 User Service New V2 Refactored
 User Services

Slide 62

Slide 62 text

Degree Cluster Coefficient 0 0.2 0.4 0.6 0.8 1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient @techiewatt COMPARING New V2 Refactored
 User Services V1 User Service V2 Mono 
 User Service (allow distmon to continue functioning)

Slide 63

Slide 63 text

@techiewatt COMPARING Degree Cluster Coefficient 0 0.2 0.4 0.6 0.8 1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient

Slide 64

Slide 64 text

0 0.2 0.4 0.6 0.8 1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient Degree Cluster Coefficient @techiewatt COMPARING V1 (5) V2 (4)

Slide 65

Slide 65 text

0 0.2 0.4 0.6 0.8 1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient Degree Cluster Coefficient @techiewatt COMPARING V2 (1.0) V1 (0.9)

Slide 66

Slide 66 text

FURTHER OPTIONS @techiewatt

Slide 67

Slide 67 text

OTHER MEASURES & ALGORITHMS @techiewatt - Strongly 
 Connected 
 Components 
 (SCC) - Help detect 
 circular dependencies

Slide 68

Slide 68 text

OTHER MEASURES & ALGORITHMS @techiewatt - Path Finding Algorithms
 Detect who is calling deprecated services

Slide 69

Slide 69 text

CONCLUSION @techiewatt

Slide 70

Slide 70 text

Early days … Can’t be applied blindly … Common sense and understanding of architectural approaches and patterns still required!! CONCLUSION @techiewatt

Slide 71

Slide 71 text

HYPOTHESIS @techiewatt General Hypothesis:
 Data Driven Architectural Improvement
 
 You can extract metrics and KPIs from a microservices architecture using graph theory AND use these to gain insight into the structure and characteristics of your microservices architecture

Slide 72

Slide 72 text

HYPOTHESIS @techiewatt General Hypothesis:
 Data Driven Architectural Improvement
 
 You can extract metrics and KPIs from a microservices architecture using graph theory AND use these to gain insight into the structure and characteristics of your microservices architecture

Slide 73

Slide 73 text

HYPOTHESIS @techiewatt Specifically 
 Can we use these metrics to detect bad microservice architectural smells and anti-patterns like a tightly coupled architecture (distributed monolith)?

Slide 74

Slide 74 text

CONCLUSION @techiewatt Demonstrated using degree and cluster coefficient measures to detect tightly coupled (distributed monolith) architectures Demonstrated using community detection algorithms to uncover related groupings (boundaries) of microservices

Slide 75

Slide 75 text

RECOMMENDED LEARNING RESOURCES @techiewatt

Slide 76

Slide 76 text

@techiewatt Come chat with me! Looking for strategic or hands-on help? We’re hiring! Want to find out more?