Slide 1

Slide 1 text

Data Architecture – Not Just for Microservices Eberhard Wolff @ewolff Fellow

Slide 2

Slide 2 text

http://continuous-delivery-buch.de/

Slide 3

Slide 3 text

http://microservices-buch.de/ http://microservices-book.com/

Slide 4

Slide 4 text

http://microservices-book.com/primer.html FREE!!!!

Slide 5

Slide 5 text

Classic Data Architecture > Centralized databases > …or services that provide data > Ensures consistency across systems > …for data model > ...and updates to data > Reuse

Slide 6

Slide 6 text

Classic Data Architecture Billing Order Process CRM Order

Slide 7

Slide 7 text

Who is using a centralized database?

Slide 8

Slide 8 text

Who likes the centralized database?

Slide 9

Slide 9 text

Microservices: Definition > No consistent definition > Microservices are modules > Independent deployment units > E.g. processes, Docker container > Microservice owned by one team

Slide 10

Slide 10 text

Microservices: Definition Server / Container Server / Container Micro Service Micro Service

Slide 11

Slide 11 text

Why Microservices? > Develop a feature > …bring it into production > ...with no coordination > Independent scaling > Free choice of technology > Robustness > Security

Slide 12

Slide 12 text

Microservices aim for decoupling

Slide 13

Slide 13 text

Microservices & Data Server / Container Server / Container Micro Service Micro Service Order Schema

Slide 14

Slide 14 text

Microservices & Data Server / Container Server / Container Micro Service Micro Service Order Schema

Slide 15

Slide 15 text

Microservices & Data > Decoupling for data, too > Separate data storage

Slide 16

Slide 16 text

Data Microservices Server / Container Server / Container Micro Service Micro Service Order Data Microservice

Slide 17

Slide 17 text

Data Microservices Server / Container Micro Service Order Data Microservice Customer Data Microservice Remote calls influence performance availability No transaction across customer and order

Slide 18

Slide 18 text

Data Microservice > Change two microservices if new feature requires change to data schema > But: data in one place > No consistency issues

Slide 19

Slide 19 text

Data microservice limits decoupling.

Slide 20

Slide 20 text

Encapsulation > Information hiding > Hide the internal data structure > Provide access only through a well defined interface > Data and databases should not be exported

Slide 21

Slide 21 text

Violates Encapsulation Billing Order Process CRM Order Shared data model Logic

Slide 22

Slide 22 text

Violates Encapsulation Server / Container Server / Container Micro Service Micro Service Order Data Microservice Shared data model Logic

Slide 23

Slide 23 text

Separate Databases Server / Container Server / Container Micro Service Micro Service Order Order

Slide 24

Slide 24 text

Different Databases Server / Container Server / Container Micro Service Micro Service neo4j Oracle

Slide 25

Slide 25 text

Different Databases > “Polyglot persistence” > Use the best tool for the job > Technology freedom – advantage of microservices > …but extra effort > Backup, disaster recovery etc. > Not as easy as e.g. different frameworks

Slide 26

Slide 26 text

Separate Schema Server / Container Server / Container Micro Service Micro Service Oracle Schema Schema

Slide 27

Slide 27 text

Separate Schemas > Less effort > Decoupled data models > ...but limited independent scaling and robustness

Slide 28

Slide 28 text

Billing Order Process CRM Order Order Order

Slide 29

Slide 29 text

Redundancy!!!

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

Domain-driven Design

Slide 33

Slide 33 text

Domain-driven Design > 2004 > Still very relevant > By Eric Evans > Focus on part IV > Free reference: http://domainlanguage.com/ ddd/reference/

Slide 34

Slide 34 text

Order Shipping address Tracking # Items Item Categories Priority shipping Customs # Account # ... Credit card # Order #

Slide 35

Slide 35 text

My Domain Model is a mess!

Slide 36

Slide 36 text

Bounded Context > Domain model is only valid for one context > There is no universal data model! > See all failed SOA attempts

Slide 37

Slide 37 text

Order Shipping address Tracking # Items Item Categories Priority shipping Customs # Account # ... Credit card # Order # Customs Order Recommen- dations Order Tracking Order Shipping address Tracking # Item Categories Priority shipping Customs # Payment Order Account # Credit card #

Slide 38

Slide 38 text

Billing Order Process CRM Order Order Order

Slide 39

Slide 39 text

Bounded Context > Microservice = BOUNDED CONTEXTS > Changes for new features are local > …even if data models need to be changed

Slide 40

Slide 40 text

Billing Order Process CRM Order Order Order

Slide 41

Slide 41 text

Redundancy?

Slide 42

Slide 42 text

Redundancy? Not really

Slide 43

Slide 43 text

Bounded Context

Slide 44

Slide 44 text

What about basic data of an order?

Slide 45

Slide 45 text

Strategic Design > How do BOUNDED CONTEXTS relate to each other? > Context can have relationships > DDD defines several relationship patterns

Slide 46

Slide 46 text

Shared Kernel > Subset of a model > …that two teams share > Eric Evans: Including code and database > Microservices: Just sharing a model

Slide 47

Slide 47 text

Anti-corruption Layer > Don’t let e.g. a legacy model influence a new model > Isolate model by additional layer > No need to modify the old system

Slide 48

Slide 48 text

Context Relationships > Team = Deployment Unit = BOUNDED CONTEXT > Context Relationships define how BOUNDED CONTEXT are used… > ...and how much teams need to collaborate

Slide 49

Slide 49 text

Coordination Effort Shared BOUNDED CONTEXT SHARED KERNEL CUSTOMER / SUPPLIER ANTICORRUPTION LAYER CONFORMIST SEPARATE WAYS

Slide 50

Slide 50 text

Context Map

Slide 51

Slide 51 text

Context Map > Show the different BOUNDED CONTEXT > …and the relation to each other > BOUNDED CONTEXT might be microservices > ...or communication links

Slide 52

Slide 52 text

Order Process Registration Basic Customer Data Basic Customer Data Customer Order Data Delivery Customer Order Data Billing Anticorruption Layer Mainframe Customer Data Customer Order Data Customer Order Data Basic Customer Data + Customer Order Data = Shared Kernel

Slide 53

Slide 53 text

Billing Order Process CRM Shared Kernel Order Additional data Additional data Additional data Order Data

Slide 54

Slide 54 text

Centralized Shared Kernel > Ensures consistency > ...but needs to be called for a lot of operations > Resilience / performance / transactions > Have one master as the source of truth

Slide 55

Slide 55 text

Billing Order Process CRM Shared Kernel Order Shared Kernel Order Shared Kernel Order Additional data Additional data Additional data

Slide 56

Slide 56 text

Decentralized Shared Kernel > Might be inconsistent > ...but all data for all requests is available in the local database > Better resilience… > ...and performance

Slide 57

Slide 57 text

How to Replicate Data?

Slide 58

Slide 58 text

Database Replication > Built into the database > Replicate schema across database instances > But: Microservices have separated schemas > Every Microservice might have different data > …so database replication is not a good fit

Slide 59

Slide 59 text

Replication with Events

Slide 60

Slide 60 text

Events > Later addition to Domain-driven Design > Events with a business meaning > Decouple time: Asynchronous > Decouple logic: System can handle event as it pleases

Slide 61

Slide 61 text

Billing Order Process CRM Shared Kernel Order Shared Kernel Order Shared Kernel Order Additional data Additional data Additional data New Order Event

Slide 62

Slide 62 text

Events & Data Replication > Events lead to data replication > i.e. each system stores information it received in an event > Data stored in separate schema > Very decoupled > Hard to repair inconsistencies

Slide 63

Slide 63 text

More Fun With Events

Slide 64

Slide 64 text

Event Sourcing > Internal Structure for Microservice with events > Current state result of all events > Calculate state on the fly?

Slide 65

Slide 65 text

Event Queue Event Event Event Event Handler Event Handler Event Store Snapshot

Slide 66

Slide 66 text

Event Sourcing > Event store and snapshot help to repair inconsistencies > Event-based architecture in microservices

Slide 67

Slide 67 text

CQRS > Command – Query Responsibility Segregation > Commands change data > Query provide data > Implement in separate modules > …or even microservices > ...with potentially different BOUNDED CONTEXTS

Slide 68

Slide 68 text

Commands vs Events > Command: Change that data! > Event: Something has happened > Component decides if data should be changed

Slide 69

Slide 69 text

Command Queue Command Command Command Command Handler Query Handler Command Store Database

Slide 70

Slide 70 text

Batch Replication

Slide 71

Slide 71 text

Batch > Get all data > Provide API > …to decouple schema > Copy interesting data into local database

Slide 72

Slide 72 text

Billing Order Process CRM Shared Kernel Order Shared Kernel Order Shared Kernel Order Additional data Additional data Additional data Batch Batch API API

Slide 73

Slide 73 text

Batch & Data Replication > Easy to repair inconsistencies > Batch run at specific points > i.e. updates take time > Data not consistent across microservices

Slide 74

Slide 74 text

CAP: Challenge for Replication

Slide 75

Slide 75 text

CAP Theorem > Consistency > All nodes see the same data > Availability > Node failures do not prevent survivors from operating > Partition Tolerance > System continues to operate despite arbitrary message loss C P A

Slide 76

Slide 76 text

CAP Theorem: P > Network partitions do occur > Even with highly available network hardware > Also: very slow response = partition > Need to deal with P

Slide 77

Slide 77 text

CAP Theorem: C or A? > Node cannot access other nodes > Might have missed updates > A, not C: Answer with a potentially wrong answer > C, not A: Don’t answer – the answer might be wrong

Slide 78

Slide 78 text

Billing Order Process CRM Shared Kernel Order Shared Kernel Order Shared Kernel Order Additional data Additional data Additional data New Order Event inconsistent or unavailable

Slide 79

Slide 79 text

Conclusion

Slide 80

Slide 80 text

Classic: Centralized Database Microservices: private database decoupling Data Microservices: Consistent but resilience / performance / transactions / decoupling? Database per Microservice: Polyglot Persistence Schema per Microservice: Simple infrastructure

Slide 81

Slide 81 text

Redundant Data or Bounded Context? Batch Database Replication Events Redundancy? Context Map and Context Relations Replication CAP Event Sourcing CQRS e.g. Shared Kernel

Slide 82

Slide 82 text

Decentralize data!