Operating Microservices

Slide 1

Slide 1 text

Michael Brunton-Spall Technical Architect Government Digital Service @bruntonspall

Slide 2

Slide 2 text

I'm a civil servant GDS Michael Brunton-Spall

Slide 3

Slide 3 text

GDS Michael Brunton-Spall

Slide 4

Slide 4 text

This isn’t the crown’s opinion on micro-services GDS Michael Brunton-Spall

Slide 5

Slide 5 text

Because everyone has an opinion GDS Michael Brunton-Spall

Slide 6

Slide 6 text

What am I not going to cover? GDS Michael Brunton-Spall

Slide 7

Slide 7 text

What are micro-services GDS Michael Brunton-Spall

Slide 8

Slide 8 text

How to develop them GDS Michael Brunton-Spall

Slide 9

Slide 9 text

Micro-service patterns GDS Michael Brunton-Spall

Slide 10

Slide 10 text

So what are micro-services? GDS Michael Brunton-Spall

Slide 11

Slide 11 text

Vertically aligned stacks that communicate via simple and standard interfaces GDS Michael Brunton-Spall

Slide 12

Slide 12 text

Ownership of data GDS Michael Brunton-Spall

Slide 13

Slide 13 text

Team ownership of code GDS Michael Brunton-Spall

Slide 14

Slide 14 text

Team ownership of code and runtime GDS Michael Brunton-Spall

Slide 15

Slide 15 text

Small is beautiful GDS Michael Brunton-Spall

Slide 16

Slide 16 text

Why are we all raving about them? GDS Michael Brunton-Spall

Slide 17

Slide 17 text

Because small owned systems can be updated more easily and frequently GDS Michael Brunton-Spall

Slide 18

Slide 18 text

Because teams can move fast and break stuff GDS Michael Brunton-Spall

Slide 19

Slide 19 text

Because teams can own the whole stack GDS Michael Brunton-Spall

Slide 20

Slide 20 text

So why do infrastructure teams hate them? GDS Michael Brunton-Spall

Slide 21

Slide 21 text

Because small owned systems can be updated more easily and frequently GDS Michael Brunton-Spall

Slide 22

Slide 22 text

Because teams can move fast and break stuff GDS Michael Brunton-Spall

Slide 23

Slide 23 text

Because teams can own the whole stack GDS Michael Brunton-Spall

Slide 24

Slide 24 text

“Microservices represent a new organisational model as much as a new architectural model” @JeffSussna GDS Michael Brunton-Spall

Slide 25

Slide 25 text

You (Engineering and Operations) exist purely for the beneﬁt of the business GDS Michael Brunton-Spall

Slide 26

Slide 26 text

Micro-services can help you have a radical focus on delivering business value GDS Michael Brunton-Spall

Slide 27

Slide 27 text

The role of engineering and operations is changing GDS Michael Brunton-Spall

Slide 28

Slide 28 text

Operations is the constraint GDS Michael Brunton-Spall

Slide 29

Slide 29 text

Operations is perceived as the constraint GDS Michael Brunton-Spall

Slide 30

Slide 30 text

Micro-services in the small GDS Michael Brunton-Spall

Slide 31

Slide 31 text

How can we get started? GDS Michael Brunton-Spall

Slide 32

Slide 32 text

Start small GDS Michael Brunton-Spall

Slide 33

Slide 33 text

Automate infrastructure or containers GDS Michael Brunton-Spall

Slide 34

Slide 34 text

Base Image GDS Michael Brunton-Spall

Slide 35

Slide 35 text

Base Image = Ubuntu LTS GDS Michael Brunton-Spall

Slide 36

Slide 36 text

Base Image + Cloud Init GDS Michael Brunton-Spall

Slide 37

Slide 37 text

Cloud Init = "apt-get update; apt-get install xxx" GDS Michael Brunton-Spall

Slide 38

Slide 38 text

Base Image + Cloud Init + Deployed Application GDS Michael Brunton-Spall

Slide 39

Slide 39 text

Deployed Application = "wget URL_TO_JAR; java - jar application.jar" GDS Michael Brunton-Spall

Slide 40

Slide 40 text

Time to new server = 1-2 minutes GDS Michael Brunton-Spall

Slide 41

Slide 41 text

Why is this useful? GDS Michael Brunton-Spall

Slide 42

Slide 42 text

Developer stupidity should have a business impact GDS Michael Brunton-Spall

Slide 43

Slide 43 text

Monitoring tools that are easy to hook into GDS Michael Brunton-Spall

Slide 44

Slide 44 text

Developers can create a new metric and graph it GDS Michael Brunton-Spall

Slide 45

Slide 45 text

GDS Michael Brunton-Spall

Slide 46

Slide 46 text

"If it moves, graph it. If it doesn't move, graph it anyway just in case it does” @lozzd GDS Michael Brunton-Spall

Slide 47

Slide 47 text

Automate log aggregation GDS Michael Brunton-Spall

Slide 48

Slide 48 text

ElasticSearch, LogStash, Kibana GDS Michael Brunton-Spall

Slide 49

Slide 49 text

GDS Michael Brunton-Spall

Slide 50

Slide 50 text

Cloud-friendly databases GDS Michael Brunton-Spall

Slide 51

Slide 51 text

Automate deployments GDS Michael Brunton-Spall

Slide 52

Slide 52 text

GDS Michael Brunton-Spall

Slide 53

Slide 53 text

GDS Michael Brunton-Spall

Slide 54

Slide 54 text

GDS Michael Brunton-Spall

Slide 55

Slide 55 text

Hand the keys for deployment to developers GDS Michael Brunton-Spall

Slide 56

Slide 56 text

Keep logs of deploys GDS Michael Brunton-Spall

Slide 57

Slide 57 text

GDS Michael Brunton-Spall

Slide 58

Slide 58 text

(Graph them even!) GDS Michael Brunton-Spall

Slide 59

Slide 59 text

GDS Michael Brunton-Spall

Slide 60

Slide 60 text

Alerting GDS Michael Brunton-Spall

Slide 61

Slide 61 text

(Sensu Screenshot) GDS Michael Brunton-Spall

Slide 62

Slide 62 text

Give developers pagers too GDS Michael Brunton-Spall

Slide 63

Slide 63 text

Developers should be exposed to the pain they cause GDS Michael Brunton-Spall

Slide 64

Slide 64 text

Developers having root on their service? GDS Michael Brunton-Spall

Slide 65

Slide 65 text

"You break it, you ﬁx it" GDS Michael Brunton-Spall

Slide 66

Slide 66 text

"You break it and cost £X million, you take the blame" GDS Michael Brunton-Spall

Slide 67

Slide 67 text

Automate Infrastructure Monitoring tools Log aggregation Databases Automated deployments Alerting GDS Michael Brunton-Spall

Slide 68

Slide 68 text

These are all solved problems GDS Michael Brunton-Spall

Slide 69

Slide 69 text

You are not *that* special GDS Michael Brunton-Spall

Slide 70

Slide 70 text

Most of what you do is wrapping libraries and other peoples code GDS Michael Brunton-Spall

Slide 71

Slide 71 text

Pareto principle GDS Michael Brunton-Spall

Slide 72

Slide 72 text

You now have a single, well supported micro service GDS Michael Brunton-Spall

Slide 73

Slide 73 text

Microservices in the large GDS Michael Brunton-Spall

Slide 74

Slide 74 text

Microservices are going to fail in more spectacular ways than equivalent monoliths GDS Michael Brunton-Spall

Slide 75

Slide 75 text

Simple GDS Michael Brunton-Spall

Slide 76

Slide 76 text

Simple Complicated GDS Michael Brunton-Spall

Slide 77

Slide 77 text

Simple Complicated Complex GDS Michael Brunton-Spall

Slide 78

Slide 78 text

Diagnosis tools GDS Michael Brunton-Spall

Slide 79

Slide 79 text

Someone will have to do diagnosis across all the services GDS Michael Brunton-Spall

Slide 80

Slide 80 text

How are services monitored? GDS Michael Brunton-Spall

Slide 81

Slide 81 text

Shallow Check - Is my service working GDS Michael Brunton-Spall

Slide 82

Slide 82 text

Deep check - Are my dependencies working? GDS Michael Brunton-Spall

Slide 83

Slide 83 text

Average response time isn't enough GDS Michael Brunton-Spall

Slide 84

Slide 84 text

90%, 95%, 99% as well GDS Michael Brunton-Spall

Slide 85

Slide 85 text

Distributed systems need to embrace failure GDS Michael Brunton-Spall

Slide 86

Slide 86 text

Partitions are network breaks and latency spikes GDS Michael Brunton-Spall

Slide 87

Slide 87 text

Timeouts are tricky, use with care GDS Michael Brunton-Spall

Slide 88

Slide 88 text

Standardise admin interfaces GDS Michael Brunton-Spall

Slide 89

Slide 89 text

Alerting tools GDS Michael Brunton-Spall

Slide 90

Slide 90 text

If citizens can’t submit a claim at 1am because the mainframe is down, who’s problem is that? GDS Michael Brunton-Spall

Slide 91

Slide 91 text

You still need 1st line support and triage GDS Michael Brunton-Spall

Slide 92

Slide 92 text

What about when the developers get bored? (They do that!) GDS Michael Brunton-Spall

Slide 93

Slide 93 text

Maturity model for support? GDS Michael Brunton-Spall

Slide 94

Slide 94 text

System must have < N faults in M weeks and complete run book to be handed off GDS Michael Brunton-Spall

Slide 95

Slide 95 text

Systems that have more than N faults in M weeks get handed back GDS Michael Brunton-Spall

Slide 96

Slide 96 text

Who pays for this support? GDS Michael Brunton-Spall

Slide 97

Slide 97 text

Operations teams should act like consultants GDS Michael Brunton-Spall

Slide 98

Slide 98 text

"You want an elastic search cluster, sure we'll help you …” GDS Michael Brunton-Spall

Slide 99

Slide 99 text

"... if we can document and automate it (+20% effort)" GDS Michael Brunton-Spall

Slide 100

Slide 100 text

Pioneers pay the cost GDS Michael Brunton-Spall

Slide 101

Slide 101 text

Settlers pave the roads behind GDS Michael Brunton-Spall

Slide 102

Slide 102 text

Everyone else follows GDS Michael Brunton-Spall

Slide 103

Slide 103 text

Don't forget to keep the lights on at the monolith GDS Michael Brunton-Spall

Slide 104

Slide 104 text

Michael Brunton-Spall Technical Architect Government Digital Service @bruntonspall