Context & Contingency: Patterns for choosing good tools

Slide 1

Slide 1 text

Context & Contingency Aaron Suggs

Slide 2

Slide 2 text

Thank you Jen, Joe, Ann Marie, Kaete, Chris, Mark, and Katie!

Slide 3

Slide 3 text

Aaron Suggs ktheory Ops Engineering

Slide 4

Slide 4 text

helps creative projects come to life

Slide 5

Slide 5 text

1. ELK 2. Management

Slide 6

Slide 6 text

“It’s not a promotion, it’s a career change” — Lindsay Holmwood MGMT 101

Slide 7

Slide 7 text

Kyle Burckhard built it @talaris

Slide 8

Slide 8 text

Business Requirements

Slide 9

Slide 9 text

“Look at the massive explosion in operational software offerings over the past 5-6 years.” — Charity Majors

Slide 10

Slide 10 text

Hosted SAAS

Slide 11

Slide 11 text

Maybe Splunk?

Slide 12

Slide 12 text

ELK

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

App logs Vendor logs lograge S3 Event ﬁlebeat SQS worker logstash (+S3) logstash AWS Elasticsearch Service (with Kibana)

Slide 15

Slide 15 text

Proof of concept

Slide 16

Slide 16 text

Slide 17

Slide 17 text

ﬁlebeat with real data!

Slide 18

Slide 18 text

ﬁlebeat with real data! “Do no harm”

Slide 19

Slide 19 text

Trivia Time!

Slide 20

Slide 20 text

Trivia Time! Why might a disk be 100% full after you delete several large ﬁles?

Slide 21

Slide 21 text

Answer " A process has open ﬁle handles (use lsof)

Slide 22

Slide 22 text

What to do?

Slide 23

Slide 23 text

What to do? - Upgrade it! Thx GitHub.

Slide 24

Slide 24 text

What to do? - Upgrade it! Thx GitHub. - Restart it.

Slide 25

Slide 25 text

What to do? - Upgrade it! Thx GitHub. - Restart it.

Slide 26

Slide 26 text

AWS Elasticsearch vs. DIY on EC2

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

AWS Elasticsearch DIY on EC2

Slide 29

Slide 29 text

AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention

Slide 30

Slide 30 text

AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention ✅

Slide 31

Slide 31 text

AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less ﬂexible Flexible, adaptable ✅

Slide 32

Slide 32 text

AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less ﬂexible Flexible, adaptable ✅ ✅

Slide 33

Slide 33 text

AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less ﬂexible Flexible, adaptable Hard to debug Lots of visibility ✅ ✅

Slide 34

Slide 34 text

AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less ﬂexible Flexible, adaptable Hard to debug Lots of visibility ✅ ✅ ✅

Slide 35

Slide 35 text

AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less ﬂexible Flexible, adaptable Hard to debug Lots of visibility Aligned w/ our use ✅ ✅ ✅

Slide 36

Slide 36 text

AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less ﬂexible Flexible, adaptable Hard to debug Lots of visibility Aligned w/ our use ✅ ✅ ✅ ✅

Slide 37

Slide 37 text

Rails log & Logstash

Slide 38

Slide 38 text

Rails log & Logstash

Slide 39

Slide 39 text

Lograge github.com/roidrage/lograge

Slide 40

Slide 40 text

When you show devs Kibana

Slide 41

Slide 41 text

When you show devs Kibana

Slide 42

Slide 42 text

MVP = better than alternative

Slide 43

Slide 43 text

T-Shaped skills

Slide 44

Slide 44 text

T-Shaped skills Me AWS Ruby

Slide 45

Slide 45 text

T-Shaped skills Me AWS Ruby Kyle AWS Ruby Docker

Slide 46

Slide 46 text

T-Shaped skills Me AWS Ruby Kyle AWS Ruby Docker ? Distributed systems Go

Slide 47

Slide 47 text

Ingesting vendor logs - S3 access logs - CDN access logs - CloudTrail logs

Slide 48

Slide 48 text

S3 Events -> SQS -> shoryuken ruby worker -> logstash

Slide 49

Slide 49 text

S3 Events -> SQS -> shoryuken ruby worker -> logstash re-import old logs

Slide 50

Slide 50 text

Autoscaling!

Slide 51

Slide 51 text

Trivia Time!

Slide 52

Slide 52 text

Trivia Time! What’s the bottleneck after you scale logstash?

Slide 53

Slide 53 text

Answer " 100% Elasticsearch CPU

Slide 54

Slide 54 text

Answer " Then it crashes. $ 100% Elasticsearch CPU

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Mitigating Burnout MGMT 102

Slide 57

Slide 57 text

Solution: Pairing 1. Fosters career growth 2. Cross-training 3. Team bonding

Slide 58

Slide 58 text

Summary

Slide 59

Slide 59 text

Summary 1. Mind the skills of your team

Slide 60

Slide 60 text

Summary 1. Mind the skills of your team 2. Know the next-best alternative

Slide 61

Slide 61 text

Summary 1. Mind the skills of your team 2. Know the next-best alternative 3. Consider a tool’s community

Slide 62

Slide 62 text

Thank you! ktheory