Aaron Suggs - Keynote - Context & Contingency: Patterns for choosing good tools

Transcript

1. Context & Contingency Aaron Suggs
2. Thank you Jen, Joe, Ann Marie, Kaete, Chris, Mark, and Katie!
3. Aaron Suggs ktheory Ops Engineering
4. helps creative projects come to life
5. 1. ELK 2. Management
6. “It’s not a promotion, it’s a career change” — Lindsay Holmwood MGMT 101
7. Kyle Burckhard built it @talaris
8. Business Requirements
9. “Look at the massive explosion in operational software offerings over the past 5-6 years.” — Charity Majors
10. Hosted SAAS
11. Maybe Splunk?
12. ELK
14. App logs Vendor logs lograge S3 Event filebeat SQS worker logstash (+S3) logstash AWS Elasticsearch Service (with Kibana)
15. Proof of concept
16. !
17. filebeat with real data!
18. filebeat with real data! “Do no harm”
19. Trivia Time!
20. Trivia Time! Why might a disk be 100% full after you delete several large files?
21. Answer " A process has open file handles (use lsof)
22. What to do?
23. What to do? - Upgrade it! Thx GitHub.
24. What to do? - Upgrade it! Thx GitHub. - Restart it.
25. What to do? - Upgrade it! Thx GitHub. - Restart it.
26. AWS Elasticsearch vs. DIY on EC2
28. AWS Elasticsearch DIY on EC2
29. AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention
30. AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention ✅
31. AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less flexible Flexible, adaptable ✅
32. AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less flexible Flexible, adaptable ✅ ✅
33. AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less flexible Flexible, adaptable Hard to debug Lots of visibility ✅ ✅
34. AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less flexible Flexible, adaptable Hard to debug Lots of visibility ✅ ✅ ✅
35. AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less flexible Flexible, adaptable Hard to debug Lots of visibility Aligned w/ our use ✅ ✅ ✅
36. AWS Elasticsearch DIY on EC2 Less dev attention* More dev attention Less flexible Flexible, adaptable Hard to debug Lots of visibility Aligned w/ our use ✅ ✅ ✅ ✅
37. Rails log & Logstash
38. Rails log & Logstash
39. Lograge github.com/roidrage/lograge
40. When you show devs Kibana
41. When you show devs Kibana
42. MVP = better than alternative
43. T-Shaped skills
44. T-Shaped skills Me AWS Ruby
45. T-Shaped skills Me AWS Ruby Kyle AWS Ruby Docker
46. T-Shaped skills Me AWS Ruby Kyle AWS Ruby Docker ? Distributed systems Go
47. Ingesting vendor logs - S3 access logs - CDN access logs - CloudTrail logs
48. S3 Events -> SQS -> shoryuken ruby worker -> logstash
49. S3 Events -> SQS -> shoryuken ruby worker -> logstash re-import old logs
50. Autoscaling!
51. Trivia Time!
52. Trivia Time! What’s the bottleneck after you scale logstash?
53. Answer " 100% Elasticsearch CPU
54. Answer " Then it crashes. $ 100% Elasticsearch CPU
56. Mitigating Burnout MGMT 102
57. Solution: Pairing 1. Fosters career growth 2. Cross-training 3. Team bonding
58. Summary
59. Summary 1. Mind the skills of your team
60. Summary 1. Mind the skills of your team 2. Know the next-best alternative
61. Summary 1. Mind the skills of your team 2. Know the next-best alternative 3. Consider a tool’s community
62. Thank you! ktheory