Unleashing Elasticsearch: Taking the Reins Off at Atlassian

Atlassian is using Elasticsearch to power its internal logging platform, depended on by dozens of internal product teams. Find out how a small team is delivering big wins by choosing to build on top of Elasticsearch and AWS, and where they're taking Elasticsearch next.

Transcript

1. ‹#› Sam Day, Code Monkey 18 Feb
 @samcday Unleashing Elasticsearch at Atlasssian
2. In the beginning…
4. And then suddenly, Cloud.
7. Cloud is hard Foundational tools A lot of people are counting on your stuff to be online, all day every day. Scale challenges Not just technical web scale problems — also organisational. World class reliability Should solve problems once. Build strong domain experience in teams.
8. Enable product teams to build world-class SaaS products. … does anyone even use the term “SaaS” anymore, or am I sounding like a total dork right now?
10. Logs are important, so we built a log platform.
11. We called it “Logging as a Service” (LaaS for short)
13. Dashboard tools Retention policy Scale What did we want in a logging platform? Security Make logs available for search for 30 days, archive them somewhere after that. Handle a high amount of logs, with the ability to handle an obscene amount of logs. Your log data is useless if you can’t effectively visualise it. Granular control over who can see what. Audit trails to know when things happened.
14. We're building a platform for storing, querying, and alerting on your logs in near- realtime, so you don't have to. LOGGING AS A SERVICE MISSION STATEMENT “ ”
15. processed every day 1.2TB
16. indexed log lines 60B
17. Active services sending logs 0 150 300 450 600 July September November Jan 559 410 357 203
18. Indexed data size (TB) 0 5 10 15 20 July September November Jan 20TB 7.5TB 3TB 0.5TB
19. Weekly active users 0 100 200 300 400 July September November Feb 336 232 161 48
20. Lots of services using LaaS. Our favourite?
22. Totally an awesome idea.
23. What did build our logging platform on?
24. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
25. ELK Stack
26. ELK Stack
27. Fluentd Elasticsearch Kibana Kinesis
28. Actually, we use Logstash too.
29. You could say we’re using the L’FEKK stack.
31. Ingestion
 service Kinesis Elasticsearch
 cluster
32. Kibana? More like Kickass…bana.
33. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
34. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
35. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
37. User analytics pipeline
38. Atlassians love hacking on top of Elasticsearch too
43. We shipped LaaS in 2 months, with a team of 3.
44. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
45. Magic?
46. Not really…
47. Just building on the shoulders of giants.
48. Kinesis Ansible AWS Tools of the trade CloudFormation When it comes to infrastructure - automate everything Because racking your own servers is so last decade. Because our platform customers hate us and want make us suffer. It’s kinda painful to work with, but infinitely less painful than working with AWS APIs :)
49. To err is human; to automate, divine
50. make prd-infrastructure
51. +
52. 3 months ago our cluster had 13 nodes
53. Today, it has 37
54. 3 months from now???
55. Automate all the things If Sam and his team die in a shocking bowling accident that also involves a bus, are we screwed? Agility Manual work is for suckers. Replace suckers with MACHINES. Confidence
56. Hey, let’s send LaaS 30,000 logs per second for a couple of hours, that’ll be fun! OUR CLOUD PRODUCTION ENVIRONMENT “ ”
57. Ingestion
 service Kinesis Elasticsearch
 cluster
58. Kinesis protects us from our beloved customers
60. How do we know when something is wrong, anyway?!
61. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
62. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
63. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
64. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
65. Q: Do I need a particular metric?
66. A: will 3am me thank not-3am me for it?
67. Use this one weird trick to scare the hell out of your friendly Elastic support engineer.
68. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
69. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
70. They be like
71. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
72. Why so many shards?
73. Eleventy trillion microservices Atlassian Account Confluence Cloud JIRA Cloud Horde Indra Atlassian Bitbucket Server Operations Engineering EAC Network Engineering Media Platform *.atlassian.com Marketplace Build Engineering Realtime Analytics Pipeline smtprelay service LaaS CSP … and each one has dev, staging and production environments
74. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
75. So what does it mean?
76. We have to throw rather a lot of RAM at the data nodes.
77. We’re have 30GB heaps on all 32 of our data nodes.
78. Generally we find that 960GB of RAM appeases the Shard Gods.
79. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
80. Hot nodes problem
81. A couple of nodes end up doing all the work.
82. Why? Shard allocation behaviour.
83. :(
84. Mitigating this can be tricky. We do lose data nodes to crazy GC pauses sometimes.
85. We’re building an indexing tier.
87. index.number_of_shards index.routing.allocation.total_shards_per_node index.routing.allocation.include.node_type
88. We haven’t actually rolled that out yet ;)
89. Current testing is looking very positive though!
92. Summary! Do weird crap with ES! We started with many gigabytes of data in ES, now it’s many terabytes. Powerful platform! Lots of cool people using our cool platform in cool ways. Many logs in ES! Life isn’t exciting unless you break some rules.
93. You should come (ab)use Elasticsearch with us!
94. Fin. SAM DAY • SOFTWARE PERSON • ATLASSIAN • @SAMCDAY