Atlassian: Unleashing Elasticsearch

Atlassian is using Elasticsearch to power its internal logging platform, depended on by dozens of internal product teams. Find out how a small team is delivering big wins by choosing to build on top of Elasticsearch and AWS, and where they're taking Elasticsearch next.

Transcript

1. SAM DAY • CODE MONKEY • ATLASSIAN • @SAMCDAY Unleashing Elasticsearch
2. In the beginning…
4. A wild Cloud appears!
7. Cloud is hard Foundational tools A lot of people are counting on your stuff to be online, all day every day. Scale challenges Not just technical web scale problems — also organisational. World class reliability Solve problems once. Build strong domain experience in teams.
8. Enable teams to build world class SaaS
10. Everybody needs their logs …
11. But they shouldn’t have to work for ‘em
12. So we built a log platform
13. We called it “Logging as a Service”
15. Dashboard tools Retention policy Scale What did we want in a logging platform? Security Make logs available for search for 30 days, archive them somewhere after that. Handle a high amount of logs, with the ability to handle an obscene amount of logs. Your log data is useless if you can’t effectively visualise it. Granular control over who can see what. Audit trails to know when things happened.
16. We're building a platform for storing, querying, and alerting on your logs in near- realtime, so you don't have to. LOGGING AS A SERVICE MISSION STATEMENT “ ”
17. So what did we use to build our logging platform?
19. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
20. data stored per day 500GB
21. indexed log lines 17B
22. Active services sending logs 0 125 250 375 500 July September November 410 357 203
23. Indexed data size 0 2000 4000 6000 8000 July September November 7500GB 3000GB 500GB
24. Weekly active users 0 75 150 225 300 July September November 232 161 48
25. So, lots of services using us. Our favourite?
27. Totally an awesome idea.
28. Kibana rocks Open source Powerful Why did we choose Elasticsearch?
29. Kibana? More like Kickass…bana.
30. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
31. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
32. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
33. Atlassians love hacking on top of Elasticsearch too
38. We built LaaS in 2 months, with a team of 3.
39. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
40. Magic?
41. Not really…
42. Just building on the shoulders of giants.
43. Kinesis Ansible AWS Tools of the trade CloudFormation When it comes to infrastructure - automate everything Because racking your own servers is so last decade. Because our platform customers are evil and out to get us. It’s kinda painful to work with, but infinitely less painful than working with AWS APIs :)
44. Today our cluster has 13 nodes
45. Tomorrow, it might have 26
46. Next week???
47. To err is human; to automate, divine
48. Need to rebuild production LaaS environment?
49. make prd-infrastructure
50. +
52. Ingestion
 service Kinesis Elasticsearch
 cluster
54. We <3 Kinesis
55. Hey, let’s send LaaS 30,000 logs per second for a couple of hours, that’ll be fun! CONFLUENCE AND JIRA CLOUD “ ”
56. Ingestion
 service Kinesis Elasticsearch
 cluster
57. Kinesis protects us from our beloved customers
58. How do we know when something is wrong, anyway?!
59. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
60. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
61. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
62. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
63. Q: Do I need a particular metric?
64. A: will 3am me thank me for it?
65. Use this one weird trick to scare the hell out of your friendly Elastic support engineer.
66. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
67. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
68. They be like
69. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
70. Why so many shards?
71. Eleventy trillion microservices Atlassian Account Confluence Cloud JIRA Cloud Horde Indra Atlassian Bitbucket Server Operations Engineering EAC Network Engineering Media Platform *.atlassian.com Marketplace Build Engineering Realtime Analytics Pipeline smtprelay service LaaS CSP … and each one has dev, staging and production environments
72. So what does it mean?
73. We have to throw rather a lot of RAM at the data nodes.
74. We’ve found that 240GB appeases the Shard Gods.
75. The Atlassian sales model is “land and expand”
76. That’s exactly what Elastic is doing at Atlassian!
77. Done! SAM DAY • SOFTWARE PERSON • ATLASSIAN • @SAMCDAY