Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Atlassian: Unleashing Elasticsearch

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
December 10, 2015

Atlassian: Unleashing Elasticsearch

Atlassian is using Elasticsearch to power its internal logging platform, depended on by dozens of internal product teams. Find out how a small team is delivering big wins by choosing to build on top of Elasticsearch and AWS, and where they're taking Elasticsearch next.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

December 10, 2015
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. SAM DAY • CODE MONKEY • ATLASSIAN • @SAMCDAY Unleashing

    Elasticsearch
  2. In the beginning…

  3. None
  4. A wild Cloud appears!

  5. None
  6. None
  7. Cloud is hard Foundational tools A lot of people are

    counting on your stuff to be online, all day every day. Scale challenges Not just technical web scale problems — also organisational. World class reliability Solve problems once. Build strong domain experience in teams.
  8. Enable teams to build world class SaaS

  9. None
  10. Everybody needs their logs …

  11. But they shouldn’t have to work for ‘em

  12. So we built a log platform

  13. We called it “Logging as a Service”

  14. None
  15. Dashboard tools Retention policy Scale What did we want in

    a logging platform? Security Make logs available for search for 30 days, archive them somewhere after that. Handle a high amount of logs, with the ability to handle an obscene amount of logs. Your log data is useless if you can’t effectively visualise it. Granular control over who can see what. Audit trails to know when things happened.
  16. We're building a platform for storing, querying, and alerting on

    your logs in near- realtime, so you don't have to. LOGGING AS A SERVICE MISSION STATEMENT “ ”
  17. So what did we use to build our logging platform?

  18. None
  19. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  20. data stored per day 500GB

  21. indexed log lines 17B

  22. Active services sending logs 0 125 250 375 500 July

    September November 410 357 203
  23. Indexed data size 0 2000 4000 6000 8000 July September

    November 7500GB 3000GB 500GB
  24. Weekly active users 0 75 150 225 300 July September

    November 232 161 48
  25. So, lots of services using us. Our favourite?

  26. None
  27. Totally an awesome idea.

  28. Kibana rocks Open source Powerful Why did we choose Elasticsearch?

  29. Kibana? More like Kickass…bana.

  30. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  31. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  32. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  33. Atlassians love hacking on top of Elasticsearch too

  34. None
  35. None
  36. None
  37. None
  38. We built LaaS in 2 months, with a team of

    3.
  39. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  40. Magic?

  41. Not really…

  42. Just building on the shoulders of giants.

  43. Kinesis Ansible AWS Tools of the trade CloudFormation When it

    comes to infrastructure - automate everything Because racking your own servers is so last decade. Because our platform customers are evil and out to get us. It’s kinda painful to work with, but infinitely less painful than working with AWS APIs :)
  44. Today our cluster has 13 nodes

  45. Tomorrow, it might have 26

  46. Next week???

  47. To err is human; to automate, divine

  48. Need to rebuild production LaaS environment?

  49. make prd-infrastructure

  50. +

  51. None
  52. Ingestion
 service Kinesis Elasticsearch
 cluster

  53. None
  54. We <3 Kinesis

  55. Hey, let’s send LaaS 30,000 logs per second for a

    couple of hours, that’ll be fun! CONFLUENCE AND JIRA CLOUD “ ”
  56. Ingestion
 service Kinesis Elasticsearch
 cluster

  57. Kinesis protects us from our beloved customers

  58. How do we know when something is wrong, anyway?!

  59. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  60. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  61. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  62. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  63. Q: Do I need a particular metric?

  64. A: will 3am me thank me for it?

  65. Use this one weird trick to scare the hell out

    of your friendly Elastic support engineer.
  66. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  67. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  68. They be like

  69. • Big cool statistic • 2,56 9 • Add-Ons in

    Marketplace
  70. Why so many shards?

  71. Eleventy trillion microservices Atlassian Account Confluence Cloud JIRA Cloud Horde

    Indra Atlassian Bitbucket Server Operations Engineering EAC Network Engineering Media Platform *.atlassian.com Marketplace Build Engineering Realtime Analytics Pipeline smtprelay service LaaS CSP … and each one has dev, staging and production environments
  72. So what does it mean?

  73. We have to throw rather a lot of RAM at

    the data nodes.
  74. We’ve found that 240GB appeases the Shard Gods.

  75. The Atlassian sales model is “land and expand”

  76. That’s exactly what Elastic is doing at Atlassian!

  77. Done! SAM DAY • SOFTWARE PERSON • ATLASSIAN • @SAMCDAY