Cooking up Splunk: Solving Cloud-scale Problems With Splunk at Cerner

Cooking up Splunk: Solving Cloud-scale Problems With Splunk at Cerner

SplunkConf - Cooking up Splunk: Solving Cloud-scale Problems With Splunk at Cerner 10/08/2014 Las Vegas, NV

260a95e08b7880ecd76b964203f25c87?s=128

Aaron Blythe

October 08, 2014
Tweet

Transcript

  1. Copyright © 2014 Cerner Corp. & Splunk Inc. Aaron Blythe

    Knowledge Architect Charlie Huggard Software Architect Cooking up Splunk
  2. Agenda Agenda Obligatory Disclaimers What is your name? What is

    your quest? Why are you in this hand basket? Where are you going?
  3. Here be dragons

  4. Marketing Corrections We have ~800 Splunk users not 8000 We

    currently use Splunk for Cloud Solutions Not for Hospital Beds just yet. Public Domain Image: The Tango! Desktop Project
  5. Disclaimer During the course of this presentation, we may make

    forward looking statements regarding future events or the expected performance of the companies. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk and Cerner undertake no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
  6. Bear Disclaimer During the course of this presentation, we may

    include pictures regarding fuzzy bears or other cute stuffed animals. We caution you that such pictures reflect our current expectations and estimates of cuddling based on factors currently known to us and a general lack of impulse control that lead us to acquire such levels of stuffed animals in the first place. Your actual collection of such animals will differ materially. Seriously. One of the bears in this presentation is Sir Winston Leonard Spencer Churchbear. He’s six feet tall and commands respect. You’ll also see the aptly named over 30 year old “Brown Bear.” Don’t laugh, he’s my childhood teddy bear and is awesome. For important factors that may cause your collection to differ from those contained in our adorable-looking pictures, please review those available at the FAO Schwartz or other fine retailers. The confession to my obsession made in the this presentation is being made as of the time and date of this live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information as by that time it will be much worse. We do not assume any obligation to update any of you with how adorable our collections are, but feel free to engage us on Twitter and Facebook and share your own.
  7. Who are you?

  8. Now Appearing Charlie Huggard Aaron Blythe

  9. None
  10. None
  11. None
  12. None
  13. But how does Splunk fit in?

  14. None
  15. Brown Bear

  16. None
  17. B.B. B.B’s Payor B.B.’s Healthcare Providers

  18. B.B. B.B’s Payor B.B.’s Healthcare Providers

  19. Sisyphus by Titian, ca 1548 Source: http://commons.wikimedia.org/wiki/ File:Punishment_sisyph.jpg

  20. ?

  21. B.B. B.B’s Payor B.B.’s Healthcare Providers and a few million

    of his closest friends
  22. None
  23. None
  24. Public Domain Image: The Tango! Desktop Project

  25. Public Domain Image: The Tango! Desktop Project

  26. By Christoph Neumüller From Wikimedia Commons

  27. Bears! Tigers! Lions?

  28. None
  29. None
  30. License Envy 100GB/day

  31. Splunk 4 Architecture Development Pre-Production / QA Production

  32. None
  33. None
  34. If When things fail Development to Production Logos refer to

    respective products
  35. Crash Email Alerts Image Source: https://www.flickr.com/photos/barryskeates/7717816416/ Creative Commons Attribution License

    2.0 60+ / Week < 1 / Month
  36. In 2013 alone 350+ Issues Identified or Resolved using Splunk

  37. Much Training

  38. The Upgrade

  39. None
  40. New Hardware Index Clustering Revisit Security

  41. None
  42. None
  43. Why?

  44. Why? CC BY 2.0 Source: https://www.flickr.com/photos/atxjen/2626148

  45. 45

  46. Le penseur de la Porte d’Enfer. CC BY 2.0 Licensed

    picture Source: http://www.flickr.com/photos/dalbera/4528252054/
  47. Public Domain Image: The Tango! Desktop Project

  48. Public Domain Image: The Tango! Desktop Project Public domain image:

    http://commons.wikimedia.org/wiki/File:Balde Indexes
  49. Built in Indexes main summary _internal _audit

  50. CC BY 2.0 Source: https://www.flickr.com/photos/peagreenchick/384744

  51. Rusty Lock Blue Doors Micanopy CC BY 2.0 Licensed picture

    Source: https://www.flickr.com/photos/42954113@N00/4877729115
  52. SCARY ACRONYMS

  53. > 100 per environment Public Domain Image: The Tango! Desktop

    Project
  54. Default Retention 500 GB / 6 Years (per index)

  55. None
  56. Our Retention 320 GB / 2 Years (default per index,

    production)
  57. Our Retention 160 GB / 1 Year (default per index,

    nonproduction)
  58. Our Retention 80 GB / 6 Months (default per index,

    development)
  59. None
  60. New Data added after Splunk 6 Splunk 4 Data in

    Splunk 6
  61. None
  62. $SPLUNK_HOME/etc/system/local/outputs.conf [tcpout] indexAndForward = true ...

  63. None
  64. +100GB/day 100GB/day 200GB/day License Envy Redux

  65. None
  66. $SPLUNK_HOME/etc/system/local/outputs.conf [tcpout:environment] server=indexer1:9997, indexer2:9997 ...

  67. None
  68. spreceiver IN A 10.x.x.1
 IN A 10.x.x.2

  69. $SPLUNK_HOME/etc/system/local/outputs.conf [tcpout:environment] server=spreceiver:9997 ...

  70. spreceiver IN A 10.x.x.1
 IN A 10.x.x.2

  71. spreceiver IN A 10.y.y.1
 IN A 10.y.y.2

  72. CC BY 2.0 Source: https://www.flickr.com/photos/atxjen/2626148

  73. More Splunk Wins

  74. Developers on Support Rotations

  75. Developers on Support Rotations + =

  76. Identifying Anomalies

  77. Measuring and Improving Performance

  78. None
  79. Where next?

  80. cerner_splunk Cookbook Used to configure Forwarders and Server Clusters LDAP

    Authentication, Roles, Indexes, etc. Will be open sourced very soon Will be announced on our Engineering Blog http://engineering.cerner.com/ https://github.com/cerner/ cerner_splunk
  81. • Windows Support • Search Head Clustering – Rumor: Splunk

    6.2 ? • Multi-Site Clustering – Prepare for the Zombie Apocalypse ☺ • Installing Packaged Splunk Apps via Chef – SplunkBase API? • Healthcare Splunk Community? – https://connect.ucern.com/community/udevelop/ splunk
  82. Also check out these other talks Presenters: – Chris Hogan

    & Tom Twait Topics: – Electronic Data Interchange – Bending DBConnect – Alerting Presenter: – Ant Lefebvre Topics: – HIPAA – Meaningful Use – Being a IT Superhero
  83. CC BY 2.0 Image: https://www.flickr.com/photos/popculturegeek/5134039427

  84. THANK YOU @ablythe : Aaron @acharlieh : Charlie