Cooking up Splunk: Solving Cloud-scale Problems With Splunk at Cerner

SplunkConf - Cooking up Splunk: Solving Cloud-scale Problems With Splunk at Cerner 10/08/2014 Las Vegas, NV

1. Copyright © 2014 Cerner Corp. & Splunk Inc. Aaron Blythe Knowledge Architect Charlie Huggard Software Architect Cooking up Splunk
2. Agenda Agenda Obligatory Disclaimers What is your name? What is your quest? Why are you in this hand basket? Where are you going?
3. Here be dragons
4. Marketing Corrections We have ~800 Splunk users not 8000 We currently use Splunk for Cloud Solutions Not for Hospital Beds just yet. Public Domain Image: The Tango! Desktop Project
5. Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the companies. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk and Cerner undertake no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
6. Bear Disclaimer During the course of this presentation, we may include pictures regarding fuzzy bears or other cute stuffed animals. We caution you that such pictures reflect our current expectations and estimates of cuddling based on factors currently known to us and a general lack of impulse control that lead us to acquire such levels of stuffed animals in the first place. Your actual collection of such animals will differ materially. Seriously. One of the bears in this presentation is Sir Winston Leonard Spencer Churchbear. He’s six feet tall and commands respect. You’ll also see the aptly named over 30 year old “Brown Bear.” Don’t laugh, he’s my childhood teddy bear and is awesome. For important factors that may cause your collection to differ from those contained in our adorable-looking pictures, please review those available at the FAO Schwartz or other fine retailers. The confession to my obsession made in the this presentation is being made as of the time and date of this live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information as by that time it will be much worse. We do not assume any obligation to update any of you with how adorable our collections are, but feel free to engage us on Twitter and Facebook and share your own.
7. Who are you?
8. Now Appearing Charlie Huggard Aaron Blythe
13. But how does Splunk fit in?
15. Brown Bear
17. B.B. B.B’s Payor B.B.’s Healthcare Providers
18. B.B. B.B’s Payor B.B.’s Healthcare Providers
19. Sisyphus by Titian, ca 1548 Source: http://commons.wikimedia.org/wiki/ File:Punishment_sisyph.jpg
20. ?
21. B.B. B.B’s Payor B.B.’s Healthcare Providers and a few million of his closest friends
24. Public Domain Image: The Tango! Desktop Project
25. Public Domain Image: The Tango! Desktop Project
26. By Christoph Neumüller From Wikimedia Commons
27. Bears! Tigers! Lions?
30. License Envy 100GB/day
31. Splunk 4 Architecture Development Pre-Production / QA Production
34. If When things fail Development to Production Logos refer to respective products
35. Crash Email Alerts Image Source: https://www.flickr.com/photos/barryskeates/7717816416/ Creative Commons Attribution License 2.0 60+ / Week < 1 / Month
36. In 2013 alone 350+ Issues Identified or Resolved using Splunk
37. Much Training
38. The Upgrade
40. New Hardware Index Clustering Revisit Security
43. Why?
44. Why? CC BY 2.0 Source: https://www.flickr.com/photos/atxjen/2626148
46. Le penseur de la Porte d’Enfer. CC BY 2.0 Licensed picture Source: http://www.flickr.com/photos/dalbera/4528252054/
47. Public Domain Image: The Tango! Desktop Project
48. Public Domain Image: The Tango! Desktop Project Public domain image: http://commons.wikimedia.org/wiki/File:Balde Indexes
49. Built in Indexes main summary _internal _audit
50. CC BY 2.0 Source: https://www.flickr.com/photos/peagreenchick/384744
51. Rusty Lock Blue Doors Micanopy CC BY 2.0 Licensed picture Source: https://www.flickr.com/photos/42954113@N00/4877729115
52. SCARY ACRONYMS
53. > 100 per environment Public Domain Image: The Tango! Desktop Project
54. Default Retention 500 GB / 6 Years (per index)
56. Our Retention 320 GB / 2 Years (default per index, production)
57. Our Retention 160 GB / 1 Year (default per index, nonproduction)
58. Our Retention 80 GB / 6 Months (default per index, development)
60. New Data added after Splunk 6 Splunk 4 Data in Splunk 6
62. $SPLUNK_HOME/etc/system/local/outputs.conf [tcpout] indexAndForward = true ...
64. +100GB/day 100GB/day 200GB/day License Envy Redux
66. $SPLUNK_HOME/etc/system/local/outputs.conf [tcpout:environment] server=indexer1:9997, indexer2:9997 ...
68. spreceiver IN A 10.x.x.1
 IN A 10.x.x.2
69. $SPLUNK_HOME/etc/system/local/outputs.conf [tcpout:environment] server=spreceiver:9997 ...
70. spreceiver IN A 10.x.x.1
 IN A 10.x.x.2
71. spreceiver IN A 10.y.y.1
 IN A 10.y.y.2
72. CC BY 2.0 Source: https://www.flickr.com/photos/atxjen/2626148
73. More Splunk Wins
74. Developers on Support Rotations
75. Developers on Support Rotations + =
76. Identifying Anomalies
77. Measuring and Improving Performance
79. Where next?
80. cerner_splunk Cookbook Used to configure Forwarders and Server Clusters LDAP Authentication, Roles, Indexes, etc. Will be open sourced very soon Will be announced on our Engineering Blog http://engineering.cerner.com/ https://github.com/cerner/ cerner_splunk
81. • Windows Support • Search Head Clustering – Rumor: Splunk 6.2 ? • Multi-Site Clustering – Prepare for the Zombie Apocalypse ☺ • Installing Packaged Splunk Apps via Chef – SplunkBase API? • Healthcare Splunk Community? – https://connect.ucern.com/community/udevelop/ splunk
82. Also check out these other talks Presenters: – Chris Hogan & Tom Twait Topics: – Electronic Data Interchange – Bending DBConnect – Alerting Presenter: – Ant Lefebvre Topics: – HIPAA – Meaningful Use – Being a IT Superhero
83. CC BY 2.0 Image: https://www.flickr.com/photos/popculturegeek/5134039427
84. THANK YOU @ablythe : Aaron @acharlieh : Charlie