$30 off During Our Annual Pro Sale. View Details »

Cooking up Splunk: Solving Cloud-scale Problems With Splunk at Cerner

Cooking up Splunk: Solving Cloud-scale Problems With Splunk at Cerner

SplunkConf - Cooking up Splunk: Solving Cloud-scale Problems With Splunk at Cerner 10/08/2014 Las Vegas, NV

Aaron Blythe

October 08, 2014
Tweet

More Decks by Aaron Blythe

Other Decks in Technology

Transcript

  1. Copyright © 2014 Cerner Corp. & Splunk
    Inc.
    Aaron Blythe
    Knowledge Architect
    Charlie Huggard
    Software Architect
    Cooking up Splunk

    View Slide

  2. Agenda
    Agenda
    Obligatory Disclaimers
    What is your name?
    What is your quest?
    Why are you in this hand basket?
    Where are you going?

    View Slide

  3. Here be dragons

    View Slide

  4. Marketing Corrections
    We have ~800 Splunk users not 8000
    We currently use Splunk for Cloud
    Solutions
    Not for Hospital Beds just yet.
    Public Domain Image: The Tango! Desktop Project

    View Slide

  5. Disclaimer
    During the course of this presentation, we may make forward looking statements regarding future events or
    the expected performance of the companies. We caution you that such statements reflect our current
    expectations and estimates based on factors currently known to us and that actual events or results could
    differ materially. For important factors that may cause actual results to differ from those contained in our
    forward-looking statements, please review our filings with the SEC. The forward-looking statements made in
    the this presentation are being made as of the time and date of its live presentation. If reviewed after its
    live presentation, this presentation may not contain current or accurate information. We do not assume any
    obligation to update any forward looking statements we may make. In addition, any information about our
    roadmap outlines our general product direction and is subject to change at any time without notice. It is for
    informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk
    and Cerner undertake no obligation either to develop the features or functionality described or to include
    any such feature or functionality in a future release.

    View Slide

  6. Bear Disclaimer
    During the course of this presentation, we may include pictures regarding fuzzy bears or other cute stuffed
    animals. We caution you that such pictures reflect our current expectations and estimates of cuddling based
    on factors currently known to us and a general lack of impulse control that lead us to acquire such levels of
    stuffed animals in the first place. Your actual collection of such animals will differ materially. Seriously. One
    of the bears in this presentation is Sir Winston Leonard Spencer Churchbear. He’s six feet tall and commands
    respect. You’ll also see the aptly named over 30 year old “Brown Bear.” Don’t laugh, he’s my childhood
    teddy bear and is awesome. For important factors that may cause your collection to differ from those
    contained in our adorable-looking pictures, please review those available at the FAO Schwartz or other fine
    retailers. The confession to my obsession made in the this presentation is being made as of the time and
    date of this live presentation. If reviewed after its live presentation, this presentation may not contain
    current or accurate information as by that time it will be much worse. We do not assume any obligation to
    update any of you with how adorable our collections are, but feel free to engage us on Twitter and Facebook
    and share your own.

    View Slide

  7. Who are you?

    View Slide

  8. Now Appearing
    Charlie Huggard
    Aaron Blythe

    View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. But how does Splunk
    fit in?

    View Slide

  14. View Slide

  15. Brown Bear

    View Slide

  16. View Slide

  17. B.B.
    B.B’s Payor
    B.B.’s Healthcare Providers

    View Slide

  18. B.B.
    B.B’s Payor
    B.B.’s Healthcare Providers

    View Slide

  19. Sisyphus by Titian, ca 1548
    Source: http://commons.wikimedia.org/wiki/
    File:Punishment_sisyph.jpg

    View Slide

  20. ?

    View Slide

  21. B.B.
    B.B’s Payor
    B.B.’s Healthcare Providers
    and a few million of his closest friends

    View Slide

  22. View Slide

  23. View Slide

  24. Public Domain Image: The Tango! Desktop Project

    View Slide

  25. Public Domain Image: The Tango! Desktop Project

    View Slide

  26. By Christoph Neumüller From Wikimedia Commons

    View Slide

  27. Bears! Tigers! Lions?

    View Slide

  28. View Slide

  29. View Slide

  30. License Envy
    100GB/day

    View Slide

  31. Splunk 4 Architecture
    Development Pre-Production /
    QA
    Production

    View Slide

  32. View Slide

  33. View Slide

  34. If When things fail
    Development to Production
    Logos refer to respective products

    View Slide

  35. Crash Email Alerts
    Image Source: https://www.flickr.com/photos/barryskeates/7717816416/
    Creative Commons Attribution License 2.0
    60+ /
    Week
    < 1 /
    Month

    View Slide

  36. In 2013 alone
    350+ Issues
    Identified or Resolved using Splunk

    View Slide

  37. Much Training

    View Slide

  38. The Upgrade

    View Slide

  39. View Slide

  40. New Hardware
    Index Clustering
    Revisit Security

    View Slide

  41. View Slide

  42. View Slide

  43. Why?

    View Slide

  44. Why?
    CC BY 2.0 Source: https://www.flickr.com/photos/atxjen/2626148

    View Slide

  45. 45

    View Slide

  46. Le penseur de la Porte d’Enfer. CC BY 2.0 Licensed picture
    Source: http://www.flickr.com/photos/dalbera/4528252054/

    View Slide

  47. Public Domain Image: The Tango! Desktop Project

    View Slide

  48. Public Domain Image: The Tango! Desktop Project
    Public domain image: http://commons.wikimedia.org/wiki/File:Balde
    Indexes

    View Slide

  49. Built in Indexes
    main
    summary
    _internal
    _audit

    View Slide

  50. CC BY 2.0 Source: https://www.flickr.com/photos/peagreenchick/384744

    View Slide

  51. Rusty Lock Blue Doors Micanopy CC BY 2.0 Licensed picture
    Source: https://www.flickr.com/photos/42954113@N00/4877729115

    View Slide

  52. SCARY
    ACRONYMS

    View Slide

  53. > 100
    per environment
    Public Domain Image: The Tango! Desktop Project

    View Slide

  54. Default Retention
    500 GB / 6
    Years
    (per index)

    View Slide

  55. View Slide

  56. Our Retention
    320 GB / 2
    Years
    (default per index, production)

    View Slide

  57. Our Retention
    160 GB / 1 Year
    (default per index, nonproduction)

    View Slide

  58. Our Retention
    80 GB / 6
    Months
    (default per index, development)

    View Slide

  59. View Slide

  60. New Data added after Splunk
    6
    Splunk 4 Data in Splunk 6

    View Slide

  61. View Slide

  62. $SPLUNK_HOME/etc/system/local/outputs.conf
    [tcpout]
    indexAndForward = true
    ...

    View Slide

  63. View Slide

  64. +100GB/day
    100GB/day
    200GB/day
    License Envy Redux

    View Slide

  65. View Slide

  66. $SPLUNK_HOME/etc/system/local/outputs.conf
    [tcpout:environment]
    server=indexer1:9997,
    indexer2:9997
    ...

    View Slide

  67. View Slide

  68. spreceiver IN A 10.x.x.1

    IN A 10.x.x.2

    View Slide

  69. $SPLUNK_HOME/etc/system/local/outputs.conf
    [tcpout:environment]
    server=spreceiver:9997
    ...

    View Slide

  70. spreceiver IN A 10.x.x.1

    IN A 10.x.x.2

    View Slide

  71. spreceiver IN A 10.y.y.1

    IN A 10.y.y.2

    View Slide

  72. CC BY 2.0 Source: https://www.flickr.com/photos/atxjen/2626148

    View Slide

  73. More Splunk Wins

    View Slide

  74. Developers on Support Rotations

    View Slide

  75. Developers on Support Rotations
    + =

    View Slide

  76. Identifying Anomalies

    View Slide

  77. Measuring and Improving Performance

    View Slide

  78. View Slide

  79. Where next?

    View Slide

  80. cerner_splunk Cookbook
    Used to configure Forwarders and Server Clusters
    LDAP Authentication, Roles, Indexes, etc.
    Will be open sourced very soon
    Will be announced on our Engineering Blog
    http://engineering.cerner.com/
    https://github.com/cerner/
    cerner_splunk

    View Slide

  81. • Windows Support
    • Search Head Clustering
    – Rumor: Splunk 6.2 ?
    • Multi-Site Clustering
    – Prepare for the Zombie Apocalypse ☺
    • Installing Packaged Splunk Apps via Chef
    – SplunkBase API?
    • Healthcare Splunk Community?
    – https://connect.ucern.com/community/udevelop/
    splunk

    View Slide

  82. Also check out these other talks
    Presenters:
    – Chris Hogan & Tom Twait
    Topics:
    – Electronic Data Interchange
    – Bending DBConnect
    – Alerting
    Presenter:
    – Ant Lefebvre
    Topics:
    – HIPAA
    – Meaningful Use
    – Being a IT Superhero

    View Slide

  83. CC BY 2.0 Image: https://www.flickr.com/photos/popculturegeek/5134039427

    View Slide

  84. THANK YOU
    @ablythe : Aaron
    @acharlieh : Charlie

    View Slide