Pass On What You Have Learned: Deploying to Production

Presentation about the LLNL experience with deploying Elastic into production over the past year. Discussion of both the good things we've seen, as well as a variety of warts we've found along the way.

Originally presented at: https://elasticpublicsectorsummit.upgather.com/

1. LLNL-PRES-861410 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE- AC52-07NA27344. Lawrence Livermore National Security, LLC Pass On What You Have Learned: Deploying to Production Elastic Public Sector Summit 2024 Ian Lee HPC Security Architect Security Operations Team Lead 2024-03-13
2. 2 LLNL-PRES-861410 Cluster Stats Today
3. 3 LLNL-PRES-861410 El Capitan https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capitan-projected-worlds-fastest-supercomputer
4. 4 LLNL-PRES-861410 • Each bubble represents a cluster • Size reflects theoretical peak performance, color indicates computing zone • Only large user-facing compute clusters are represented El Capitan vs the Rest El Capitan ~ 2 EF (~ 2,000,000 TF)
5. 5 LLNL-PRES-861410 system.syslog per hour
6. 6 LLNL-PRES-861410 § Significantly better performance à Opens new doors for analysis of log data § Splunk (fast mode) — index=lc | stats count by source | sort –count § Elastic (ESQL) — from logs-* | stats count = count() by log.file.path | limit 10000 | sort count desc Performance is Noticeably Better Lookback # documents Splunk (fast mode) Elastic (ESQL) 60 minutes 50 M 133 sec ~ 2 sec 24 hours 1.8 B 2,294 sec ~ 10 sec 7 days 14 B 10,440 sec ~ 20 sec
7. 7 LLNL-PRES-861410 Example: Auditd log issue
8. 8 LLNL-PRES-861410 Jan 1 – Jan 2 (1.25 seconds)
9. 9 LLNL-PRES-861410 Jan 1 – Jan 8 (20 seconds)
10. 10 LLNL-PRES-861410 Nov 1 – Jan 8 (42 seconds)
11. 11 LLNL-PRES-861410 § Explore other offerings — ML / Anomaly Detection — Enterprise Search (unified search across web + confluence + gitlab) — Elastic Defend § More automated alerts / processes Monitoring Vision Going Forward
12. 12 LLNL-PRES-861410
13. 13 LLNL-PRES-861410
14. 14 LLNL-PRES-861410 § We’ve taken a significant hit to capabilities we’ve enjoyed for years. § Operational monitoring of our HPC systems is not as good today as it was with Splunk. — User Experience in particular is not as polished. — Enrich/Transform/Watcher system has been a significant pain point. § Looking forward to partnering with the Elastic and Federal communities further — https://github.com/LLNL/elastic-stacker — Upcoming Continuous Monitoring Dashboard repository — ESQL “Difficult to see; always in motion is the future” - Yoda
15. 15 LLNL-PRES-861410 § Kibana — [Dashboard][Research] Refactor Grid and Layout Systems #88710 ** — Filter only the relevant panels in a dashboard #170395 ** — Sparklines #3395 — Allow dynamic naming of file attachments to watcher emails #169891 — [Fleet] Allow KQL queries with no field specified in fleet endpoints #171425 § Elastic Agent — Reusable integration policies #2227 ** — Fleet Server configuration does not contain all the hosts available in the Elasticsearch cluster #2784 § Elasticsearch — GPU accelerated Machine learning #61690 § Integrations — Gitlab #1741 Ongoing Work We’re Excited About
16. 16 LLNL-PRES-861410 § LLNL product decisions and timelines are decades long — We are expected to deliver on our timelines and roadmaps § Will issues that we care about receive meaningful attention? Product Roadmaps? https://github.com/elastic/kibana/issues/17888
17. Working on similar problems? I would love to chat! ian@llnl.gov @IanLee1521