Elastic{ON} 2018 - Reinventing Fermilab's Scientific Computing Grid Accounting with the Elastic Stack

Transcript

1. Fermi National Accelerator Laboratory (Fermilab) February 28, 2018 @kevin_retzke Reinventing Fermilab's Scientific Computing Grid Accounting with the Elastic Stack Kevin Retzke, App. Dev. and Sys. Analyst
2. K.M.Retzke/Fermilab About me
3. Degree in Nuclear Engineering from Purdue University
4. Performed analysis of reactor vessels and dry cask storage
5. Since a young age my passion was in development
6. Distributed computing support and monitoring at Fermilab
7. K.M.Retzke/Fermilab About Fermilab
8. K.M.Retzke/Fermilab
9. K.M.Retzke/Fermilab
10. K.M.Retzke/Fermilab
11. K.M.Retzke/Fermilab
12. K.M.Retzke/Fermilab
13. K.M.Retzke/Fermilab
14. K.M.Retzke/Fermilab
15. K.M.Retzke/Fermilab
16. K.M.Retzke/Fermilab Protons begin at the Cockcroft-Walton Generator Accelerators
17. K.M.Retzke/Fermilab Protons begin at the Cockcroft-Walton Generator
18. K.M.Retzke/Fermilab Linac accelerates them to 400 MeV
19. K.M.Retzke/Fermilab Booster boosts them to 8 GeV (0.99 c)
20. K.M.Retzke/Fermilab Main Ring drives them to design energy of 200 GeV (0.99999 c)
21. K.M.Retzke/Fermilab Then to 500 GeV (0.999999 c) after four years of development
22. K.M.Retzke/Fermilab Such high-energy protons facilitate many areas of research… Research
23. Neutron and proton cancer therapy
24. Particle observation in the world’s largest bubble chamber
25. Discovery of the bottom quark in 1977
26. K.M.Retzke/Fermilab The super- conducting Tevatron was commissioned in 1985 The Tevatron
27. K.M.Retzke/Fermilab
28. K.M.Retzke/Fermilab CDF DØ
29. The top quark was discovered in 1995
30. The tau neutrino discovered in 2000
31. K.M.Retzke/Fermilab The Tevatron was shut down in 2011 to focus on the new LHC at CERN
32. K.M.Retzke/Fermilab
33. The Higgs boson discovered at CERN in 2012 CMS
34. K.M.Retzke/Fermilab The Main Injector was commissioned in 1999 Neutrinos and Muons
35. The Main Injector/ Recycler combines batches of protons from the Booster into highly intense beams
36. K.M.Retzke/Fermilab The Main Injector was commissioned in 1999
37. K.M.Retzke/Fermilab The MINOS experiment observes neutrino oscillations MINOS
38. K.M.Retzke/Fermilab The MINOS experiment observes neutrino oscillations
39. K.M.Retzke/Fermilab The massive NOvA far detector is a further 50 miles away NOvA
40. K.M.Retzke/Fermilab µBooNE represents the next generation of high-resolution neutrino detectors
41. K.M.Retzke/Fermilab DUNE
42. K.M.Retzke/Fermilab
43. K.M.Retzke/Fermilab Muon g-2
44. K.M.Retzke/Fermilab Mu2e
45. K.M.Retzke/Fermilab The Dark Energy Survey looks to the sky Dark Energy Survey
46. K.M.Retzke/Fermilab The Dark Energy Survey looks to the sky
48. Joint observation of cosmic phenomena by multiple carriers yields incredible information about the universe
49. K.M.Retzke/Fermilab Computing
50. Science drove early advancements in high-performance supercomputing
51. Today researchers run high-throughput computing on thousands of commodity servers
52. Data is stored on 40 PiB of disk and 130 PiB of tape archive
53. Data collection and analysis jobs transfer over 20 PiB per month
54. K.M.Retzke/Fermilab Monitoring such complex batch and data Monitoring
55. K.M.Retzke/Fermilab FIFEBatch FermiGrid CMS LPC CMS T1 dCache FTS IFDH NAS SAM GlideinWMS JobSub Enstore EOS CASTOR AWS EC2 NERSC VOMS GUMS FermiCloud SNOW
56. K.M.Retzke/Fermilab FIFEBatch FermiGrid CMS LPC CMS T1 dCache FTS IFDH NAS SAM GlideinWMS JobSub Enstore EOS CASTOR AWS EC2 NERSC VOMS GUMS FermiCloud SNOW getData() pushData()
57. K.M.Retzke/Fermilab FIFEBatch FermiGrid CMS LPC CMS T1 dCache FTS IFDH NAS SAM GlideinWMS JobSub Enstore EOS CASTOR AWS EC2 NERSC VOMS GUMS FermiCloud SNOW getData() pushData()
58. K.M.Retzke/Fermilab FIFEBatch FermiGrid CMS LPC CMS T1 dCache FTS IFDH NAS SAM GlideinWMS JobSub Enstore EOS CASTOR AWS EC2 NERSC VOMS GUMS FermiCloud SNOW getData() pushData()
59. K.M.Retzke/Fermilab FIFEBatch FermiGrid CMS LPC CMS T1 dCache FTS IFDH NAS SAM GlideinWMS JobSub Enstore EOS CASTOR AWS EC2 NERSC VOMS GUMS FermiCloud SNOW getData() pushData()
60. K.M.Retzke/Fermilab FIFEBatch FermiGrid CMS LPC CMS T1 dCache FTS IFDH NAS SAM GlideinWMS JobSub Enstore EOS CASTOR AWS EC2 NERSC VOMS GUMS FermiCloud SNOW getData() pushData()
61. K.M.Retzke/Fermilab
62. K.M.Retzke/Fermilab
63. K.M.Retzke/Fermilab
64. K.M.Retzke/Fermilab
65. Monitoring based on the Elastic stack is growing throughout the physics community
66. K.M.Retzke/Fermilab The Open Science Grid connects scientific computing sites worldwide Open Science Grid
67. Distributing jobs among sites makes unused resources available to others
68. Scientists from organizations large and small get access to vast computing resources
69. K.M.Retzke/Fermilab OSG Accounting • OSG management and stakeholders • Site admins and stakeholders • VO/experiment coordinators and support • Researchers
70. Gratia was developed for the OSG ca. 2005 HTCondor dCache SLURM PBS Enstore FermiCloud AWS EC2 BOINC <UsageRecord></UsageRecord> Gratia Collector MySQL
71. K.M.Retzke/Fermilab <JobUsageRecord xmlns="http://www.gridforum.org/2003/ur-wg" xmlns:urwg="http://www.gridforum.org/2003/ur-wg" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.gridforum.org/2003/ur-wg file:///u:/OSG/urwg-schema.11.xsd"> <RecordIdentity urwg:createTime="2016-05-02T16:36:19Z" urwg:recordId="ce01.brazos.tamu.edu:3976909.14"/> <JobIdentity> <GlobalJobId>slurm:SLURM/brazos/brazos.13634504_4</GlobalJobId> <LocalJobId>13634504_4</LocalJobId> </JobIdentity> <UserIdentity> <LocalUserId>georgemm01</LocalUserId> <VOName>cms</VOName> <ReportableVOName>CMS</ReportableVOName> </UserIdentity> <JobName>CDMSBulldozer</JobName> <Status>0</Status> <Processors urwg:metric="total">1</Processors> <WallDuration>PT21.0S</WallDuration> <CpuDuration urwg:description="Was entered in seconds" urwg:usageType="user">PT0.55S</CpuDuration> <CpuDuration urwg:description="Was entered in seconds" urwg:usageType="system">PT0.26S</CpuDuration> <EndTime urwg:description="Was entered in seconds">2016-05-02T16:00:46Z</EndTime> <StartTime urwg:description="Was entered in seconds">2016-05-02T16:00:25Z</StartTime> <Queue>stakeholder</Queue> <ProjectName>hepx</ProjectName> <Processors urwg:metric="total">1</Processors> <ProbeName>slurm:ce01.brazos.tamu.edu</ProbeName> <SiteName>TAMU_BRAZOS_CE</SiteName> <Grid>Local</Grid> <Njobs>1</Njobs> <Resource urwg:description="ResourceType">Batch</Resource> </JobUsageRecord>
72. K.M.Retzke/Fermilab
73. K.M.Retzke/Fermilab
74. K.M.Retzke/Fermilab
75. Over the years the OSG has grown
76. Techical debt also grew
77. K.M.Retzke/Fermilab In early 2016 OSG decided to re-design Gratia GRÅCC • Flexible architecture and storage format • Scalable and evolvable design • Utilize open-source visualization tools • Responsive multi-year queries • Preserve historical data • Maintain compatibility with existing probes
78. K.M.Retzke/Fermilab
79. The gracc-collector provides a Gratia- compatible interface
80. RabbitMQ distributes and queues records
81. Logstash fingerprints and stores records into Elasticsearch
82. Periodic summarization process rolls up records into daily aggregations
83. Visualizations moved to Grafana and Kibana
84. K.M.Retzke/Fermilab
85. K.M.Retzke/Fermilab
86. K.M.Retzke/Fermilab
87. K.M.Retzke/Fermilab
88. K.M.Retzke/Fermilab GRACC was deployed in mid 2017 What did we learn?
89. Grafana or Kibana? Yes!
90. Version your indices and use aliases
91. Carefully review and evaluate defaults for your use case
92. Use rollup aggregations to speed up common queries
93. Liberally add derived fields to your documents and use scripted fields sparingly
94. Schema-less really isn’t
95. Running Elasticsearch on VMs with distributed filesystems can work…
96. Upgrade often, don’t let technical debt build
97. Elasticsearch has enabled more accessible insight into grid accounting and monitoring
98. K.M.Retzke/Fermilab Thanks to my colleagues at Fermilab and the GRACC project team at OSG Thanks to Elastic and the thousands of other contributors and supporters of open source software Fermilab is operated by Fermi Research Alliance, LLC for the U.S. Department of Energy Office of Science The Open Science Grid is funded by the U. S. Department of Energy and the National Science Foundation Acknowledgements
99. K.M.Retzke/Fermilab Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. Any opinions are solely those of the speaker. Disclaimer
100. K.M.Retzke/Fermilab More Questions? Visit us at the AMA Online at www.fnal.gov Or visit the lab in person! Hi!