2013 September NCAR presentation

Presentation of the Assimilation Project to NCAR Boulder

Transcript

1. IT Discovery and Monitoring Without Limit using The Assimilation Project #AssimProj @OSSAlanR http://assimproj.org/ http://bit.ly/AssimNCAR2013 Alan Robertson <alanr@unix.sh> Assimilation Systems Limited http://assimilationsystems.com
2. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 2/38 Why Am I Here? • Understand Your Environment and Needs – What you currently do for discovery – What you currently do for monitoring – How those are working for you – Are multi-tenant capabilities appealing? • Give an overview of the project • Engage with your communities • Understand if we ought to stay in touch
3. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 3/38 Upcoming Events National Center for Atmospheric Research (today!) Denver Open Source User’s Group Facebook presentation GraphConnect San Francisco Open Source Monitoring Conference - Nürnberg NSA / Homeland Security Assimilation Technical Talk Large Installation System Administration Conference - DC Colorado Springs Open Source User’s Group linux.conf.au – Linux Conference in Australia - Perth Details on http://assimilationsystems.com/
4. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 4/38 Discovery Discovering • systems you've forgotten • what you're not monitoring • whatever you'd like • without setting off security alarms
5. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 5/38 Monitoring Monitoring • extreme scale • topology aware • integrated with discovery • easy-to-configure
6. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 6/38 Assimilation Project History • Inspired by 2 million core computer (cyclops64) • Concerns for extreme scale • Topology aware monitoring • Topology discovery w/out security issues =►Discovery of everything!
7. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 7/38 Project Scope Zero-network-footprint continuous Discovery integrated with extreme-scale Monitoring • Continuous extensible discovery – systems, switches, services, dependencies – zero network footprint • Extensible exception monitoring – more than 100K systems • All data goes into central graph database
8. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 8/38 Why Assimilation Software? • Management Perspective • DevOps Perspective
9. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 9/38 Risk Management/Mitigation • Intrusions • Licensed Software • Audit Risk • Outages • System management
10. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 10/38 Why Discovery? (DevOps) • Documentation: incomplete, incorrect • Dependencies: unknown • Planning: Needs accurate data • Best Practices: Verification needs data • ITIL CMDB (Configuration Mgmt DataBase) Our Discovery: continuous, low-profile
11. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 11/38 Why Our Monitoring? • Simpler to configure (in theory) • Growth is non-issue • Extremely low network traffic • Ideal for cross-WAN monitoring • Highlight cascading failure root causes • Not confused by switch failures • Most switches get monitored “for free”
12. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 12/38 This all sounds unreasonable... • Huge scalability without complexity? • Discovery without sending packets? Really?
13. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 13/38 Architectural Overview Collective Management Authority • One CMA per installation Nanoprobes • One nanoprobe per OS image Data Storage • Central Neo4j graph database General Rule: “No News Is Good News”
14. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 14/38 Simple Scalability • I can explain how we scale so your grandmother would understand
15. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 15/38 Massive Scalability – or “I see dead servers in O(1) time” • Adding systems does not increase the monitoring work on any system • Each server monitors 2 (or 4) neighbors • Each server monitors its own services • Ring repair and alerting is O(n) – but a very small amount of work • Ring repair for a million nodes is less than 10K packets per day (approximately 1 packet per 9 seconds) Current Implementation
16. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 16/38 Minimizing Network Footprint (planned) • Support diagnosing switch issues • Minimize network traffic • Ideal for multi-site arrangements
17. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 17/38 Service Monitoring Based on Linux-HA LRM • LRM == Local Resource Manager • Well-proven architecture: – “no news is good news” AKA management by exception • Implements Open Cluster Framework standard (and others) • Each system monitors own services • Can also start, stop, migrate services
18. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 18/38 Monitoring Pros and Cons Pros Simple & Scalable Uniform work distribution No single point of failure Distinguishes switch vs host failure Easy on LAN, WAN Multi-tenant approach Cons Active agents Potential slowness at power-on
19. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 19/38 How does this apply to clouds? • Fits nicely into a cloud infrastructure – Should integrate into OpenStack, et al – Can control VMs • Can monitor customer VMs – Add nanoprobe to base image – bottom level of rings disappear without LLDP or CDP
20. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 20/38 Architectural Details • Nanoprobes • CMA • Neo4j
21. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 21/38 Nanoprobe Functions ('C') Announce self to CMA • Reserved multicast address (can be unicast address or name if no multicast) Do what CMA says • receive configuration information – CMA addresses, ports, defaults • send/expect heartbeats • perform discovery actions • perform monitoring actions No persistent state across reboots
22. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 22/38 Basic CMA Functions (python) Nanoprobe management • Configure & direct • Hear alerts & discovery • Update rings: join/leave Update database Issue alerts
23. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 23/38 Why a graph database? (Neo4j) • Dependency & Discovery information: graph • Speed of graph traversals depends on size of subgraph, not total graph size • Root cause queries  graph traversals – notoriously slow in relational databases • Visualization of relationships • Schema-less design: good for constantly changing heterogeneous environment • Graph Model === Object Model
24. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 24/38 How does discovery work? Nanoprobe scripts perform discovery • Each discovers one kind of information • Can take arguments (in environment) • Output JSON CMA stores Discovery Information • JSON stored in Neo4j database • CMA discovery plugins => graph nodes and relationships
25. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 25/38 sshd Service JSON Snippet (from netstat and /proc) "sshd": { "exe": "/usr/sbin/sshd", "cmdline": [ "/usr/sbin/sshd", "-D" ], "uid": "root", "gid": "root", "cwd": "/", "listenaddrs": { "0.0.0.0:22": { "proto": "tcp", "addr": "0.0.0.0", "port": 22 }, and so on...
26. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 26/38 ssh Client JSON Snippet (from netstat and /proc) "ssh": { "exe": "/usr/sbin/ssh", "cmdline": [ "ssh", "servidor" ], "uid": "alanr", "gid": "alanr", "cwd": "/home/alanr/monitor/src", "clientaddrs": { "10.10.10.5:22": { "proto": "tcp", "addr": "10.10.10.5", "port": 22 }, and so on...
27. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 27/38 ssh -> sshd dependency graph
28. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 28/38 Switch Discovery Data from LLDP (or CDP) CRM transforms LLDP (CDP) Data to JSON
29. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 29/38 Current State • First release was April 2013 • Great unit test infrastructure • Nanoprobe code – works well • Service monitoring works • Lacks digital signatures, encryption, compression • Reliable UDP comm code working • Several discovery methods written • CMA and database code restructuring near-complete • UI development underway • Licensed under the GPL, commercial license available
30. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 30/38 Future Plans • Production grade by end of year • Purchased support • “Real digital signatures, compression, encryption • Other security enhancements • Much more discovery • GUI • Alerting • Reporting • Add Statistical Monitoring • Best Practice Audits • Dynamic (aka cloud) specialization • Hundreds more ideas – See: https://trello.com/b/OpaED3AT
31. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 31/38 Get Involved! Powerful Ideas and Infrastucture Fun, ground-breaking project Looking for early adopters, testers!! Needs for every kind of skill • Awesome User Interfaces (UI/UX) • Evangelism, community building • Test Code (simulate 106 servers!) • Python, C, script coding • Documentation • Feedback: Testing, Ideas, Plans • Many others!
32. NCAR 26 September 2013 © 2013 Assimilation Systems Limited 32/38 Resistance Is Futile! #AssimProj @OSSAlanR #AssimMon Project Web Site http://assimproj.org Blog techthoughts.typepad.com lists.community.tummy.com/cgi-bin/mailman/admin/assimilation