Share
Embed
2013 September NCAR presentation
Presentation of the Assimilation Project to NCAR Boulder
Alan Robertson
September 26, 2013
Transcript
32 slides
-
IT Discovery and Monitoring Without Limit using The Assimilation Project
#AssimProj @OSSAlanR http://assimproj.org/ http://bit.ly/AssimNCAR2013 Alan Robertson <[email protected]> Assimilation Systems Limited http://assimilationsystems.com -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 2/38
Why Am I Here? • Understand Your Environment and Needs – What you currently do for discovery – What you currently do for monitoring – How those are working for you – Are multi-tenant capabilities appealing? • Give an overview of the project • Engage with your communities • Understand if we ought to stay in touch -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 3/38
Upcoming Events National Center for Atmospheric Research (today!) Denver Open Source User’s Group Facebook presentation GraphConnect San Francisco Open Source Monitoring Conference - Nürnberg NSA / Homeland Security Assimilation Technical Talk Large Installation System Administration Conference - DC Colorado Springs Open Source User’s Group linux.conf.au – Linux Conference in Australia - Perth Details on http://assimilationsystems.com/ -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 4/38
Discovery Discovering • systems you've forgotten • what you're not monitoring • whatever you'd like • without setting off security alarms -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 5/38
Monitoring Monitoring • extreme scale • topology aware • integrated with discovery • easy-to-configure -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 6/38
Assimilation Project History • Inspired by 2 million core computer (cyclops64) • Concerns for extreme scale • Topology aware monitoring • Topology discovery w/out security issues =►Discovery of everything! -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 7/38
Project Scope Zero-network-footprint continuous Discovery integrated with extreme-scale Monitoring • Continuous extensible discovery – systems, switches, services, dependencies – zero network footprint • Extensible exception monitoring – more than 100K systems • All data goes into central graph database -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 8/38
Why Assimilation Software? • Management Perspective • DevOps Perspective -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 9/38
Risk Management/Mitigation • Intrusions • Licensed Software • Audit Risk • Outages • System management -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 10/38
Why Discovery? (DevOps) • Documentation: incomplete, incorrect • Dependencies: unknown • Planning: Needs accurate data • Best Practices: Verification needs data • ITIL CMDB (Configuration Mgmt DataBase) Our Discovery: continuous, low-profile -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 11/38
Why Our Monitoring? • Simpler to configure (in theory) • Growth is non-issue • Extremely low network traffic • Ideal for cross-WAN monitoring • Highlight cascading failure root causes • Not confused by switch failures • Most switches get monitored “for free” -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 12/38
This all sounds unreasonable... • Huge scalability without complexity? • Discovery without sending packets? Really? -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 13/38
Architectural Overview Collective Management Authority • One CMA per installation Nanoprobes • One nanoprobe per OS image Data Storage • Central Neo4j graph database General Rule: “No News Is Good News” -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 14/38
Simple Scalability • I can explain how we scale so your grandmother would understand -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 15/38
Massive Scalability – or “I see dead servers in O(1) time” • Adding systems does not increase the monitoring work on any system • Each server monitors 2 (or 4) neighbors • Each server monitors its own services • Ring repair and alerting is O(n) – but a very small amount of work • Ring repair for a million nodes is less than 10K packets per day (approximately 1 packet per 9 seconds) Current Implementation -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 16/38
Minimizing Network Footprint (planned) • Support diagnosing switch issues • Minimize network traffic • Ideal for multi-site arrangements -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 17/38
Service Monitoring Based on Linux-HA LRM • LRM == Local Resource Manager • Well-proven architecture: – “no news is good news” AKA management by exception • Implements Open Cluster Framework standard (and others) • Each system monitors own services • Can also start, stop, migrate services -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 18/38
Monitoring Pros and Cons Pros Simple & Scalable Uniform work distribution No single point of failure Distinguishes switch vs host failure Easy on LAN, WAN Multi-tenant approach Cons Active agents Potential slowness at power-on -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 19/38
How does this apply to clouds? • Fits nicely into a cloud infrastructure – Should integrate into OpenStack, et al – Can control VMs • Can monitor customer VMs – Add nanoprobe to base image – bottom level of rings disappear without LLDP or CDP -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 20/38
Architectural Details • Nanoprobes • CMA • Neo4j -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 21/38
Nanoprobe Functions ('C') Announce self to CMA • Reserved multicast address (can be unicast address or name if no multicast) Do what CMA says • receive configuration information – CMA addresses, ports, defaults • send/expect heartbeats • perform discovery actions • perform monitoring actions No persistent state across reboots -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 22/38
Basic CMA Functions (python) Nanoprobe management • Configure & direct • Hear alerts & discovery • Update rings: join/leave Update database Issue alerts -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 23/38
Why a graph database? (Neo4j) • Dependency & Discovery information: graph • Speed of graph traversals depends on size of subgraph, not total graph size • Root cause queries graph traversals – notoriously slow in relational databases • Visualization of relationships • Schema-less design: good for constantly changing heterogeneous environment • Graph Model === Object Model -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 24/38
How does discovery work? Nanoprobe scripts perform discovery • Each discovers one kind of information • Can take arguments (in environment) • Output JSON CMA stores Discovery Information • JSON stored in Neo4j database • CMA discovery plugins => graph nodes and relationships -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 25/38
sshd Service JSON Snippet (from netstat and /proc) "sshd": { "exe": "/usr/sbin/sshd", "cmdline": [ "/usr/sbin/sshd", "-D" ], "uid": "root", "gid": "root", "cwd": "/", "listenaddrs": { "0.0.0.0:22": { "proto": "tcp", "addr": "0.0.0.0", "port": 22 }, and so on... -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 26/38
ssh Client JSON Snippet (from netstat and /proc) "ssh": { "exe": "/usr/sbin/ssh", "cmdline": [ "ssh", "servidor" ], "uid": "alanr", "gid": "alanr", "cwd": "/home/alanr/monitor/src", "clientaddrs": { "10.10.10.5:22": { "proto": "tcp", "addr": "10.10.10.5", "port": 22 }, and so on... -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 27/38
ssh -> sshd dependency graph -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 28/38
Switch Discovery Data from LLDP (or CDP) CRM transforms LLDP (CDP) Data to JSON -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 29/38
Current State • First release was April 2013 • Great unit test infrastructure • Nanoprobe code – works well • Service monitoring works • Lacks digital signatures, encryption, compression • Reliable UDP comm code working • Several discovery methods written • CMA and database code restructuring near-complete • UI development underway • Licensed under the GPL, commercial license available -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 30/38
Future Plans • Production grade by end of year • Purchased support • “Real digital signatures, compression, encryption • Other security enhancements • Much more discovery • GUI • Alerting • Reporting • Add Statistical Monitoring • Best Practice Audits • Dynamic (aka cloud) specialization • Hundreds more ideas – See: https://trello.com/b/OpaED3AT -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 31/38
Get Involved! Powerful Ideas and Infrastucture Fun, ground-breaking project Looking for early adopters, testers!! Needs for every kind of skill • Awesome User Interfaces (UI/UX) • Evangelism, community building • Test Code (simulate 106 servers!) • Python, C, script coding • Documentation • Feedback: Testing, Ideas, Plans • Many others! -
NCAR 26 September 2013 © 2013 Assimilation Systems Limited 32/38
Resistance Is Futile! #AssimProj @OSSAlanR #AssimMon Project Web Site http://assimproj.org Blog techthoughts.typepad.com lists.community.tummy.com/cgi-bin/mailman/admin/assimilation