2013 September Assimilation presentation to TWC

We provide open source discovery with zero network footprint integrated with highly-scalable monitoring - http://assimproj.org

Transcript

1. IT Discovery and Monitoring Without Limit using The Assimilation Project #AssimProj @OSSAlanR http://assimproj.org/ http://bit.ly/AssimTWC2013 Alan Robertson <alanr@unix.sh> Assimilation Systems Limited http://assimilationsystems.com
2. Upcoming Events Time-Warner Cable Developer Conference National Center for Atmospheric Research Denver Open Source User’s Group GraphConnect San Francisco Open Source Monitoring Conference - Nürnberg NSA / Homeland Security Assimilation Technical Talk Large Installation System Administration Conference - DC Colorado Springs Open Source User’s Group linux.conf.au – Linux Conference in Australia - Perth
3. 12 September 2013 © 2013 Assimilation Systems Limited 3/37 Discovery Discovering • systems you've forgotten • what you're not monitoring • whatever you'd like • without setting off security alarms
4. 12 September 2013 © 2013 Assimilation Systems Limited 4/37 Monitoring Monitoring • extreme scale • topology aware • integrated with discovery • easy-to-configure
5. 12 September 2013 © 2013 Assimilation Systems Limited 5/37 Assimilation Project History • Inspired by 2 million core computer • Concerns for extreme scale • Topology aware monitoring • Topology discovery w/out security issues • Discovery of everything!
6. 12 September 2013 © 2013 Assimilation Systems Limited 6/37 Project Scope Zero-network-footprint continuous Discovery integrated with extreme-scale Monitoring • Continuous extensible discovery – systems, switches, services, dependencies – zero network footprint • Extensible exception monitoring – more than 100K systems • All data goes into central graph database
7. 12 September 2013 © 2013 Assimilation Systems Limited 7/37 Why Assimilation Software? • Management Perspective • DevOps Perspective
8. 12 September 2013 © 2013 Assimilation Systems Limited 8/37 Risk Management/Mitigation • Intrusions • Licensed Software • Audit Risk • Outages • System management
9. 12 September 2013 © 2013 Assimilation Systems Limited 9/37 Why Discovery? (DevOps) • Documentation: incomplete, incorrect • Dependencies: unknown • Planning: Needs accurate data • Best Practices: Verification needs data • ITIL CMDB (Configuration Mgmt DataBase)
10. 12 September 2013 © 2013 Assimilation Systems Limited 10/37 Why Our Monitoring? • Simpler to configure (in theory) • Growth is non-issue • Extremely low network traffic • Ideal for cross-WAN monitoring • Highlight cascading failure root causes • Not confused by switch failures • Most switches get monitored “for free”
11. 12 September 2013 © 2013 Assimilation Systems Limited 11/37 This all sounds unreasonable... • Huge scalability without complexity? • Discovery without sending packets? Really?
12. 12 September 2013 © 2013 Assimilation Systems Limited 12/37 Architectural Overview Collective Management Authority • One CMA per installation Nanoprobes • One nanoprobe per OS image Data Storage • Central Neo4j graph database General Rule: “No News Is Good News”
13. 12 September 2013 © 2013 Assimilation Systems Limited 13/37 Simple Scalability • I can explain how we scale so your grandmother would understand
14. 12 September 2013 © 2013 Assimilation Systems Limited 14/37 Massive Scalability – or “I see dead servers in O(1) time” • Adding systems does not increase the monitoring work on any system • Each server monitors 2 (or 4) neighbors • Each server monitors its own services • Ring repair and alerting is O(n) – but a very small amount of work • Ring repair for a million nodes is less than 10K packets per day (approximately 1 packet per 9 seconds) Current Implementation
15. 12 September 2013 © 2013 Assimilation Systems Limited 15/37 Decreasing Network Footprint (planned) • Support diagnosing switch issues • Minimize network traffic • Ideal for multi-site arrangements
16. 12 September 2013 © 2013 Assimilation Systems Limited 16/37 Service Monitoring Based on Linux-HA LRM • LRM == Local Resource Manager • Well-proven architecture: – “no news is good news” AKA management by exception • Implements Open Cluster Framework standard (and others) • Each system monitors own services • Can also start, stop, migrate services
17. 12 September 2013 © 2013 Assimilation Systems Limited 17/37 Monitoring Pros and Cons Pros Simple & Scalable Uniform work distribution No single point of failure Distinguishes switch vs host failure Easy on LAN, WAN Cons Active agents Potential slowness at power-on
18. 12 September 2013 © 2013 Assimilation Systems Limited 18/37 How does this apply to clouds? • Fits nicely into a cloud infrastructure – Should integrate into OpenStack, et al – Can control VMs • Can monitor customer VMs – Add nanoprobe to base image – bottom level of rings disappear without LLDP or CDP
19. 12 September 2013 © 2013 Assimilation Systems Limited 19/37 Architectural Details • Nanoprobes • CMA • Neo4j
20. 12 September 2013 © 2013 Assimilation Systems Limited 20/37 Nanoprobe Functions ('C') Announce self to CMA • Reserved multicast address (can be unicast address or name if no multicast) Do what CMA says • receive configuration information – CMA addresses, ports, defaults • send/expect heartbeats • perform discovery actions • perform monitoring actions No persistent state across reboots
21. 12 September 2013 © 2013 Assimilation Systems Limited 21/37 Basic CMA Functions (python) Nanoprobe management • Configure & direct • Hear alerts & discovery • Update rings: join/leave Update database Issue alerts
22. 12 September 2013 © 2013 Assimilation Systems Limited 23/37 Why a graph database? (Neo4j) • Dependency & Discovery information: graph • Speed of graph traversals depends on size of subgraph, not total graph size • Root cause queries  graph traversals – notoriously slow in relational databases • Visualization of relationships • Schema-less design: good for constantly changing heterogeneous environment
23. 12 September 2013 © 2013 Assimilation Systems Limited 24/37 How does discovery work? Nanoprobe scripts perform discovery • Each discovers one kind of information • Can take arguments (in environment) • Output JSON CMA stores Discovery Information • JSON stored in Neo4j database • CMA discovery plugins => graph nodes and relationships
24. 12 September 2013 © 2013 Assimilation Systems Limited 25/37 sshd Service JSON Snippet (from netstat and /proc) "sshd": { "exe": "/usr/sbin/sshd", "cmdline": [ "/usr/sbin/sshd", "-D" ], "uid": "root", "gid": "root", "cwd": "/", "listenaddrs": { "0.0.0.0:22": { "proto": "tcp", "addr": "0.0.0.0", "port": 22 }, and so on...
25. 12 September 2013 © 2013 Assimilation Systems Limited 26/37 ssh Client JSON Snippet (from netstat and /proc) "ssh": { "exe": "/usr/sbin/ssh", "cmdline": [ "ssh", "servidor" ], "uid": "alanr", "gid": "alanr", "cwd": "/home/alanr/monitor/src", "clientaddrs": { "10.10.10.5:22": { "proto": "tcp", "addr": "10.10.10.5", "port": 22 }, and so on...
26. 12 September 2013 © 2013 Assimilation Systems Limited 27/37 ssh -> sshd dependency graph
27. 12 September 2013 © 2013 Assimilation Systems Limited 28/37 Switch Discovery Data from LLDP (or CDP) CRM transforms LLDP (CDP) Data to JSON
28. 12 September 2013 © 2013 Assimilation Systems Limited 29/37 Current State • First release was April 2013 • Great unit test infrastructure • Nanoprobe code – works well • Service monitoring works • Lacking real digital signatures, encryption, compression • Reliable UDP comm code all working • CMA code works, much more to go • Several discovery methods written • Licensed under the GPL
29. 12 September 2013 © 2013 Assimilation Systems Limited 30/37 Future Plans • Production grade by end of year • Support, commercial licenses • “Real digital signatures, compression, encryption • Other security enhancements • Much more discovery • GUI • Alerting • Reporting • Add Statistical Monitoring • Best Practice Audits • Dynamic (aka cloud) specialization • Hundreds more ideas – See: https://trello.com/b/OpaED3AT
30. 12 September 2013 © 2013 Assimilation Systems Limited 31/37 Get Involved! Powerful Ideas and Infrastucture Fun, ground-breaking project Looking for early adopters, testers!! Needs for every kind of skill • Awesome User Interfaces (UI/UX) • Evangelism, community building • Test Code (simulate 106 servers!) • Python, C, script coding • Documentation • Feedback: Testing, Ideas, Plans • Many others!
31. 12 September 2013 © 2013 Assimilation Systems Limited 32/37 Resistance Is Futile! #AssimProj @OSSAlanR #AssimMon Project Web Site http://assimproj.org Blog techthoughts.typepad.com lists.community.tummy.com/cgi-bin/mailman/admin/assimilation