2013 August BLUG - Discovery and Monitoring Without Limit

Discovery and Monitoring
Without Limit
using
The Assimilation Project
#AssimProj @OSSAlanR
http://assimproj.org/
Alan Robertson
Assimilation Systems Limited
http://assimilationsystems.com

View Slide

8 August 2013
© 2013 Assimilation Systems Limited 2/33
Biography
●
Founded Linux-HA project - led 1998-2007 -
now called Pacemaker
●
Founded Assimilation Project in 2010
●
Founded Assimilation Systems Limited in
2013
●
Alumnus of Bell Labs, SuSE, IBM

View Slide

8 August 2013
Project background
●
Available as GPL (or commercial)
– your fork/exec scripts not required to be GPL
●
Founded in late 2010
●
Now my full time endeavor
– Assimilation Systems Limited
●
Currently around 25K lines of code
●
First release: April 2013

View Slide

8 August 2013
T.A.N.S.T.A.A.F.L.
What I need from you...
●
Feedback on the project/product
– Is it useful – why or why not?
– Would it sell to management?
●
Feedback on my approach to presenting it
●
Other presentation feedback
– Clarity, Style, etc...

View Slide

8 August 2013
Project Scope
Zero-network-footprint continuous Discovery
integrated with extreme-scale Monitoring
●
Continuous extensible discovery
– systems, switches, services, dependencies
– zero network footprint
●
Extensible exception monitoring
– more than 100K systems
●
All data goes into central graph database

View Slide

8 August 2013
Questions
●
How many of you have monitoring?
– Open or closed source?
– How many of you are happy with it?
●
How many of you have discovery?
– Open or closed source?
– Is it continuous?
– How many of you are happy with it?

View Slide

8 August 2013
Why Assimilation Software?
●
Management Perspective
●
DevOps Perspective

View Slide

8 August 2013
Risk Management/Mitigation
●
Intrusions
●
Licensed Software
●
Audit Risk
●
Outages
●
System management

View Slide

8 August 2013
Why Discovery? (DevOps)
●
Documentation: incomplete, incorrect
●
Dependencies: unknown
●
Planning: Needs accurate data
●
Best Practices: Verification needs
data
●
ITIL CMDB (Configuration Mgmt
DataBase)

View Slide

8 August 2013
Why Our Monitoring?
●
Simpler to configure (in theory)
●
Growth is non-issue
●
Extremely low network traffic
●
Ideal for cross-WAN monitoring
●
Highlight cascading failure root causes
●
Not confused by switch failures
●
Most switches get monitored “for free”

View Slide

8 August 2013
This all sounds unreasonable...
●
Huge scalability without complexity?
●
Discovery without sending packets?
Really?

View Slide

8 August 2013
Architectural Overview
Collective Management Authority
●
One CMA per installation
Nanoprobes
●
One nanoprobe per OS image
Data Storage
●
Central Neo4j graph database
General Rule: “No News Is Good News”

View Slide

8 August 2013
Simple Scalability
●
I can explain how we scale so
your grandmother would
understand

View Slide

8 August 2013
Massive Scalability – or
“I see dead servers in O(1) time”
●
Adding systems does not increase the monitoring work on any
system
●
Each server monitors 2 (or 4) neighbors
●
Each server monitors its own services
●
Ring repair and alerting is O(n) – but a very small amount of work
●
Ring repair for a million nodes is less than 10K packets per day
(approximately 1 packet per 9 seconds)
Current Implementation

View Slide

8 August 2013
Decreasing Network Footprint
(planned)
●
Support diagnosing switch issues
●
Minimize network traffic
●
Ideal for multi-site arrangements

View Slide

8 August 2013
Service Monitoring
Based on Linux-HA LRM
●
LRM == Local Resource Manager
●
Well-proven architecture:
– “no news is good news” AKA
management by exception
●
Implements Open Cluster Framework
standard (and others)
●
Each system monitors own services
●
Can also start, stop, migrate services

View Slide

8 August 2013
Monitoring Pros and Cons
Pros
Simple & Scalable
Uniform work
distribution
No single point of
failure
Distinguishes switch
vs host failure
Easy on LAN, WAN
Cons
Active agents
Potential slowness at
power-on

View Slide

8 August 2013
How does this apply to clouds?
●
Fits nicely into a cloud infrastructure
– Should integrate into OpenStack, et al
– Can control VMs
●
Can monitor customer VMs
– Add nanoprobe to base image
– bottom level of rings disappear without
LLDP or CDP

View Slide

8 August 2013
Architectural Details
●
Nanoprobes
●
CMA
●
Neo4j

View Slide

8 August 2013
Nanoprobe Functions ('C')
Announce self to CMA
●
Reserved multicast address (can be
unicast address or name if no multicast)
Do what CMA says
●
receive configuration information
– CMA addresses, ports, defaults
●
send/expect heartbeats
●
perform discovery actions
●
perform monitoring actions
No persistent state across reboots

View Slide

8 August 2013
Basic CMA Functions (python)
Nanoprobe management
●
Configure & direct
●
Hear alerts & discovery
●
Update rings: join/leave
Update database
Issue alerts

View Slide

8 August 2013
Why a graph database? (Neo4j)
●
Dependency & Discovery information: graph
●
Speed of graph traversals depends on size
of subgraph, not total graph size
●
Root cause queries  graph traversals –
notoriously slow in relational databases
●
Visualization of relationships
●
Schema-less design: good for constantly
changing heterogeneous environment

View Slide

8 August 2013
How does discovery work?
Nanoprobe scripts perform discovery
●
Each discovers one kind of information
●
Can take arguments (in environment)
●
Output JSON
CMA stores Discovery Information
●
JSON stored in Neo4j database
●
CMA discovery plugins => graph nodes and
relationships

View Slide

8 August 2013
sshd Service JSON Snippet
(from netstat and /proc)
"sshd": {
"exe": "/usr/sbin/sshd",
"cmdline": [ "/usr/sbin/sshd", "-D" ],
"uid": "root",
"gid": "root",
"cwd": "/",
"listenaddrs": {
"0.0.0.0:22": {
"proto": "tcp",
"addr": "0.0.0.0",
"port": 22
}, and so on...

View Slide

8 August 2013
ssh Client JSON Snippet
(from netstat and /proc)
"ssh": {
"exe": "/usr/sbin/ssh",
"cmdline": [ "ssh", "servidor" ],
"uid": "alanr",
"gid": "alanr",
"cwd": "/home/alanr/monitor/src",
"clientaddrs": {
"10.10.10.5:22": {
"proto": "tcp",
"addr": "10.10.10.5",
"port": 22
}, and so on...

View Slide

8 August 2013
ssh -> sshd dependency graph

View Slide

8 August 2013
Switch Discovery Data
from LLDP (or CDP)
CRM transforms LLDP (CDP) Data to JSON

View Slide

8 August 2013
Current State
●
First release was April 2013
●
Great unit test infrastructure
●
Nanoprobe code – works well
●
Service monitoring works
●
Lacking real digital signatures, encryption,
compression
●
Reliable UDP comm code all working
●
CMA code works, much more to go
●
Several discovery methods written
●
Licensed under the GPL

View Slide

8 August 2013
Future Plans
●
Production grade by end of year
●
Support, commercial licenses
●
“Real digital signatures, compression, encryption
●
Other security enhancements
●
Much more discovery
●
GUI
●
Alerting
●
Reporting
●
Add Statistical Monitoring
●
Best Practice Audits
●
Dynamic (aka cloud) specialization
●
Hundreds more ideas
– See: https://trello.com/b/OpaED3AT

View Slide

8 August 2013
Get Involved!
Powerful Ideas and Infrastucture
Fun, ground-breaking project
Looking for early adopters, testers!!
Needs for every kind of skill
●
Awesome User Interfaces (UI/UX)
●
Evangelism, community building
●
Test Code (simulate 106 servers!)
●
Python, C, script coding
●
Documentation
●
Feedback: Testing, Ideas, Plans
●
Many others!

View Slide

8 August 2013
Resistance Is Futile!
#AssimProj @OSSAlanR
#AssimMon
Project Web Site
http://assimproj.org
Blog
techthoughts.typepad.com
lists.community.tummy.com/cgi-bin/mailman/admin/assimilation

View Slide

2013 August BLUG - Discovery and Monitoring Without Limit

2013 August BLUG - Discovery and Monitoring Without Limit

Alan Robertson

More Decks by Alan Robertson

Featured

Transcript