2013 OSMC Assimilation Presentation

O
S
M
C
IT Discovery and Monitoring
Without Limit
using
The Assimilation Project
#AssimProj @OSSAlanR
http://assimproj.org/
http://bit.ly/AssimOSMC2013
Alan Robertson
Assimilation Systems Limited
http://assimilationsystems.com

View Slide

OSMC
24 October
2013
© 2013 Assimilation Systems Limited 2/45
O
S
M
C
Project Scope
Zero-network-footprint continuous Discovery
integrated with extreme-scale Monitoring
●
Continuous extensible discovery
– systems, switches, services, dependencies
– zero network footprint
●
Extensible exception monitoring
– more than 100K systems
●
All data goes into central graph database

View Slide

OSMC
24 October
2013
O
S
M
C
Assimilation Project History
●
Inspired by 2 million core computer (cyclops64)
●
Concerns for extreme scale
●
Topology aware monitoring
●
Topology discovery w/out security issues
=►Discovery of everything!

View Slide

OSMC
24 October
2013
O
S
M
C

View Slide

OSMC
24 October
2013
O
S
M
C
An 8-dimensional overview
●
Problems Addressed
●
Unique Capabilities
●
Distribution of Work
●
Architectural Components
●
Discovery Graph Schema
●
Extensible Discovery API
●
Current Status
●
Project Needs

View Slide

OSMC
24 October
2013
O
S
M
C
First Dimension:
Problems Addressed
Risk Management at extreme scale
1. Maintaining detailed
discovery database
2. Discovering systems
you've forgotten about
3. Discovering what (licensed)
software you're running – and where
4. Monitoring services, systems and
switches
5. Finding services you aren't monitoring

View Slide

OSMC
24 October
2013
O
S
M
C
Risk Management/Mitigation
●
Intrusions
●
Licensed Software
●
Audit Risk
●
Outages
●
System management

View Slide

OSMC
24 October
2013
O
S
M
C
Why Discovery? (DevOps)
●
Documentation: incomplete, incorrect
●
Dependencies: unknown
●
Planning: Needs accurate data
●
Best Practices: Verification needs
data
●
ITIL CMDB (Configuration Mgmt
DataBase)
Our Discovery: continuous, low-profile

View Slide

OSMC
24 October
2013
O
S
M
C
Second Dimension:
Unique Powerful Features
1. Continuous Discovery
2. Zero network footprint
3. Centralized graph database
4. We know everything that
changes
5. Discover and update dependency
information

View Slide

OSMC
24 October
2013
O
S
M
C
(even more) Features...
6. Discovery and monitoring tightly
integrated
7. Discovery and monitoring easily
extensible
8. Naturally scalable to > 100K systems
9. Server failures distinguishable
from switch failures
10.Minimal network load
11.Multi-tenant support

View Slide

OSMC
24 October
2013
O
S
M
C
This all sounds unreasonable...
●
Huge scalability without complexity?
●
Discovery without sending packets?
Really?

View Slide

OSMC
24 October
2013
O
S
M
C
Third Dimension:
Uniformly, fully distributed work
Two philosophical underpinnings
1. Monitoring and Discovery
are fully distributed
2. Reliable “no news is good news”
Only responses to changes are centralized

View Slide

OSMC
24 October
2013
O
S
M
C
Simple Scalability
●
I can explain how we distribute
work so your grandmother
would understand

View Slide

OSMC
24 October
2013
O
S
M
C
Massive Scalability – or
“I see dead servers in O(1) time”
●
Adding systems does not increase the monitoring work on any
system
●
Each server monitors 2 (or 4) neighbors
●
Each server monitors its own services
●
Ring repair and alerting is O(n) – but a very small amount of work
●
Ring repair for a million nodes is less than 10K packets per day
(approximately 1 packet per 9 seconds)
Current Implementation

View Slide

OSMC
24 October
2013
O
S
M
C
Minimizing Network Footprint
(planned)
●
Support diagnosing switch issues
●
Minimize network traffic
●
Ideal for multi-site arrangements

View Slide

OSMC
24 October
2013
O
S
M
C
Fourth Dimension:
Architectural Components
Three Architectural Compnents
Collective Management
Authority
●
One CMA per installation
Nanoprobes
●
One nanoprobe per system
Data Storage
●
Central Neo4j graph database

View Slide

OSMC
24 October
2013
O
S
M
C
Nanoprobe Functions ('C')
Announce self to CMA
●
Reserved multicast address (can be
unicast address or name if no multicast)
Do what CMA says
●
receive configuration information
– CMA addresses, ports, defaults
●
send/expect heartbeats
●
perform discovery actions
●
perform monitoring actions
No persistent state across reboots

View Slide

OSMC
24 October
2013
O
S
M
C
Service Monitoring based on
Linux-HA/Pacemaker LRM
●
LRM == Local Resource Manager
●
Well-proven architecture:
– “no news is good news” AKA
management by exception
●
Implements Open Cluster Framework
standard (and others)
●
Each system monitors own services
●
Can also start, stop, migrate services

View Slide

OSMC
24 October
2013
O
S
M
C
Basic CMA Functions (python)
Nanoprobe management
●
Configure & direct
●
Hear alerts & discovery
●
Update rings: join/leave
Update database
Issue alerts

View Slide

OSMC
24 October
2013
O
S
M
C
Monitoring Pros and Cons
Pros
Simple & Scalable
Uniform work
distribution
No single point of
failure
Distinguishes switch
vs host failure
Easy on LAN, WAN
Multi-tenant approach
Cons
Active agents
Potential slowness at
power-on

View Slide

OSMC
24 October
2013
O
S
M
C
Why a graph database? (Neo4j)
●
Humans describe systems as graphs
●
Dependency & Discovery information: graph
●
Speed of graph traversals depends on size
of subgraph, not total graph size
●
Root cause queries  graph traversals –
notoriously slow in relational databases
●
Visualization is Natural
●
Schema-less design: good for constantly
changing heterogeneous environment
●
Graph Model === Object Model

View Slide

OSMC
24 October
2013
O
S
M
C
Fifth Dimension:
Discovery API
Scripts perform discovery
– output JSON
Three Discovery Snippets
●
OS information
●
Service discovery
●
Client discovery

View Slide

OSMC
24 October
2013
O
S
M
C
How does discovery work?
Nanoprobe scripts perform discovery
●
Each discovers one kind of information
●
Can take arguments from environment
●
Output JSON
CMA stores Discovery Information
●
JSON stored in Neo4j database
●
CMA discovery plugins => graph nodes and
relationships

View Slide

OSMC
24 October
2013
O
S
M
C
OS discovery JSON Snippet
{ "nodename": "alanr-1225B",
"operating-system": "GNU/Linux",
"machine": "x86_64",
"processor": "x86_64",
"hardware-platform": "x86_64",
"kernel-name": "Linux",
"kernel-release": "3.8.0-31-generic",
"kernel-version": "#46-Ubuntu SMP ...",
"Distributor ID": "Ubuntu",
"Description": "Ubuntu 13.04",
"Release": "13.04",
"Codename": "raring"
}

View Slide

OSMC
24 October
2013
O
S
M
C
sshd Service JSON Snippet
(from netstat and /proc)
"sshd": {
"exe": "/usr/sbin/sshd",
"cmdline": [ "/usr/sbin/sshd", "-D" ],
"uid": "root",
"gid": "root",
"cwd": "/",
"listenaddrs": {
"0.0.0.0:22": {
"proto": "tcp",
"addr": "0.0.0.0",
"port": 22
}, and so on...

View Slide

OSMC
24 October
2013
O
S
M
C
ssh Client JSON Snippet
(from netstat and /proc)
"ssh": {
"exe": "/usr/sbin/ssh",
"cmdline": [ "ssh", "servidor" ],
"uid": "alanr",
"gid": "alanr",
"cwd": "/home/alanr/monitor/src",
"clientaddrs": {
"10.10.10.5:22": {
"proto": "tcp",
"addr": "10.10.10.5",
"port": 22
}, and so on...

View Slide

OSMC
24 October
2013
O
S
M
C
Sixth Dimension:
Graph Schema
Two Schema subgraphs
●
Client / server
dependency
●
Switch interconnect

View Slide

OSMC
24 October
2013
O
S
M
C
ssh -> sshd dependency graph

View Slide

OSMC
24 October
2013
O
S
M
C
Switch Discovery Data
from LLDP (or CDP)
CRM transforms LLDP (CDP) Data to JSON

View Slide

OSMC
24 October
2013
O
S
M
C
Seventh Dimension:
Current Status
●
First release April 2013
●
Great unit tests
●
Nanoprobe code works well
●
Several discovery methods written
●
CMA restructuring finishing up
●
UI development underway
●
Licensed under GPL: commercial
options available

View Slide

OSMC
24 October
2013
O
S
M
C
Eighth Dimension:
Get Involved!
We need every talent!
●
Early adopters
●
Testers, Continuous Integration
●
Designers
●
Developers (C,Python, Shell, PowerShell, JavaScript)
●
Porters (esp Windows)
●
Promoters, publicists
●
Packagers
●
And so on...

View Slide

OSMC
24 October
2013
O
S
M
C
Get Involved!
Powerful Ideas and Infrastucture
Fun, ground-breaking project
Looking for early adopters, testers!!
Needs for every kind of skill
●
Awesome User Interfaces (UI/UX)
●
Evangelism, community building
●
Test Code (simulate 106 servers!)
●
Python, C, script coding
●
Documentation
●
Feedback: Testing, Ideas, Plans
●
Many others!

View Slide

OSMC
24 October
2013
O
S
M
C
Resistance Is Futile!
Mailing List bit.ly/AssimML
#AssimProj @OSSAlanR
Project Web Site
assimproj.org
Blog
techthoughts.typepad.com
assimilationsystems.com

View Slide

OSMC
24 October
2013
O
S
M
C
My Older GeekGirl

View Slide

OSMC
24 October
2013
O
S
M
C
Discovery
Discovering
●
systems you've forgotten
●
what you're not monitoring
●
whatever you'd like
●
without setting off network security alarms

View Slide

OSMC
24 October
2013
O
S
M
C
Monitoring
Monitoring
●
extreme scale
●
topology aware
●
integrated with discovery
●
easy-to-configure

View Slide

OSMC
24 October
2013
O
S
M
C
Why Assimilation Software?
●
Management Perspective
●
DevOps Perspective

View Slide

OSMC
24 October
2013
O
S
M
C
How does this apply to clouds?
●
Fits nicely into a cloud infrastructure
– Should integrate into OpenStack, et al
– Can control VMs
●
Can monitor customer VMs
– Add nanoprobe to base image
– bottom level of rings disappear without
LLDP or CDP

View Slide

OSMC
24 October
2013
O
S
M
C
Future Plans
●
Production grade by end of year
●
Purchased support
●
“Real digital signatures, compression, encryption
●
Other security enhancements
●
Much more discovery
●
GUI
●
Alerting
●
Reporting
●
Add Statistical Monitoring
●
Best Practice Audits
●
Dynamic (aka cloud) specialization
●
Hundreds more ideas
– See: https://trello.com/b/OpaED3AT

View Slide

2013 OSMC Assimilation Presentation

2013 OSMC Assimilation Presentation

Alan Robertson

More Decks by Alan Robertson

Other Decks in Technology

Featured

Transcript