2013 September Assimilation presentation to TWC

We provide open source discovery with zero network footprint integrated with highly-scalable monitoring - http://assimproj.org

Transcript

1. IT Discovery and Monitoring
   Without Limit
   using
   The Assimilation Project
   #AssimProj @OSSAlanR
   http://assimproj.org/
   http://bit.ly/AssimTWC2013
   Alan Robertson
   Assimilation Systems Limited
   http://assimilationsystems.com
   

   View Slide

2. Upcoming Events
   Time-Warner Cable Developer Conference
   National Center for Atmospheric Research
   Denver Open Source User’s Group
   GraphConnect San Francisco
   Open Source Monitoring Conference - Nürnberg
   NSA / Homeland Security Assimilation Technical Talk
   Large Installation System Administration Conference - DC
   Colorado Springs Open Source User’s Group
   linux.conf.au – Linux Conference in Australia - Perth
   

   View Slide

3. 12 September
   2013
   © 2013 Assimilation Systems Limited
   3/37
   Discovery
   Discovering
   ●
   systems you've forgotten
   ●
   what you're not monitoring
   ●
   whatever you'd like
   ●
   without setting off security alarms
   

   View Slide

4. 12 September
   2013
   © 2013 Assimilation Systems Limited
   4/37
   Monitoring
   Monitoring
   ●
   extreme scale
   ●
   topology aware
   ●
   integrated with discovery
   ●
   easy-to-configure
   

   View Slide

5. 12 September
   2013
   © 2013 Assimilation Systems Limited
   5/37
   Assimilation Project History
   ●
   Inspired by 2 million core computer
   ●
   Concerns for extreme scale
   ●
   Topology aware monitoring
   ●
   Topology discovery w/out security issues
   ●
   Discovery of everything!
   

   View Slide

6. 12 September
   2013
   © 2013 Assimilation Systems Limited
   6/37
   Project Scope
   Zero-network-footprint continuous Discovery
   integrated with extreme-scale Monitoring
   ●
   Continuous extensible discovery
   – systems, switches, services, dependencies
   – zero network footprint
   ●
   Extensible exception monitoring
   – more than 100K systems
   ●
   All data goes into central graph database
   

   View Slide

7. 12 September
   2013
   © 2013 Assimilation Systems Limited
   7/37
   Why Assimilation Software?
   ●
   Management Perspective
   ●
   DevOps Perspective
   

   View Slide

8. 12 September
   2013
   © 2013 Assimilation Systems Limited
   8/37
   Risk Management/Mitigation
   ●
   Intrusions
   ●
   Licensed Software
   ●
   Audit Risk
   ●
   Outages
   ●
   System management
   

   View Slide

9. 12 September
   2013
   © 2013 Assimilation Systems Limited
   9/37
   Why Discovery? (DevOps)
   ●
   Documentation: incomplete, incorrect
   ●
   Dependencies: unknown
   ●
   Planning: Needs accurate data
   ●
   Best Practices: Verification needs
   data
   ●
   ITIL CMDB (Configuration Mgmt
   DataBase)
   

   View Slide

10. 12 September
    2013
    © 2013 Assimilation Systems Limited
    10/37
    Why Our Monitoring?
    ●
    Simpler to configure (in theory)
    ●
    Growth is non-issue
    ●
    Extremely low network traffic
    ●
    Ideal for cross-WAN monitoring
    ●
    Highlight cascading failure root causes
    ●
    Not confused by switch failures
    ●
    Most switches get monitored “for free”
    

    View Slide

11. 12 September
    2013
    © 2013 Assimilation Systems Limited
    11/37
    This all sounds unreasonable...
    ●
    Huge scalability without complexity?
    ●
    Discovery without sending packets?
    Really?
    

    View Slide

12. 12 September
    2013
    © 2013 Assimilation Systems Limited
    12/37
    Architectural Overview
    Collective Management Authority
    ●
    One CMA per installation
    Nanoprobes
    ●
    One nanoprobe per OS image
    Data Storage
    ●
    Central Neo4j graph database
    General Rule: “No News Is Good News”
    

    View Slide

13. 12 September
    2013
    © 2013 Assimilation Systems Limited
    13/37
    Simple Scalability
    ●
    I can explain how we scale so
    your grandmother would
    understand
    

    View Slide

14. 12 September
    2013
    © 2013 Assimilation Systems Limited
    14/37
    Massive Scalability – or
    “I see dead servers in O(1) time”
    ●
    Adding systems does not increase the monitoring work on any
    system
    ●
    Each server monitors 2 (or 4) neighbors
    ●
    Each server monitors its own services
    ●
    Ring repair and alerting is O(n) – but a very small amount of work
    ●
    Ring repair for a million nodes is less than 10K packets per day
    (approximately 1 packet per 9 seconds)
    Current Implementation
    

    View Slide

15. 12 September
    2013
    © 2013 Assimilation Systems Limited
    15/37
    Decreasing Network Footprint
    (planned)
    ●
    Support diagnosing switch issues
    ●
    Minimize network traffic
    ●
    Ideal for multi-site arrangements
    

    View Slide

16. 12 September
    2013
    © 2013 Assimilation Systems Limited
    16/37
    Service Monitoring
    Based on Linux-HA LRM
    ●
    LRM == Local Resource Manager
    ●
    Well-proven architecture:
    – “no news is good news” AKA
    management by exception
    ●
    Implements Open Cluster Framework
    standard (and others)
    ●
    Each system monitors own services
    ●
    Can also start, stop, migrate services
    

    View Slide

17. 12 September
    2013
    © 2013 Assimilation Systems Limited
    17/37
    Monitoring Pros and Cons
    Pros
    Simple & Scalable
    Uniform work
    distribution
    No single point of
    failure
    Distinguishes switch
    vs host failure
    Easy on LAN, WAN
    Cons
    Active agents
    Potential slowness at
    power-on
    

    View Slide

18. 12 September
    2013
    © 2013 Assimilation Systems Limited
    18/37
    How does this apply to clouds?
    ●
    Fits nicely into a cloud infrastructure
    – Should integrate into OpenStack, et al
    – Can control VMs
    ●
    Can monitor customer VMs
    – Add nanoprobe to base image
    – bottom level of rings disappear without
    LLDP or CDP
    

    View Slide

19. 12 September
    2013
    © 2013 Assimilation Systems Limited
    19/37
    Architectural Details
    ●
    Nanoprobes
    ●
    CMA
    ●
    Neo4j
    

    View Slide

20. 12 September
    2013
    © 2013 Assimilation Systems Limited
    20/37
    Nanoprobe Functions ('C')
    Announce self to CMA
    ●
    Reserved multicast address (can be
    unicast address or name if no multicast)
    Do what CMA says
    ●
    receive configuration information
    – CMA addresses, ports, defaults
    ●
    send/expect heartbeats
    ●
    perform discovery actions
    ●
    perform monitoring actions
    No persistent state across reboots
    

    View Slide

21. 12 September
    2013
    © 2013 Assimilation Systems Limited
    21/37
    Basic CMA Functions (python)
    Nanoprobe management
    ●
    Configure & direct
    ●
    Hear alerts & discovery
    ●
    Update rings: join/leave
    Update database
    Issue alerts
    

    View Slide

22. 12 September
    2013
    © 2013 Assimilation Systems Limited
    23/37
    Why a graph database? (Neo4j)
    ●
    Dependency & Discovery information: graph
    ●
    Speed of graph traversals depends on size
    of subgraph, not total graph size
    ●
    Root cause queries  graph traversals –
    notoriously slow in relational databases
    ●
    Visualization of relationships
    ●
    Schema-less design: good for constantly
    changing heterogeneous environment
    

    View Slide

23. 12 September
    2013
    © 2013 Assimilation Systems Limited
    24/37
    How does discovery work?
    Nanoprobe scripts perform discovery
    ●
    Each discovers one kind of information
    ●
    Can take arguments (in environment)
    ●
    Output JSON
    CMA stores Discovery Information
    ●
    JSON stored in Neo4j database
    ●
    CMA discovery plugins => graph nodes and
    relationships
    

    View Slide

24. 12 September
    2013
    © 2013 Assimilation Systems Limited
    25/37
    sshd Service JSON Snippet
    (from netstat and /proc)
    "sshd": {
    "exe": "/usr/sbin/sshd",
    "cmdline": [ "/usr/sbin/sshd", "-D" ],
    "uid": "root",
    "gid": "root",
    "cwd": "/",
    "listenaddrs": {
    "0.0.0.0:22": {
    "proto": "tcp",
    "addr": "0.0.0.0",
    "port": 22
    }, and so on...
    

    View Slide

25. 12 September
    2013
    © 2013 Assimilation Systems Limited
    26/37
    ssh Client JSON Snippet
    (from netstat and /proc)
    "ssh": {
    "exe": "/usr/sbin/ssh",
    "cmdline": [ "ssh", "servidor" ],
    "uid": "alanr",
    "gid": "alanr",
    "cwd": "/home/alanr/monitor/src",
    "clientaddrs": {
    "10.10.10.5:22": {
    "proto": "tcp",
    "addr": "10.10.10.5",
    "port": 22
    }, and so on...
    

    View Slide

26. 12 September
    2013
    © 2013 Assimilation Systems Limited
    27/37
    ssh -> sshd dependency graph
    

    View Slide

27. 12 September
    2013
    © 2013 Assimilation Systems Limited
    28/37
    Switch Discovery Data
    from LLDP (or CDP)
    CRM transforms LLDP (CDP) Data to JSON
    

    View Slide

28. 12 September
    2013
    © 2013 Assimilation Systems Limited
    29/37
    Current State
    ●
    First release was April 2013
    ●
    Great unit test infrastructure
    ●
    Nanoprobe code – works well
    ●
    Service monitoring works
    ●
    Lacking real digital signatures, encryption,
    compression
    ●
    Reliable UDP comm code all working
    ●
    CMA code works, much more to go
    ●
    Several discovery methods written
    ●
    Licensed under the GPL
    

    View Slide

29. 12 September
    2013
    © 2013 Assimilation Systems Limited
    30/37
    Future Plans
    ●
    Production grade by end of year
    ●
    Support, commercial licenses
    ●
    “Real digital signatures, compression, encryption
    ●
    Other security enhancements
    ●
    Much more discovery
    ●
    GUI
    ●
    Alerting
    ●
    Reporting
    ●
    Add Statistical Monitoring
    ●
    Best Practice Audits
    ●
    Dynamic (aka cloud) specialization
    ●
    Hundreds more ideas
    – See: https://trello.com/b/OpaED3AT
    

    View Slide

30. 12 September
    2013
    © 2013 Assimilation Systems Limited
    31/37
    Get Involved!
    Powerful Ideas and Infrastucture
    Fun, ground-breaking project
    Looking for early adopters, testers!!
    Needs for every kind of skill
    ●
    Awesome User Interfaces (UI/UX)
    ●
    Evangelism, community building
    ●
    Test Code (simulate 106 servers!)
    ●
    Python, C, script coding
    ●
    Documentation
    ●
    Feedback: Testing, Ideas, Plans
    ●
    Many others!
    

    View Slide

31. 12 September
    2013
    © 2013 Assimilation Systems Limited
    32/37
    Resistance Is Futile!
    #AssimProj @OSSAlanR
    #AssimMon
    Project Web Site
    http://assimproj.org
    Blog
    techthoughts.typepad.com
    lists.community.tummy.com/cgi-bin/mailman/admin/assimilation
    

    View Slide