2013 September NCAR presentation

Presentation of the Assimilation Project to NCAR Boulder

Transcript

1. IT Discovery and Monitoring
   Without Limit
   using
   The Assimilation Project
   #AssimProj @OSSAlanR
   http://assimproj.org/
   http://bit.ly/AssimNCAR2013
   Alan Robertson
   Assimilation Systems Limited
   http://assimilationsystems.com
   

   View Slide

2. NCAR
   26 September
   2013
   © 2013 Assimilation Systems Limited 2/38
   Why Am I Here?
   ●
   Understand Your Environment and Needs
   – What you currently do for discovery
   – What you currently do for monitoring
   – How those are working for you
   – Are multi-tenant capabilities appealing?
   ●
   Give an overview of the project
   ●
   Engage with your communities
   ●
   Understand if we ought to stay in touch
   

   View Slide

3. NCAR
   26 September
   2013
   © 2013 Assimilation Systems Limited 3/38
   Upcoming Events
   National Center for Atmospheric Research (today!)
   Denver Open Source User’s Group
   Facebook presentation
   GraphConnect San Francisco
   Open Source Monitoring Conference - Nürnberg
   NSA / Homeland Security Assimilation Technical Talk
   Large Installation System Administration Conference - DC
   Colorado Springs Open Source User’s Group
   linux.conf.au – Linux Conference in Australia - Perth
   Details on http://assimilationsystems.com/
   

   View Slide

4. NCAR
   26 September
   2013
   © 2013 Assimilation Systems Limited 4/38
   Discovery
   Discovering
   ●
   systems you've forgotten
   ●
   what you're not monitoring
   ●
   whatever you'd like
   ●
   without setting off security alarms
   

   View Slide

5. NCAR
   26 September
   2013
   © 2013 Assimilation Systems Limited 5/38
   Monitoring
   Monitoring
   ●
   extreme scale
   ●
   topology aware
   ●
   integrated with discovery
   ●
   easy-to-configure
   

   View Slide

6. NCAR
   26 September
   2013
   © 2013 Assimilation Systems Limited 6/38
   Assimilation Project History
   ●
   Inspired by 2 million core computer (cyclops64)
   ●
   Concerns for extreme scale
   ●
   Topology aware monitoring
   ●
   Topology discovery w/out security issues
   =►Discovery of everything!
   

   View Slide

7. NCAR
   26 September
   2013
   © 2013 Assimilation Systems Limited 7/38
   Project Scope
   Zero-network-footprint continuous Discovery
   integrated with extreme-scale Monitoring
   ●
   Continuous extensible discovery
   – systems, switches, services, dependencies
   – zero network footprint
   ●
   Extensible exception monitoring
   – more than 100K systems
   ●
   All data goes into central graph database
   

   View Slide

8. NCAR
   26 September
   2013
   © 2013 Assimilation Systems Limited 8/38
   Why Assimilation Software?
   ●
   Management Perspective
   ●
   DevOps Perspective
   

   View Slide

9. NCAR
   26 September
   2013
   © 2013 Assimilation Systems Limited 9/38
   Risk Management/Mitigation
   ●
   Intrusions
   ●
   Licensed Software
   ●
   Audit Risk
   ●
   Outages
   ●
   System management
   

   View Slide

10. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 10/38
    Why Discovery? (DevOps)
    ●
    Documentation: incomplete, incorrect
    ●
    Dependencies: unknown
    ●
    Planning: Needs accurate data
    ●
    Best Practices: Verification needs
    data
    ●
    ITIL CMDB (Configuration Mgmt
    DataBase)
    Our Discovery: continuous, low-profile
    

    View Slide

11. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 11/38
    Why Our Monitoring?
    ●
    Simpler to configure (in theory)
    ●
    Growth is non-issue
    ●
    Extremely low network traffic
    ●
    Ideal for cross-WAN monitoring
    ●
    Highlight cascading failure root causes
    ●
    Not confused by switch failures
    ●
    Most switches get monitored “for free”
    

    View Slide

12. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 12/38
    This all sounds unreasonable...
    ●
    Huge scalability without complexity?
    ●
    Discovery without sending packets?
    Really?
    

    View Slide

13. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 13/38
    Architectural Overview
    Collective Management Authority
    ●
    One CMA per installation
    Nanoprobes
    ●
    One nanoprobe per OS image
    Data Storage
    ●
    Central Neo4j graph database
    General Rule: “No News Is Good News”
    

    View Slide

14. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 14/38
    Simple Scalability
    ●
    I can explain how we scale so
    your grandmother would
    understand
    

    View Slide

15. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 15/38
    Massive Scalability – or
    “I see dead servers in O(1) time”
    ●
    Adding systems does not increase the monitoring work on any
    system
    ●
    Each server monitors 2 (or 4) neighbors
    ●
    Each server monitors its own services
    ●
    Ring repair and alerting is O(n) – but a very small amount of work
    ●
    Ring repair for a million nodes is less than 10K packets per day
    (approximately 1 packet per 9 seconds)
    Current Implementation
    

    View Slide

16. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 16/38
    Minimizing Network Footprint
    (planned)
    ●
    Support diagnosing switch issues
    ●
    Minimize network traffic
    ●
    Ideal for multi-site arrangements
    

    View Slide

17. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 17/38
    Service Monitoring
    Based on Linux-HA LRM
    ●
    LRM == Local Resource Manager
    ●
    Well-proven architecture:
    – “no news is good news” AKA
    management by exception
    ●
    Implements Open Cluster Framework
    standard (and others)
    ●
    Each system monitors own services
    ●
    Can also start, stop, migrate services
    

    View Slide

18. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 18/38
    Monitoring Pros and Cons
    Pros
    Simple & Scalable
    Uniform work
    distribution
    No single point of
    failure
    Distinguishes switch
    vs host failure
    Easy on LAN, WAN
    Multi-tenant approach
    Cons
    Active agents
    Potential slowness at
    power-on
    

    View Slide

19. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 19/38
    How does this apply to clouds?
    ●
    Fits nicely into a cloud infrastructure
    – Should integrate into OpenStack, et al
    – Can control VMs
    ●
    Can monitor customer VMs
    – Add nanoprobe to base image
    – bottom level of rings disappear without
    LLDP or CDP
    

    View Slide

20. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 20/38
    Architectural Details
    ●
    Nanoprobes
    ●
    CMA
    ●
    Neo4j
    

    View Slide

21. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 21/38
    Nanoprobe Functions ('C')
    Announce self to CMA
    ●
    Reserved multicast address (can be
    unicast address or name if no multicast)
    Do what CMA says
    ●
    receive configuration information
    – CMA addresses, ports, defaults
    ●
    send/expect heartbeats
    ●
    perform discovery actions
    ●
    perform monitoring actions
    No persistent state across reboots
    

    View Slide

22. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 22/38
    Basic CMA Functions (python)
    Nanoprobe management
    ●
    Configure & direct
    ●
    Hear alerts & discovery
    ●
    Update rings: join/leave
    Update database
    Issue alerts
    

    View Slide

23. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 23/38
    Why a graph database? (Neo4j)
    ●
    Dependency & Discovery information: graph
    ●
    Speed of graph traversals depends on size
    of subgraph, not total graph size
    ●
    Root cause queries  graph traversals –
    notoriously slow in relational databases
    ●
    Visualization of relationships
    ●
    Schema-less design: good for constantly
    changing heterogeneous environment
    ●
    Graph Model === Object Model
    

    View Slide

24. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 24/38
    How does discovery work?
    Nanoprobe scripts perform discovery
    ●
    Each discovers one kind of information
    ●
    Can take arguments (in environment)
    ●
    Output JSON
    CMA stores Discovery Information
    ●
    JSON stored in Neo4j database
    ●
    CMA discovery plugins => graph nodes and
    relationships
    

    View Slide

25. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 25/38
    sshd Service JSON Snippet
    (from netstat and /proc)
    "sshd": {
    "exe": "/usr/sbin/sshd",
    "cmdline": [ "/usr/sbin/sshd", "-D" ],
    "uid": "root",
    "gid": "root",
    "cwd": "/",
    "listenaddrs": {
    "0.0.0.0:22": {
    "proto": "tcp",
    "addr": "0.0.0.0",
    "port": 22
    }, and so on...
    

    View Slide

26. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 26/38
    ssh Client JSON Snippet
    (from netstat and /proc)
    "ssh": {
    "exe": "/usr/sbin/ssh",
    "cmdline": [ "ssh", "servidor" ],
    "uid": "alanr",
    "gid": "alanr",
    "cwd": "/home/alanr/monitor/src",
    "clientaddrs": {
    "10.10.10.5:22": {
    "proto": "tcp",
    "addr": "10.10.10.5",
    "port": 22
    }, and so on...
    

    View Slide

27. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 27/38
    ssh -> sshd dependency graph
    

    View Slide

28. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 28/38
    Switch Discovery Data
    from LLDP (or CDP)
    CRM transforms LLDP (CDP) Data to JSON
    

    View Slide

29. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 29/38
    Current State
    ●
    First release was April 2013
    ●
    Great unit test infrastructure
    ●
    Nanoprobe code – works well
    ●
    Service monitoring works
    ●
    Lacks digital signatures, encryption, compression
    ●
    Reliable UDP comm code working
    ●
    Several discovery methods written
    ●
    CMA and database code restructuring near-complete
    ●
    UI development underway
    ●
    Licensed under the GPL, commercial license available
    

    View Slide

30. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 30/38
    Future Plans
    ●
    Production grade by end of year
    ●
    Purchased support
    ●
    “Real digital signatures, compression, encryption
    ●
    Other security enhancements
    ●
    Much more discovery
    ●
    GUI
    ●
    Alerting
    ●
    Reporting
    ●
    Add Statistical Monitoring
    ●
    Best Practice Audits
    ●
    Dynamic (aka cloud) specialization
    ●
    Hundreds more ideas
    – See: https://trello.com/b/OpaED3AT
    

    View Slide

31. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 31/38
    Get Involved!
    Powerful Ideas and Infrastucture
    Fun, ground-breaking project
    Looking for early adopters, testers!!
    Needs for every kind of skill
    ●
    Awesome User Interfaces (UI/UX)
    ●
    Evangelism, community building
    ●
    Test Code (simulate 106 servers!)
    ●
    Python, C, script coding
    ●
    Documentation
    ●
    Feedback: Testing, Ideas, Plans
    ●
    Many others!
    

    View Slide

32. NCAR
    26 September
    2013
    © 2013 Assimilation Systems Limited 32/38
    Resistance Is Futile!
    #AssimProj @OSSAlanR
    #AssimMon
    Project Web Site
    http://assimproj.org
    Blog
    techthoughts.typepad.com
    lists.community.tummy.com/cgi-bin/mailman/admin/assimilation
    

    View Slide