GraphConnect 2013 - Modeling IT infrastructure using the Assimilation Project

A talk on using Neo4j to model IT infrastructure given at the 2013 GraphConnect conference in San Francisco.

Transcript

1. G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Modeling IT Infrastucture
   using
   The Assimilation Project
   #AssimProj @OSSAlanR
   http://assimproj.org/
   http://bit.ly/AssimGC2013
   Alan Robertson
   Assimilation Systems Limited
   http://assimilationsystems.com
   

   View Slide

2. GraphConnect
   4 October
   2013
   © 2013 Assimilation Systems Limited 2/25
   G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Upcoming Events
   GraphConnect San Francisco (today!)
   Open Source Monitoring Conference - Nürnberg
   NSA / Homeland Security Assimilation Technical Talk
   Large Installation System Administration Conference - DC
   Colorado Springs Open Source User’s Group
   linux.conf.au – Awesome Australian Linux Conf - Perth
   Details on http://assimilationsystems.com/
   

   View Slide

3. GraphConnect
   4 October
   2013
   © 2013 Assimilation Systems Limited 3/25
   G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Talk Outline
   ●
   Project Overview
   ●
   Infrastructure Schema
   ●
   Python Object-Graph-Mapping API
   

   View Slide

4. GraphConnect
   4 October
   2013
   © 2013 Assimilation Systems Limited 4/25
   G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Assimilation Project History
   ●
   Inspired by 2 million core computer (cyclops64)
   ●
   Concerns for extreme scale
   ●
   Topology aware monitoring
   ●
   Topology discovery w/out security issues
   =►Discovery of everything!
   

   View Slide

5. GraphConnect
   4 October
   2013
   © 2013 Assimilation Systems Limited 5/25
   G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Project Scope
   Zero-network-footprint continuous Discovery
   integrated with extreme-scale Monitoring
   ●
   Continuous extensible discovery
   – systems, switches, services, dependencies
   – zero network footprint
   ●
   Extensible exception monitoring
   – more than 100K systems
   ●
   All data goes into central graph database
   

   View Slide

6. GraphConnect
   4 October
   2013
   © 2013 Assimilation Systems Limited 6/25
   G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Discovery
   Discovering
   ●
   systems you've forgotten
   ●
   what you're not monitoring
   ●
   whatever you'd like
   ●
   without setting off security alarms
   

   View Slide

7. GraphConnect
   4 October
   2013
   © 2013 Assimilation Systems Limited 7/25
   G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Why Discovery?
   ●
   Documentation: incomplete, incorrect
   ●
   Dependencies: unknown
   ●
   Planning: Needs accurate data
   ●
   Best Practices: Verification needs
   data
   ●
   ITIL CMDB (Configuration Mgmt
   DataBase)
   Our Discovery: continuous, low-profile
   

   View Slide

8. GraphConnect
   4 October
   2013
   © 2013 Assimilation Systems Limited 8/25
   G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Why Neo4j (graph db)?
   ●
   Dependency & Discovery information: graph
   ●
   Speed of graph traversals depends on size
   of subgraph, not total graph size
   ●
   Root cause queries  graph traversals –
   notoriously slow in relational databases
   ●
   Visualization of relationships
   ●
   Schema-less design: good for constantly
   changing heterogeneous environment
   ●
   Graph Model === Object Model
   

   View Slide

9. GraphConnect
   4 October
   2013
   © 2013 Assimilation Systems Limited 9/25
   G
   R
   A
   P
   H
   C
   O
   N
   N
   E
   C
   T
   Assimilation Communication
   Neighbor-Rings
   

   View Slide

10. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 10/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Ring Representation Schema
    

    View Slide

11. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 11/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    ssh -> sshd dependency graph
    

    View Slide

12. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 12/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Switch Discovery Data
    from LLDP (or CDP)
    CRM transforms LLDP (CDP) Data to JSON
    

    View Slide

13. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 13/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM – Object Graph Mapping
    ●
    Managing the Graph Nodes “disappears”
    ●
    The Object Model is the Graph Model
    ●
    Significant Improvement in Thinking
    Clarity
    – The “mud” is gone
    

    View Slide

14. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 14/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM rules
    ●
    Don't use Constructors
    ●
    Relationships replace hash tables and
    object references and so on
    ●
    Constructor parameter names match
    attribute names
    

    View Slide

15. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 15/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    @RegisterGraphClass
    class Person(GraphNode):
    'A Person - or at least an electronic representation of one'
    def __init__(self, firstname, lastname, dateofbirth=None):
    GraphNode.__init__(self, domain='global')
    self.firstname = firstname
    self.lastname = lastname
    if dateofbirth is not None:
    self.dateofbirth = dateofbirth
    else:
    self.dateofbirth='unknown'
    @staticmethod
    def __meta_keyattrs__():
    'Return our key attributes in order of significance'
    return ['lastname', 'firstname']
    

    View Slide

16. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 16/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    @RegisterGraphClass
    class TestSystem(GraphNode):
    'Some kind of semi-intelligent system'
    def __init__(self, designation, roles=[]):
    GraphNode.__init__(self, domain)
    self.designation = designation.lower()
    if roles == []:
    roles = ['']
    self.roles = roles
    def addroles(self, role):
    if self.roles[0] == '''':
    del self.roles[0]
    if isinstance(role, list):
    for arole in role:
    self.addroles(arole)
    elif role not in self.roles:
    self.roles.append(role)
    

    View Slide

17. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 17/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    @staticmethod
    def __meta_keyattrs__():
    'Return our key attributes in order of significance'
    return ['designation']
    

    View Slide

18. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 18/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    # (seven)-[:formerly]->(Annika)
    Annika = store.load_or_create(Person,
    firstname='Annika',
    lastname='Hansen')
    seven = store.load_or_create(Drone,
    designation='SevenOfNine', roles='Borg')
    store.relate(seven, 'formerly', Annika)
    store.commit()
    

    View Slide

19. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 19/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    @RegisterGraphClass
    class TestDrone(TestSystem):
    def __init__(self, designation, roles=[]):
    TestSystem.__init__(self, designation=designation)
    if isinstance(roles, str):
    roles = [roles]
    roles.extend(['host', 'Drone'])
    System.__init__(self, designation, roles=roles)
    

    View Slide

20. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 20/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Current State
    ●
    First release was April 2013
    ●
    Great unit test infrastructure
    ●
    Nanoprobe code – works well
    ●
    Service monitoring works
    ●
    Lacks digital signatures, encryption, compression
    ●
    Reliable UDP comm code working
    ●
    Several discovery methods written
    ●
    CMA and database code restructuring near-complete
    ●
    UI development underway
    ●
    Licensed under the GPL, commercial license available
    

    View Slide

21. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 21/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Future Plans
    ●
    Production grade by end of year
    ●
    Purchased support
    ●
    “Real digital signatures, compression, encryption
    ●
    Other security enhancements
    ●
    Much more discovery
    ●
    GUI
    ●
    Alerting
    ●
    Reporting
    ●
    Add Statistical Monitoring
    ●
    Best Practice Audits
    ●
    Dynamic (aka cloud) specialization
    ●
    Hundreds more ideas
    – See: https://trello.com/b/OpaED3AT
    

    View Slide

22. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 22/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Get Involved!
    Powerful Ideas and Infrastucture
    Fun, ground-breaking project
    Looking for early adopters, testers!!
    Needs for every kind of skill
    ●
    Awesome User Interfaces (UI/UX)
    ●
    Evangelism, community building
    ●
    Test Code (simulate 106 servers!)
    ●
    Python, C, script coding
    ●
    Documentation
    ●
    Feedback: Testing, Ideas, Plans
    ●
    Many others!
    

    View Slide

23. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 23/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Resistance Is Futile!
    #AssimProj @OSSAlanR
    #AssimMon
    Project Web Site
    http://assimproj.org
    Blog
    techthoughts.typepad.com
    lists.community.tummy.com/cgi-bin/mailman/admin/assimilation
    

    View Slide

24. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 24/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    The Elder GeekGirl
    

    View Slide

25. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 25/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Younger GeekGirl's Computer
    Running Linux
    Of Course!
    

    View Slide