Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GraphConnect 2013 - Modeling IT infrastructure using the Assimilation Project

GraphConnect 2013 - Modeling IT infrastructure using the Assimilation Project

A talk on using Neo4j to model IT infrastructure given at the 2013 GraphConnect conference in San Francisco.

Alan Robertson

October 04, 2013
Tweet

More Decks by Alan Robertson

Other Decks in Technology

Transcript

  1. G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Modeling IT Infrastucture
    using
    The Assimilation Project
    #AssimProj @OSSAlanR
    http://assimproj.org/
    http://bit.ly/AssimGC2013
    Alan Robertson
    Assimilation Systems Limited
    http://assimilationsystems.com

    View Slide

  2. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 2/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Upcoming Events
    GraphConnect San Francisco (today!)
    Open Source Monitoring Conference - Nürnberg
    NSA / Homeland Security Assimilation Technical Talk
    Large Installation System Administration Conference - DC
    Colorado Springs Open Source User’s Group
    linux.conf.au – Awesome Australian Linux Conf - Perth
    Details on http://assimilationsystems.com/

    View Slide

  3. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 3/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Talk Outline

    Project Overview

    Infrastructure Schema

    Python Object-Graph-Mapping API

    View Slide

  4. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 4/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Assimilation Project History

    Inspired by 2 million core computer (cyclops64)

    Concerns for extreme scale

    Topology aware monitoring

    Topology discovery w/out security issues
    =►Discovery of everything!

    View Slide

  5. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 5/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Project Scope
    Zero-network-footprint continuous Discovery
    integrated with extreme-scale Monitoring

    Continuous extensible discovery
    – systems, switches, services, dependencies
    – zero network footprint

    Extensible exception monitoring
    – more than 100K systems

    All data goes into central graph database

    View Slide

  6. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 6/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Discovery
    Discovering

    systems you've forgotten

    what you're not monitoring

    whatever you'd like

    without setting off security alarms

    View Slide

  7. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 7/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Why Discovery?

    Documentation: incomplete, incorrect

    Dependencies: unknown

    Planning: Needs accurate data

    Best Practices: Verification needs
    data

    ITIL CMDB (Configuration Mgmt
    DataBase)
    Our Discovery: continuous, low-profile

    View Slide

  8. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 8/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Why Neo4j (graph db)?

    Dependency & Discovery information: graph

    Speed of graph traversals depends on size
    of subgraph, not total graph size

    Root cause queries  graph traversals –
    notoriously slow in relational databases

    Visualization of relationships

    Schema-less design: good for constantly
    changing heterogeneous environment

    Graph Model === Object Model

    View Slide

  9. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 9/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Assimilation Communication
    Neighbor-Rings

    View Slide

  10. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 10/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Ring Representation Schema

    View Slide

  11. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 11/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    ssh -> sshd dependency graph

    View Slide

  12. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 12/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Switch Discovery Data
    from LLDP (or CDP)
    CRM transforms LLDP (CDP) Data to JSON

    View Slide

  13. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 13/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM – Object Graph Mapping

    Managing the Graph Nodes “disappears”

    The Object Model is the Graph Model

    Significant Improvement in Thinking
    Clarity
    – The “mud” is gone

    View Slide

  14. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 14/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM rules

    Don't use Constructors

    Relationships replace hash tables and
    object references and so on

    Constructor parameter names match
    attribute names

    View Slide

  15. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 15/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    @RegisterGraphClass
    class Person(GraphNode):
    'A Person - or at least an electronic representation of one'
    def __init__(self, firstname, lastname, dateofbirth=None):
    GraphNode.__init__(self, domain='global')
    self.firstname = firstname
    self.lastname = lastname
    if dateofbirth is not None:
    self.dateofbirth = dateofbirth
    else:
    self.dateofbirth='unknown'
    @staticmethod
    def __meta_keyattrs__():
    'Return our key attributes in order of significance'
    return ['lastname', 'firstname']

    View Slide

  16. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 16/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    @RegisterGraphClass
    class TestSystem(GraphNode):
    'Some kind of semi-intelligent system'
    def __init__(self, designation, roles=[]):
    GraphNode.__init__(self, domain)
    self.designation = designation.lower()
    if roles == []:
    roles = ['']
    self.roles = roles
    def addroles(self, role):
    if self.roles[0] == '''':
    del self.roles[0]
    if isinstance(role, list):
    for arole in role:
    self.addroles(arole)
    elif role not in self.roles:
    self.roles.append(role)

    View Slide

  17. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 17/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    @staticmethod
    def __meta_keyattrs__():
    'Return our key attributes in order of significance'
    return ['designation']

    View Slide

  18. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 18/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    # (seven)-[:formerly]->(Annika)
    Annika = store.load_or_create(Person,
    firstname='Annika',
    lastname='Hansen')
    seven = store.load_or_create(Drone,
    designation='SevenOfNine', roles='Borg')
    store.relate(seven, 'formerly', Annika)
    store.commit()

    View Slide

  19. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 19/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    OGM sample
    @RegisterGraphClass
    class TestDrone(TestSystem):
    def __init__(self, designation, roles=[]):
    TestSystem.__init__(self, designation=designation)
    if isinstance(roles, str):
    roles = [roles]
    roles.extend(['host', 'Drone'])
    System.__init__(self, designation, roles=roles)

    View Slide

  20. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 20/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Current State

    First release was April 2013

    Great unit test infrastructure

    Nanoprobe code – works well

    Service monitoring works

    Lacks digital signatures, encryption, compression

    Reliable UDP comm code working

    Several discovery methods written

    CMA and database code restructuring near-complete

    UI development underway

    Licensed under the GPL, commercial license available

    View Slide

  21. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 21/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Future Plans

    Production grade by end of year

    Purchased support

    “Real digital signatures, compression, encryption

    Other security enhancements

    Much more discovery

    GUI

    Alerting

    Reporting

    Add Statistical Monitoring

    Best Practice Audits

    Dynamic (aka cloud) specialization

    Hundreds more ideas
    – See: https://trello.com/b/OpaED3AT

    View Slide

  22. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 22/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Get Involved!
    Powerful Ideas and Infrastucture
    Fun, ground-breaking project
    Looking for early adopters, testers!!
    Needs for every kind of skill

    Awesome User Interfaces (UI/UX)

    Evangelism, community building

    Test Code (simulate 106 servers!)

    Python, C, script coding

    Documentation

    Feedback: Testing, Ideas, Plans

    Many others!

    View Slide

  23. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 23/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Resistance Is Futile!
    #AssimProj @OSSAlanR
    #AssimMon
    Project Web Site
    http://assimproj.org
    Blog
    techthoughts.typepad.com
    lists.community.tummy.com/cgi-bin/mailman/admin/assimilation

    View Slide

  24. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 24/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    The Elder GeekGirl

    View Slide

  25. GraphConnect
    4 October
    2013
    © 2013 Assimilation Systems Limited 25/25
    G
    R
    A
    P
    H
    C
    O
    N
    N
    E
    C
    T
    Younger GeekGirl's Computer
    Running Linux
    Of Course!

    View Slide