Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Salt Lake City DevOpsDays

Salt Lake City DevOpsDays

A talk about security and DevOps and the Assimilation project.

Alan Robertson

June 14, 2016
Tweet

More Decks by Alan Robertson

Other Decks in Technology

Transcript

  1. Security Automation for DevOps
    Security Automation for DevOps
    #AssimProj @OSSAlanR
    Alan Robertson
    Assimilation Systems Limited
    http://AssimilationSystems.com

    View Slide

  2. 2/35
    Biography
    Biography

    35+ years in IT/development – 10 years in system
    management (SysAdmin)

    Founded Linux-HA project - led 1998-2007 – aka “Heartbeat” -
    now called Pacemaker

    Founded Assimilation Project in 2010

    Founded Assimilation Systems Limited in 2013

    Alumnus of Bell Labs, SuSE, IBM

    View Slide

  3. © 2015 Assimilation Systems Limited
    4/35
    Security questions
    Security questions

    Do you think good security staff is easily available?

    Do you think security is going to get better soon?

    Do you think you have enough staff for security to keep up
    with changes at DevOps / Agile rates?

    View Slide

  4. © 2015 Assimilation Systems Limited
    5/35
    Disturbing Trends...
    Disturbing Trends...
    30% of break-ins come through “lost” systems (Verizon)
    90% have had failures of unmonitored services (Turnbull)
    71% are unable to stay in compliance (Verizon)
    30% only start monitoring only after a problem (Turnbull)
    30% of systems doing nothing useful (Koomey)

    View Slide

  5. Highly Scalable Discovery-Driven
    Highly Scalable Discovery-Driven
    Automation
    Automation
    Continuous Discovery drives everything

    Continuous extensible discovery (CMDB)
    – systems, switches, services, dependencies – zero network footprint
    discovery process

    Extensible exception monitoring
    – more than 100K systems

    Discovery Drives Best Practice Analyses
    – Initially concentrating on security

    All data goes into central graph CMDB (Config Mgmt Data Base)

    View Slide

  6. © 2015 Assimilation Systems Limited
    7/35
    This all sounds unreasonable...
    This all sounds unreasonable...

    Huge scalability without complexity?

    Discovery without pings or port scans?
    Really?

    View Slide

  7. © 2015 Assimilation Systems Limited
    8/35
    Simple Scalability
    Simple Scalability
    I can explain how we scale so your
    grandmother would understand...

    View Slide

  8. Massive Scalability –
    Massive Scalability – or
    or
    “I see dead servers in
    “I see dead servers in O
    O(1) time”
    (1) time”

    Adding systems does not increase the monitoring work on any system

    Each server monitors 2 (or 4) neighbors

    Each server monitors and discovers its own services

    Ring repair and alerting is O(n)
    – but a very small amount of work
    Current Implementation

    View Slide

  9. © 2015 Assimilation Systems Limited
    10/35
    Minimizing Network Footprint
    Minimizing Network Footprint
    (in roadmap)
    (in roadmap)

    Support diagnosing switch issues

    Minimize network traffic

    Ideal for multi-site arrangements

    View Slide

  10. © 2015 Assimilation Systems Limited
    11/35
    Assimilation Architecture
    Assimilation Architecture

    Central Collective Management Authority
    – written in Python
    – delegates most work to nanoprobes – does nothing as much as possible
    – Doing nothing scaless really well – should be into the 100K system range

    Fully distributed “nanoprobe” agents
    – Simple, policy-free
    – Written in 'C'
    – Run scripts for monitoring or discovery
    – Send/receive heartbeats
    – Listen for ARP, LLDP, CDP packets

    Neo4j graph database

    View Slide

  11. © 2015 Assimilation Systems Limited
    12/35
    Assimilation Security Discovery
    Assimilation Security Discovery

    All IP+MAC Addresses on subnet

    Network Connectivity (CDP/LLDP)

    Packages and versions installed

    Services Offered / Used (netstat)

    Checksums of network-facing binaries, libraries, JARs

    Many other detailed security settings, permissions
    – (sshd_config, PAM, /proc/sys, auditd_conf, sudoers, etc)

    View Slide

  12. © 2015 Assimilation Systems Limited
    13/35
    Assimilation Analysis / Reports
    Assimilation Analysis / Reports

    Comparison to best practices

    Risk scoring

    Unknown IP Addresses

    Automatic service monitoring

    Unmonitored services

    Triage-related risk scores

    View Slide

  13. © 2015 Assimilation Systems Limited
    14/35
    B
    Best Practice Analyses
    est Practice Analyses

    Triggered by Discovery Updates
    – Analysis occurs within seconds of change
    – No change => No analysis

    We can analyze anything discovered

    You can easily discover anything you want

    Alerts and Reports available

    View Slide

  14. © 2015 Assimilation Systems Limited
    15/35
    Demo
    Demo
    Everything will be discovered – nothing will be configured manually

    What needs hardening

    How to Triage your hardening issues

    How to Demonstrate and Track Progress

    How to keep them in compliance (hardened)

    Visualizing Your Attack Surface

    Who has what package+version
    – Docker package discovery too!

    View Slide

  15. © 2015 Assimilation Systems Limited
    16/35
    Where to find this Information online?
    Where to find this Information online?
    http://assimilationsystems.com/category/getting-started/
    1. 15 Minutes To Better Security
    2. An Hour To Better Security
    3. A Half-Day To Better Security
    Where to See Similar Demos

    http://assimilationsystems.com/category/videos/

    http://assimilationsystems.com/sample-demo-output/

    View Slide

  16. © 2015 Assimilation Systems Limited
    17/35
    Future Plans
    Future Plans

    Checksum whitelist/blacklist ==> risk scores

    Checksum queries

    Create service consolidating vendor patches
    ==> risk scores

    Interactive User Interface(s) (GUI)

    View Slide

  17. © 2015 Assimilation Systems Limited
    18/35
    Get Involved!
    Get Involved!

    Get Assimilated!

    Contribute!
    – Testers, System Management, Continuous Integration
    – Best practice experts
    – Designers
    – Developers (C,Python, Shell, PowerShell, JavaScript)
    – Porters (esp Windows)
    – Promoters, Publicists, Packagers, etc.

    View Slide

  18. © 2015 Assimilation Systems Limited
    19/35
    Resistance Is Futile!
    Resistance Is Futile!
    These slides: bit.ly/DevOpsDaysRox16
    Mailing List: bit.ly/AssimML
    @OSSAlanR
    #assimilation on irc.freenode.net
    Project Web Site: assimproj.org
    Company Web Site: assimilationsystems.com
    Download: assimilationsystems.com/download

    View Slide

  19. © 2015 Assimilation Systems Limited
    20/35
    Switch Discovery Graph from LLDP / CDP
    Switch Discovery Graph from LLDP / CDP

    View Slide

  20. © 2015 Assimilation Systems Limited
    21/35
    Security Demo / HOWTO
    Security Demo / HOWTO

    No configuration was supplied
    – everything comes from discovery

    View Slide

  21. © 2015 Assimilation Systems Limited
    22/35
    Risk Management/Mitigation
    Risk Management/Mitigation

    Intrusions

    Vulnerable Software

    Licensed Software

    Audit Risk

    Outages

    System management

    View Slide

  22. © 2015 Assimilation Systems Limited
    23/35
    Why a graph database? (Neo4j)
    Why a graph database? (Neo4j)

    Humans describe systems as graphs

    Dependency & Discovery information: graph

    Speed of graph traversals depends on size of subgraph,
    not total graph size

    Root cause queries  graph traversals – notoriously slow in
    relational databases

    Visualization is Natural

    Schema-less design: good for constantly changing
    heterogeneous environment

    Graph Model === Object Model

    View Slide

  23. © 2015 Assimilation Systems Limited
    24/35
    Monitoring Pros and Cons
    Monitoring Pros and Cons
    Pros
    Simple & Scalable
    Uniform work distribution
    No single point of failure
    Distinguishes switch vs
    host failure
    Easy on LAN, WAN
    Multi-tenant approach
    Cons
    Active agents
    Potential slowness
    at power-on

    View Slide

  24. © 2015 Assimilation Systems Limited
    25/35
    Sixth Dimension: Graph Schema
    Sixth Dimension: Graph Schema
    Two Schema subgraphs

    Client / server
    dependency

    Switch interconnect

    View Slide

  25. © 2015 Assimilation Systems Limited
    26/35
    "sshd": {
    "exe": "/usr/sbin/sshd",
    "cmdline": [ "/usr/sbin/sshd", "-D" ],
    "uid": "root",
    "gid": "root",
    "cwd": "/",
    "listenaddrs": {
    "0.0.0.0:22": {
    "proto": "tcp",
    "addr": "0.0.0.0",
    "port": 22 },
    sshd
    sshd Service
    Service JSON Snippet (from netstat and /proc)
    JSON Snippet (from netstat and /proc)

    View Slide

  26. © 2015 Assimilation Systems Limited
    27/35
    "ssh": {
    "exe": "/usr/sbin/ssh",
    "cmdline": [ "ssh", "servidor" ],
    "uid": "alanr",
    "gid": "alanr",
    "cwd": "/home/alanr/monitor/src",
    "clientaddrs": {
    "10.10.10.5:22": {
    "proto": "tcp",
    "addr": "10.10.10.5",
    "port": 22 },
    ssh
    ssh Client
    Client JSON Snippet(from netstat and /proc)
    JSON Snippet(from netstat and /proc)

    View Slide

  27. © 2015 Assimilation Systems Limited
    30/35
    First Dimension
    First Dimension:
    :
    Problems Addressed
    Problems Addressed

    Discovering and maintaining documentation (CMDB) using
    continuous discovery
    – Services, Systems, Dependencies, Switches, Interconnects, Configuration

    Monitoring and alerting: services, systems and compliance

    Managing compliance

    Mitigating risk

    View Slide

  28. © 2015 Assimilation Systems Limited
    31/35
    Why Discovery? (DevOps)
    Why Discovery? (DevOps)

    Documentation: incomplete, incorrect

    Dependencies: unknown

    Planning: Needs accurate data

    Best Practices: Verification needs data

    ITIL CMDB (Configuration Management
    Data Base)
    Our Discovery: continuous, low-profile

    View Slide

  29. © 2015 Assimilation Systems Limited
    32/35
    Second Dimension:
    Second Dimension:
    Unique Powerful Features
    Unique Powerful Features
    1. Continuous Discovery
    2. Discovery: Zero network footprint
    3. Centralized graph database
    4. We know everything that changes
    5. Discover and update dependency information
    6. Discovery and monitoring tightly integrated –
    discovery drives automation

    View Slide

  30. © 2015 Assimilation Systems Limited
    33/35
    (even more) Features...
    (even more) Features...
    7. Discovery and monitoring easily extensible
    8. Naturally scalable to > 100K systems
    9. Minimal network load
    10.Server failures distinguishable from switch failures
    11.Best practice and vulnerability alerts
    12.Multi-tenant support

    View Slide

  31. © 2015 Assimilation Systems Limited
    34/35
    Third Dimension:
    Third Dimension:
    Fully distributed work
    Fully distributed work
    Two philosophical underpinnings
    1. Monitoring and Discovery are fully distributed
    2. Reliable “no news is good news”
    Only responses to changes are centralized

    View Slide

  32. © 2015 Assimilation Systems Limited
    35/35
    Service Monitoring based on HA Technologies
    Service Monitoring based on HA Technologies

    Well-proven architecture:
    – reliable “no news is good news”

    Implements Open Cluster Framework
    standard (LSB and others – Nagios coming!)

    Each system monitors own services

    Can also start, stop, migrate services

    View Slide

  33. © 2015 Assimilation Systems Limited
    36/35
    How does discovery work?
    How does discovery work?
    Nanoprobe scripts perform discovery

    Each discovers one kind of information

    Can take arguments from environment

    Output JSON
    CMA stores Discovery Information

    JSON stored in Neo4j database

    CMA discovery plugins => graph nodes and relationships

    View Slide

  34. © 2015 Assimilation Systems Limited
    37/35
    A Few Canned Queries
    A Few Canned Queries
    allipports get all port/ip/service/hosts
    allswitchports get switch connections
    crashed get crashed servers
    shutdown get gracefully shutdown servers
    downservices get nonworking services
    findip get system owning IP
    findmac get system owning MAC
    unknownips get unknown IP addresses
    unmonitored get unmonitored services

    View Slide

  35. © 2015 Assimilation Systems Limited
    38/35
    OS discovery JSON Snippet
    OS discovery JSON Snippet
    { "nodename": "alanr-1225B",
    "operating-system": "GNU/Linux",
    "machine": "x86_64",
    "processor": "x86_64",
    "hardware-platform": "x86_64",
    "kernel-name": "Linux",
    "kernel-release": "3.8.0-31-generic",
    "kernel-version": "#46-Ubuntu SMP ...",
    "Distributor ID": "Ubuntu",
    "Description": "Ubuntu 13.04",
    "Release": "13.04",
    "Codename": "raring" }

    View Slide