Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2015 OSCON - Security Best Practices as Code

2015 OSCON - Security Best Practices as Code

Assimilation Presentation at OSCON featuring security best practices. Video at https://youtu.be/SsiKLqxz3Yo

The statistics on system management are alarming - 30% of all break-ins come through systems people have lost track of, 90% of all organizations have failures of services they aren’t monitoring, 80% of all organizations are unable to keep their systems in compliance after getting them there initially, and 30% admit that they rarely start monitoring until after they have a problem, 30% of all systems are doing nothing useful, and admins of larger sites often don’t know the inter-dependencies between systems, services, and switches.

The Assimilation System Management Suite helps you deal with these problems by creating a detailed graph database and driving audits, monitoring, and security policies from it in a way that scales like nothing else, and providing detailed data of what changed and what happened, and each piece relates to the other to help determine the root cause of an outage.

Event Link: http://assimilationsystems.com/events/oscon-2015/

Alan Robertson

July 23, 2015

More Decks by Alan Robertson

Other Decks in Technology


  1. Security Best Practices As Code
    Security Best Practices As Code
    #AssimProj @OSSAlanR
    Alan Robertson
    Assimilation Systems Limited

    View full-size slide

  2. 2/48

    35+ years in IT/development – 10 years in system
    management (SysAdmin)

    Founded Linux-HA project - led 1998-2007 – aka “Heartbeat” -
    now called Pacemaker

    Founded Assimilation Project in 2010

    Founded Assimilation Systems Limited in 2013

    Alumnus of Bell Labs, SuSE, IBM

    View full-size slide

  3. © 2015 Assimilation Systems Limited
    Assimilation Project Evolution
    Assimilation Project Evolution

    Inspired by 2 million core computer (cyclops64)

    Concerns for extreme scale

    Topology aware monitoring

    Topology discovery w/out security issues
    =►Discovery of everything!

    Take actions based on discovery!

    View full-size slide

  4. © 2015 Assimilation Systems Limited
    A 6-dimensional overview
    A 6-dimensional overview
    1.System Management Suite Overview
    2.Basic Technology
    3.Best Practice Analyses
    4.“Toy” Best Practice Demo
    5.Current Status
    6.What You Need To Do!

    View full-size slide

  5. © 2015 Assimilation Systems Limited
    What's in the Suite?
    What's in the Suite?

    Graph CMDB

    Exception Monitoring


    Network Connections

    View full-size slide

  6. © 2015 Assimilation Systems Limited
    Disturbing Trends...
    Disturbing Trends...

    30% of all break-ins come through “lost” systems (Verizon)

    90% have had failures of unmonitored services (Turnbull)

    80% are unable to keep systems in compliance (Verizon)

    30% start monitoring only after a problem (Turnbull)

    30% of all systems are doing nothing useful (Koomey)

    Larger site admins often don’t know dependencies

    View full-size slide

  7. © 2015 Assimilation Systems Limited
    Why the Assimilation Suite?
    the Assimilation Suite?

    Provides insight and details through a graph-model CMDB

    Helps you understand and automate your environment
    – Reduce Errors
    – Speed up problem resolution

    Reduces Manual Documentation

    Discovery-driven configuration => near-zero configuration

    Automates Monitoring

    Enhances Security

    Designed for Extreme Scale (≈100K systems)

    View full-size slide

  8. © 2015 Assimilation Systems Limited
    “Complexity is the enemy of reliability”

    Complexity likely your single biggest problem
    – Near-zero configuration reduces complexity
    – Seamless integration reduces complexity
    – Accurate detailed view improves complexity management

    View full-size slide

  9. © 2015 Assimilation Systems Limited
    Highly Scalable Discovery-Driven Automation
    Highly Scalable Discovery-Driven Automation

    Continuous extensible discovery (CMDB)

    Extensible exception monitoring

    Discovery Drives (Security) Best Practice Analyses

    All data goes into central graph CMDB

    View full-size slide

  10. © 2015 Assimilation Systems Limited
    This all sounds unreasonable...
    This all sounds unreasonable...

    Huge scalability without complexity.

    Discovery without pings or port scans.

    View full-size slide

  11. © 2015 Assimilation Systems Limited
    Simple Scalability
    imple Scalability
    I can explain how we scale so your grandmother
    would understand...

    View full-size slide

  12. © 2015 Assimilation Systems Limited
    Massive Scalability –
    Massive Scalability – or
    “I see dead servers in
    “I see dead servers in O
    O(1) time”
    (1) time”

    Adding systems does not increase the monitoring work on any system

    Each server monitors 2 (or 4) neighbors

    Each server monitors and discovers its own services

    Ring repair and alerting is O(n) – but a very small amount of work
    Current Implementation

    View full-size slide

  13. © 2015 Assimilation Systems Limited
    Service Dependency Graph
    Service Dependency Graph

    View full-size slide

  14. © 2015 Assimilation Systems Limited
    Assimilation Architecture
    Assimilation Architecture

    Central Collective Management Authority
    – written in Python
    – delegates most work to nanoprobes

    Fully distributed “nanoprobe” agents
    – Simple, policy-free
    – Written in 'C'
    – Invoke scripts for monitoring or discovery
    – Send/receive heartbeats
    – Listen for ARP, LLDP, CDP packets

    Neo4j graph database

    View full-size slide

  15. © 2015 Assimilation Systems Limited
    Switch Discovery Graph from LLDP / CDP
    Switch Discovery Graph from LLDP / CDP

    View full-size slide

  16. © 2015 Assimilation Systems Limited
    Security questions
    Security questions

    Do you think good security staff is easily available?

    Do you think security is going to get better soon?

    Do you think you have enough staff for security to keep up
    with changes at DevOps / Agile rates?

    View full-size slide

  17. © 2015 Assimilation Systems Limited
    Sample Security Features
    Sample Security Features

    Discovery of “forgotten” IP addresses

    Monitoring of Open Ports and Services

    Collection of network-facing app checksums

    Nmon OS profiling of new MAC addresses

    Checksum outlier analysis

    Security Best Practice Analyses

    View full-size slide

  18. © 2015 Assimilation Systems Limited
    Best Practice Analyses
    est Practice Analyses
    This is next major planned capability

    Triggered by Discovery Updates
    – Analysis occurs within seconds of change
    – No change => No analysis

    We can analyze anything discovered

    Expect to create alerts and reports

    View full-size slide

  19. © 2015 Assimilation Systems Limited
    Sample Security Best Practices
    Sample Security Best Practices

    Inappropriate OS settings (hardening)

    Security Patch Coverage
    – OS vendor (RedHat, SuSE, Canonical, etc)
    – Application (Oracle, IBM, WordPress, etc)

    Inappropriate services (telnet, etc)

    Inappropriate Application Settings (hardening)

    View full-size slide

  20. © 2015 Assimilation Systems Limited
    IT Best Practices Project
    IT Best Practices Project

    IT-Bestpractices GitHub project

    Apache 2 License

    Initial Sources
    – NIST STIGs
    – Lynis project
    – Individual contributions

    View full-size slide

  21. © 2015 Assimilation Systems Limited
    IT Best Practices Goals
    IT Best Practices Goals

    Make Best Practice rules available in JSON
    – Curate potentially mechanically-verifiable practices
    – Human-readable descriptions of issues and remedies
    – Multiple language support
    – Not limited to security best practices
    – Eventually available through a web server
    – Source text may change to YAML for readability

    View full-size slide

  22. © 2015 Assimilation Systems Limited
    Sample NIST STIG-based short description
    Sample NIST STIG-based short description
    The system must limit the ability of processes to
    have simultaneous write and execute access to

    View full-size slide

  23. © 2015 Assimilation Systems Limited
    Sample NIST STIG Security Rule check
    Sample NIST STIG Security Rule check
    The status of the "kernel.exec-shield" kernel parameter can be
    queried by running the following command:
    $ sysctl kernel.exec-shield
    $ grep kernel.exec-shield /etc/sysctl.conf
    The output of the command should indicate a value of "1". If this value
    is not the default value, investigate how it could have been adjusted
    at runtime, and verify it is not set improperly in "/etc/sysctl.conf".
    If the correct value is not returned, this is a finding.

    View full-size slide

  24. © 2015 Assimilation Systems Limited
    Sample NIST STIG-based long description
    Sample NIST STIG-based long description
    ExecShield uses the segmentation feature on all x86 systems
    to prevent execution in memory higher than a certain
    address. It writes an address as a limit in the code segment
    descriptor, to control where code can be executed, on a per-
    process basis. When the kernel places a process's memory
    regions such as the stack and heap higher than this address,
    the hardware prevents execution in that address range.

    View full-size slide

  25. © 2015 Assimilation Systems Limited
    Assimilation /proc/sys Rule
    Assimilation /proc/sys Rule
    Disallow executing code on writable pages
    {“rule”: “EQ($kernel.exec-shield, 1)”}

    View full-size slide

  26. © 2015 Assimilation Systems Limited
    Assimilation Networking Rule
    Assimilation Networking Rule
    Buffer bloat prevention
    {“rule”: “IN($kernel.core.default_qdisc,
    fq_codel, codel)”

    View full-size slide

  27. © 2015 Assimilation Systems Limited

    Toy” Assim. Best Practices Demo
    oy” Assim. Best Practices Demo

    Demo is of test code in our source tree

    Test data was discovered from a real system

    This code was written for feasibility testing

    View full-size slide

  28. © 2015 Assimilation Systems Limited
    Current Status
    Current Status

    1.0 (Independence Day) release out 4 July 2015

    Strongly encrypted communication

    Security is next future emphasis (best practices, etc)

    Discovery => Automatic Monitoring + Network-Facing Checksums

    Compatible with Nagios remote monitoring agent API

    REST + Command Line Queries

    Great unit and system tests

    Licenses: Commercial or GPLv3

    View full-size slide

  29. © 2015 Assimilation Systems Limited
    Get Involved!
    Get Involved!

    Early adopters – customers!

    – Testers, Continuous Integration
    – Best practice experts
    – Designers
    – Developers (C,Python, Shell, PowerShell, JavaScript)
    – Porters (esp Windows)
    – Promoters, Publicists, Packagers, etc.

    View full-size slide

  30. © 2015 Assimilation Systems Limited
    Assimilation BOF TONIGHT!
    Assimilation BOF TONIGHT!

    Get All Your Questions Answered


    7-8 PM (1900-2000)

    Room E145

    View full-size slide

  31. © 2015 Assimilation Systems Limited
    Resistance Is Futile!
    Resistance Is Futile!
    These slides: bit.ly/AssimOSCON15
    Mailing List: bit.ly/AssimML
    #assimilation on irc.freenode.net
    Project Web Site: assimproj.org
    Company Web Site: assimilationsystems.com
    Download: assimilationsystems.com/download

    View full-size slide

  32. © 2015 Assimilation Systems Limited
    Minimizing Network Footprint (in roadmap)
    Minimizing Network Footprint (in roadmap)

    Support diagnosing switch issues

    Minimize network traffic

    Ideal for multi-site arrangements

    View full-size slide

  33. © 2015 Assimilation Systems Limited
    Discovery / Monitoring Demo
    iscovery / Monitoring Demo

    Demonstrate basic capabilities
    – Discovery
    – Discovery-driven monitoring configuration
    – Discovery-driven 'tripwire-like' checksums
    – Monitoring – failures / successes
    – Host down notification

    No configuration was supplied
    – everything comes from discovery

    View full-size slide

  34. © 2015 Assimilation Systems Limited
    Risk Management/Mitigation
    Risk Management/Mitigation


    Vulnerable Software

    Licensed Software

    Audit Risk


    System management

    View full-size slide

  35. © 2015 Assimilation Systems Limited
    Why a graph database? (Neo4j)
    Why a graph database? (Neo4j)

    Humans describe systems as graphs

    Dependency & Discovery information: graph

    Speed of graph traversals depends on size of subgraph, not
    total graph size

    Root cause queries  graph traversals – notoriously slow in
    relational databases

    Visualization is Natural

    Schema-less design: good for constantly changing
    heterogeneous environment

    Graph Model === Object Model

    View full-size slide

  36. © 2015 Assimilation Systems Limited
    Monitoring Pros and Cons
    Monitoring Pros and Cons
    Simple & Scalable
    Uniform work distribution
    No single point of failure
    Distinguishes switch vs
    host failure
    Easy on LAN, WAN
    Multi-tenant approach
    Active agents
    Potential slowness
    at power-on

    View full-size slide

  37. © 2015 Assimilation Systems Limited
    Sixth Dimension: Graph Schema
    Sixth Dimension: Graph Schema
    Two Schema subgraphs

    Client / server

    Switch interconnect

    View full-size slide

  38. © 2015 Assimilation Systems Limited
    "sshd": {
    "exe": "/usr/sbin/sshd",
    "cmdline": [ "/usr/sbin/sshd", "-D" ],
    "uid": "root",
    "gid": "root",
    "cwd": "/",
    "listenaddrs": {
    "": {
    "proto": "tcp",
    "addr": "",
    "port": 22 },
    sshd Service
    Service JSON Snippet (from netstat and /proc)
    JSON Snippet (from netstat and /proc)

    View full-size slide

  39. © 2015 Assimilation Systems Limited
    "ssh": {
    "exe": "/usr/sbin/ssh",
    "cmdline": [ "ssh", "servidor" ],
    "uid": "alanr",
    "gid": "alanr",
    "cwd": "/home/alanr/monitor/src",
    "clientaddrs": {
    "": {
    "proto": "tcp",
    "addr": "",
    "port": 22 },
    ssh Client
    Client JSON Snippet(from netstat and /proc)
    JSON Snippet(from netstat and /proc)

    View full-size slide

  40. © 2015 Assimilation Systems Limited
    First Dimension
    First Dimension:
    Problems Addressed
    Problems Addressed

    Discovering and maintaining documentation (CMDB) using
    continuous discovery
    – Services, Systems, Dependencies, Switches, Interconnects, Configuration

    Monitoring and alerting: services, systems and compliance

    Managing compliance

    Mitigating risk

    View full-size slide

  41. © 2015 Assimilation Systems Limited
    Why Discovery? (DevOps)
    Why Discovery? (DevOps)

    Documentation: incomplete, incorrect

    Dependencies: unknown

    Planning: Needs accurate data

    Best Practices: Verification needs data

    ITIL CMDB (Configuration Management
    Data Base)
    Our Discovery: continuous, low-profile

    View full-size slide

  42. © 2015 Assimilation Systems Limited
    Second Dimension:
    Second Dimension:
    Unique Powerful Features
    Unique Powerful Features
    1. Continuous Discovery
    2. Discovery: Zero network footprint
    3. Centralized graph database
    4. We know everything that changes
    5. Discover and update dependency information
    6. Discovery and monitoring tightly integrated –
    discovery drives automation

    View full-size slide

  43. © 2015 Assimilation Systems Limited
    (even more) Features...
    (even more) Features...
    7. Discovery and monitoring easily extensible
    8. Naturally scalable to > 100K systems
    9. Minimal network load
    10.Server failures distinguishable from switch failures
    11.Best practice and vulnerability alerts
    12.Multi-tenant support

    View full-size slide

  44. © 2015 Assimilation Systems Limited
    Third Dimension:
    Third Dimension:
    Fully distributed work
    Fully distributed work
    Two philosophical underpinnings
    1. Monitoring and Discovery are fully distributed
    2. Reliable “no news is good news”
    Only responses to changes are centralized

    View full-size slide

  45. © 2015 Assimilation Systems Limited
    Service Monitoring based on HA Technologies
    Service Monitoring based on HA Technologies

    Well-proven architecture:
    – reliable “no news is good news”

    Implements Open Cluster Framework
    standard (LSB and others – Nagios coming!)

    Each system monitors own services

    Can also start, stop, migrate services

    View full-size slide

  46. © 2015 Assimilation Systems Limited
    How does discovery work?
    How does discovery work?
    Nanoprobe scripts perform discovery

    Each discovers one kind of information

    Can take arguments from environment

    Output JSON
    CMA stores Discovery Information

    JSON stored in Neo4j database

    CMA discovery plugins => graph nodes and relationships

    View full-size slide

  47. © 2015 Assimilation Systems Limited
    A Few Canned Queries
    A Few Canned Queries
    allipports get all port/ip/service/hosts
    allswitchports get switch connections
    crashed get crashed servers
    shutdown get gracefully shutdown servers
    downservices get nonworking services
    findip get system owning IP
    findmac get system owning MAC
    unknownips get unknown IP addresses
    unmonitored get unmonitored services

    View full-size slide

  48. © 2015 Assimilation Systems Limited
    OS discovery JSON Snippet
    OS discovery JSON Snippet
    { "nodename": "alanr-1225B",
    "operating-system": "GNU/Linux",
    "machine": "x86_64",
    "processor": "x86_64",
    "hardware-platform": "x86_64",
    "kernel-name": "Linux",
    "kernel-release": "3.8.0-31-generic",
    "kernel-version": "#46-Ubuntu SMP ...",
    "Distributor ID": "Ubuntu",
    "Description": "Ubuntu 13.04",
    "Release": "13.04",
    "Codename": "raring" }

    View full-size slide