Security Automation for DevOps Security Automation for DevOps #AssimProj @OSSAlanR Alan Robertson Assimilation Systems Limited http://AssimilationSystems.com

2/35 Biography Biography ● 35+ years in IT/development – 10 years in system management (SysAdmin) ● Founded Linux-HA project - led 1998-2007 – aka “Heartbeat” - now called Pacemaker ● Founded Assimilation Project in 2010 ● Founded Assimilation Systems Limited in 2013 ● Alumnus of Bell Labs, SuSE, IBM

© 2015 Assimilation Systems Limited 4/35 Security questions Security questions ● Do you think good security staff is easily available? ● Do you think security is going to get better soon? ● Do you think you have enough staff for security to keep up with changes at DevOps / Agile rates?

© 2015 Assimilation Systems Limited 5/35 Disturbing Trends... Disturbing Trends... 30% of break-ins come through “lost” systems (Verizon) 90% have had failures of unmonitored services (Turnbull) 71% are unable to stay in compliance (Verizon) 30% only start monitoring only after a problem (Turnbull) 30% of systems doing nothing useful (Koomey)

Highly Scalable Discovery-Driven Highly Scalable Discovery-Driven Automation Automation Continuous Discovery drives everything ● Continuous extensible discovery (CMDB) – systems, switches, services, dependencies – zero network footprint discovery process ● Extensible exception monitoring – more than 100K systems ● Discovery Drives Best Practice Analyses – Initially concentrating on security ● All data goes into central graph CMDB (Config Mgmt Data Base)

© 2015 Assimilation Systems Limited 7/35 This all sounds unreasonable... This all sounds unreasonable... ● Huge scalability without complexity? ● Discovery without pings or port scans? Really?

© 2015 Assimilation Systems Limited 8/35 Simple Scalability Simple Scalability I can explain how we scale so your grandmother would understand...

Massive Scalability – Massive Scalability – or or “I see dead servers in “I see dead servers in O O(1) time” (1) time” ● Adding systems does not increase the monitoring work on any system ● Each server monitors 2 (or 4) neighbors ● Each server monitors and discovers its own services ● Ring repair and alerting is O(n) – but a very small amount of work Current Implementation

© 2015 Assimilation Systems Limited 10/35 Minimizing Network Footprint Minimizing Network Footprint (in roadmap) (in roadmap) ● Support diagnosing switch issues ● Minimize network traffic ● Ideal for multi-site arrangements

© 2015 Assimilation Systems Limited 11/35 Assimilation Architecture Assimilation Architecture ● Central Collective Management Authority – written in Python – delegates most work to nanoprobes – does nothing as much as possible – Doing nothing scaless really well – should be into the 100K system range ● Fully distributed “nanoprobe” agents – Simple, policy-free – Written in 'C' – Run scripts for monitoring or discovery – Send/receive heartbeats – Listen for ARP, LLDP, CDP packets ● Neo4j graph database

© 2015 Assimilation Systems Limited 12/35 Assimilation Security Discovery Assimilation Security Discovery ● All IP+MAC Addresses on subnet ● Network Connectivity (CDP/LLDP) ● Packages and versions installed ● Services Offered / Used (netstat) ● Checksums of network-facing binaries, libraries, JARs ● Many other detailed security settings, permissions – (sshd_config, PAM, /proc/sys, auditd_conf, sudoers, etc)

© 2015 Assimilation Systems Limited 13/35 Assimilation Analysis / Reports Assimilation Analysis / Reports ● Comparison to best practices ● Risk scoring ● Unknown IP Addresses ● Automatic service monitoring ● Unmonitored services ● Triage-related risk scores

© 2015 Assimilation Systems Limited 14/35 B Best Practice Analyses est Practice Analyses ● Triggered by Discovery Updates – Analysis occurs within seconds of change – No change => No analysis ● We can analyze anything discovered ● You can easily discover anything you want ● Alerts and Reports available

© 2015 Assimilation Systems Limited 15/35 Demo Demo Everything will be discovered – nothing will be configured manually ● What needs hardening ● How to Triage your hardening issues ● How to Demonstrate and Track Progress ● How to keep them in compliance (hardened) ● Visualizing Your Attack Surface ● Who has what package+version – Docker package discovery too!

© 2015 Assimilation Systems Limited 16/35 Where to find this Information online? Where to find this Information online? http://assimilationsystems.com/category/getting-started/ 1. 15 Minutes To Better Security 2. An Hour To Better Security 3. A Half-Day To Better Security Where to See Similar Demos ● http://assimilationsystems.com/category/videos/ ● http://assimilationsystems.com/sample-demo-output/

© 2015 Assimilation Systems Limited 17/35 Future Plans Future Plans ● Checksum whitelist/blacklist ==> risk scores ● Checksum queries ● Create service consolidating vendor patches ==> risk scores ● Interactive User Interface(s) (GUI)

© 2015 Assimilation Systems Limited 18/35 Get Involved! Get Involved! ● Get Assimilated! ● Contribute! – Testers, System Management, Continuous Integration – Best practice experts – Designers – Developers (C,Python, Shell, PowerShell, JavaScript) – Porters (esp Windows) – Promoters, Publicists, Packagers, etc.

© 2015 Assimilation Systems Limited 19/35 Resistance Is Futile! Resistance Is Futile! These slides: bit.ly/DevOpsDaysRox16 Mailing List: bit.ly/AssimML @OSSAlanR #assimilation on irc.freenode.net Project Web Site: assimproj.org Company Web Site: assimilationsystems.com Download: assimilationsystems.com/download

© 2015 Assimilation Systems Limited 20/35 Switch Discovery Graph from LLDP / CDP Switch Discovery Graph from LLDP / CDP

© 2015 Assimilation Systems Limited 21/35 Security Demo / HOWTO Security Demo / HOWTO ● No configuration was supplied – everything comes from discovery

© 2015 Assimilation Systems Limited 22/35 Risk Management/Mitigation Risk Management/Mitigation ● Intrusions ● Vulnerable Software ● Licensed Software ● Audit Risk ● Outages ● System management

© 2015 Assimilation Systems Limited 23/35 Why a graph database? (Neo4j) Why a graph database? (Neo4j) ● Humans describe systems as graphs ● Dependency & Discovery information: graph ● Speed of graph traversals depends on size of subgraph, not total graph size ● Root cause queries  graph traversals – notoriously slow in relational databases ● Visualization is Natural ● Schema-less design: good for constantly changing heterogeneous environment ● Graph Model === Object Model

© 2015 Assimilation Systems Limited 24/35 Monitoring Pros and Cons Monitoring Pros and Cons Pros Simple & Scalable Uniform work distribution No single point of failure Distinguishes switch vs host failure Easy on LAN, WAN Multi-tenant approach Cons Active agents Potential slowness at power-on

© 2015 Assimilation Systems Limited 25/35 Sixth Dimension: Graph Schema Sixth Dimension: Graph Schema Two Schema subgraphs ● Client / server dependency ● Switch interconnect

© 2015 Assimilation Systems Limited 26/35 "sshd": { "exe": "/usr/sbin/sshd", "cmdline": [ "/usr/sbin/sshd", "-D" ], "uid": "root", "gid": "root", "cwd": "/", "listenaddrs": { "0.0.0.0:22": { "proto": "tcp", "addr": "0.0.0.0", "port": 22 }, sshd sshd Service Service JSON Snippet (from netstat and /proc) JSON Snippet (from netstat and /proc)

© 2015 Assimilation Systems Limited 27/35 "ssh": { "exe": "/usr/sbin/ssh", "cmdline": [ "ssh", "servidor" ], "uid": "alanr", "gid": "alanr", "cwd": "/home/alanr/monitor/src", "clientaddrs": { "10.10.10.5:22": { "proto": "tcp", "addr": "10.10.10.5", "port": 22 }, ssh ssh Client Client JSON Snippet(from netstat and /proc) JSON Snippet(from netstat and /proc)

© 2015 Assimilation Systems Limited 30/35 First Dimension First Dimension: : Problems Addressed Problems Addressed ● Discovering and maintaining documentation (CMDB) using continuous discovery – Services, Systems, Dependencies, Switches, Interconnects, Configuration ● Monitoring and alerting: services, systems and compliance ● Managing compliance ● Mitigating risk

© 2015 Assimilation Systems Limited 31/35 Why Discovery? (DevOps) Why Discovery? (DevOps) ● Documentation: incomplete, incorrect ● Dependencies: unknown ● Planning: Needs accurate data ● Best Practices: Verification needs data ● ITIL CMDB (Configuration Management Data Base) Our Discovery: continuous, low-profile

© 2015 Assimilation Systems Limited 32/35 Second Dimension: Second Dimension: Unique Powerful Features Unique Powerful Features 1. Continuous Discovery 2. Discovery: Zero network footprint 3. Centralized graph database 4. We know everything that changes 5. Discover and update dependency information 6. Discovery and monitoring tightly integrated – discovery drives automation

© 2015 Assimilation Systems Limited 33/35 (even more) Features... (even more) Features... 7. Discovery and monitoring easily extensible 8. Naturally scalable to > 100K systems 9. Minimal network load 10.Server failures distinguishable from switch failures 11.Best practice and vulnerability alerts 12.Multi-tenant support

© 2015 Assimilation Systems Limited 34/35 Third Dimension: Third Dimension: Fully distributed work Fully distributed work Two philosophical underpinnings 1. Monitoring and Discovery are fully distributed 2. Reliable “no news is good news” Only responses to changes are centralized

© 2015 Assimilation Systems Limited 35/35 Service Monitoring based on HA Technologies Service Monitoring based on HA Technologies ● Well-proven architecture: – reliable “no news is good news” ● Implements Open Cluster Framework standard (LSB and others – Nagios coming!) ● Each system monitors own services ● Can also start, stop, migrate services

© 2015 Assimilation Systems Limited 36/35 How does discovery work? How does discovery work? Nanoprobe scripts perform discovery ● Each discovers one kind of information ● Can take arguments from environment ● Output JSON CMA stores Discovery Information ● JSON stored in Neo4j database ● CMA discovery plugins => graph nodes and relationships

© 2015 Assimilation Systems Limited 37/35 A Few Canned Queries A Few Canned Queries allipports get all port/ip/service/hosts allswitchports get switch connections crashed get crashed servers shutdown get gracefully shutdown servers downservices get nonworking services findip get system owning IP findmac get system owning MAC unknownips get unknown IP addresses unmonitored get unmonitored services

© 2015 Assimilation Systems Limited 38/35 OS discovery JSON Snippet OS discovery JSON Snippet { "nodename": "alanr-1225B", "operating-system": "GNU/Linux", "machine": "x86_64", "processor": "x86_64", "hardware-platform": "x86_64", "kernel-name": "Linux", "kernel-release": "3.8.0-31-generic", "kernel-version": "#46-Ubuntu SMP ...", "Distributor ID": "Ubuntu", "Description": "Ubuntu 13.04", "Release": "13.04", "Codename": "raring" }