Slide 1

Slide 1 text

Security Best Practices As Code Security Best Practices As Code #AssimProj @OSSAlanR http://assimproj.org/ Alan Robertson Assimilation Systems Limited http://AssimilationSystems.com

Slide 2

Slide 2 text

2/48 Biography Biography ● 35+ years in IT/development – 10 years in system management (SysAdmin) ● Founded Linux-HA project - led 1998-2007 – aka “Heartbeat” - now called Pacemaker ● Founded Assimilation Project in 2010 ● Founded Assimilation Systems Limited in 2013 ● Alumnus of Bell Labs, SuSE, IBM

Slide 3

Slide 3 text

© 2015 Assimilation Systems Limited 3/48 Assimilation Project Evolution Assimilation Project Evolution ● Inspired by 2 million core computer (cyclops64) ● Concerns for extreme scale ● Topology aware monitoring ● Topology discovery w/out security issues =►Discovery of everything! ● Take actions based on discovery!

Slide 4

Slide 4 text

© 2015 Assimilation Systems Limited 4/48 A 6-dimensional overview A 6-dimensional overview 1.System Management Suite Overview 2.Basic Technology 3.Best Practice Analyses 4.“Toy” Best Practice Demo 5.Current Status 6.What You Need To Do!

Slide 5

Slide 5 text

© 2015 Assimilation Systems Limited 5/48 What's in the Suite? What's in the Suite? ● Graph CMDB ● Exception Monitoring ● Security ● Network Connections

Slide 6

Slide 6 text

© 2015 Assimilation Systems Limited 6/48 Disturbing Trends... Disturbing Trends... ● 30% of all break-ins come through “lost” systems (Verizon) ● 90% have had failures of unmonitored services (Turnbull) ● 80% are unable to keep systems in compliance (Verizon) ● 30% start monitoring only after a problem (Turnbull) ● 30% of all systems are doing nothing useful (Koomey) ● Larger site admins often don’t know dependencies

Slide 7

Slide 7 text

© 2015 Assimilation Systems Limited 7/48 Why Why the Assimilation Suite? the Assimilation Suite? ● Provides insight and details through a graph-model CMDB ● Helps you understand and automate your environment – Reduce Errors – Speed up problem resolution ● Reduces Manual Documentation ● Discovery-driven configuration => near-zero configuration ● Automates Monitoring ● Enhances Security ● Designed for Extreme Scale (≈100K systems)

Slide 8

Slide 8 text

© 2015 Assimilation Systems Limited 8/48 Complexity Complexity “Complexity is the enemy of reliability” ● Complexity likely your single biggest problem – Near-zero configuration reduces complexity – Seamless integration reduces complexity – Accurate detailed view improves complexity management

Slide 9

Slide 9 text

© 2015 Assimilation Systems Limited 9/48 Highly Scalable Discovery-Driven Automation Highly Scalable Discovery-Driven Automation ● Continuous extensible discovery (CMDB) ● Extensible exception monitoring ● Discovery Drives (Security) Best Practice Analyses ● All data goes into central graph CMDB

Slide 10

Slide 10 text

© 2015 Assimilation Systems Limited 10/48 This all sounds unreasonable... This all sounds unreasonable... ● Huge scalability without complexity. ● Discovery without pings or port scans. Really?

Slide 11

Slide 11 text

© 2015 Assimilation Systems Limited 11/48 S Simple Scalability imple Scalability I can explain how we scale so your grandmother would understand... istockphoto ©bowdenimages

Slide 12

Slide 12 text

© 2015 Assimilation Systems Limited 12/48 Massive Scalability – Massive Scalability – or or “I see dead servers in “I see dead servers in O O(1) time” (1) time” ● Adding systems does not increase the monitoring work on any system ● Each server monitors 2 (or 4) neighbors ● Each server monitors and discovers its own services ● Ring repair and alerting is O(n) – but a very small amount of work Current Implementation

Slide 13

Slide 13 text

© 2015 Assimilation Systems Limited 13/48 Service Dependency Graph Service Dependency Graph

Slide 14

Slide 14 text

© 2015 Assimilation Systems Limited 14/48 Assimilation Architecture Assimilation Architecture ● Central Collective Management Authority – written in Python – delegates most work to nanoprobes ● Fully distributed “nanoprobe” agents – Simple, policy-free – Written in 'C' – Invoke scripts for monitoring or discovery – Send/receive heartbeats – Listen for ARP, LLDP, CDP packets ● Neo4j graph database

Slide 15

Slide 15 text

© 2015 Assimilation Systems Limited 15/48 Switch Discovery Graph from LLDP / CDP Switch Discovery Graph from LLDP / CDP

Slide 16

Slide 16 text

© 2015 Assimilation Systems Limited 16/48 Security questions Security questions ● Do you think good security staff is easily available? ● Do you think security is going to get better soon? ● Do you think you have enough staff for security to keep up with changes at DevOps / Agile rates?

Slide 17

Slide 17 text

© 2015 Assimilation Systems Limited 17/48 Sample Security Features Sample Security Features ● Discovery of “forgotten” IP addresses ● Monitoring of Open Ports and Services ● Collection of network-facing app checksums ● Nmon OS profiling of new MAC addresses ● Checksum outlier analysis ● Security Best Practice Analyses

Slide 18

Slide 18 text

© 2015 Assimilation Systems Limited 18/48 B Best Practice Analyses est Practice Analyses This is next major planned capability ● Triggered by Discovery Updates – Analysis occurs within seconds of change – No change => No analysis ● We can analyze anything discovered ● Expect to create alerts and reports

Slide 19

Slide 19 text

© 2015 Assimilation Systems Limited 19/48 Sample Security Best Practices Sample Security Best Practices ● Inappropriate OS settings (hardening) ● Security Patch Coverage – OS vendor (RedHat, SuSE, Canonical, etc) – Application (Oracle, IBM, WordPress, etc) ● Inappropriate services (telnet, etc) ● Inappropriate Application Settings (hardening)

Slide 20

Slide 20 text

© 2015 Assimilation Systems Limited 20/48 IT Best Practices Project IT Best Practices Project ITBestPractices.info ● IT-Bestpractices GitHub project ● Apache 2 License ● Initial Sources – NIST STIGs – Lynis project – Individual contributions

Slide 21

Slide 21 text

© 2015 Assimilation Systems Limited 21/48 IT Best Practices Goals IT Best Practices Goals ● Make Best Practice rules available in JSON – Curate potentially mechanically-verifiable practices – Human-readable descriptions of issues and remedies – Multiple language support – Not limited to security best practices – Eventually available through a web server – Source text may change to YAML for readability

Slide 22

Slide 22 text

© 2015 Assimilation Systems Limited 22/48 Sample NIST STIG-based short description Sample NIST STIG-based short description The system must limit the ability of processes to have simultaneous write and execute access to memory.

Slide 23

Slide 23 text

© 2015 Assimilation Systems Limited 23/48 Sample NIST STIG Security Rule check Sample NIST STIG Security Rule check The status of the "kernel.exec-shield" kernel parameter can be queried by running the following command: $ sysctl kernel.exec-shield $ grep kernel.exec-shield /etc/sysctl.conf The output of the command should indicate a value of "1". If this value is not the default value, investigate how it could have been adjusted at runtime, and verify it is not set improperly in "/etc/sysctl.conf". If the correct value is not returned, this is a finding.

Slide 24

Slide 24 text

© 2015 Assimilation Systems Limited 24/48 Sample NIST STIG-based long description Sample NIST STIG-based long description ExecShield uses the segmentation feature on all x86 systems to prevent execution in memory higher than a certain address. It writes an address as a limit in the code segment descriptor, to control where code can be executed, on a per- process basis. When the kernel places a process's memory regions such as the stack and heap higher than this address, the hardware prevents execution in that address range.

Slide 25

Slide 25 text

© 2015 Assimilation Systems Limited 25/48 Assimilation /proc/sys Rule Assimilation /proc/sys Rule Disallow executing code on writable pages “nist_V-38597”: {“rule”: “EQ($kernel.exec-shield, 1)”}

Slide 26

Slide 26 text

© 2015 Assimilation Systems Limited 26/48 Assimilation Networking Rule Assimilation Networking Rule Buffer bloat prevention “itbp-0001”: {“rule”: “IN($kernel.core.default_qdisc, fq_codel, codel)” }

Slide 27

Slide 27 text

© 2015 Assimilation Systems Limited 27/48 “ “T Toy” Assim. Best Practices Demo oy” Assim. Best Practices Demo ● Demo is of test code in our source tree ● Test data was discovered from a real system ● This code was written for feasibility testing

Slide 28

Slide 28 text

© 2015 Assimilation Systems Limited 28/48 Current Status Current Status ● 1.0 (Independence Day) release out 4 July 2015 ● Strongly encrypted communication ● Security is next future emphasis (best practices, etc) ● Discovery => Automatic Monitoring + Network-Facing Checksums ● Compatible with Nagios remote monitoring agent API ● REST + Command Line Queries ● Great unit and system tests ● Licenses: Commercial or GPLv3

Slide 29

Slide 29 text

© 2015 Assimilation Systems Limited 29/48 Get Involved! Get Involved! ● Early adopters – customers! ● Contributors – Testers, Continuous Integration – Best practice experts – Designers – Developers (C,Python, Shell, PowerShell, JavaScript) – Porters (esp Windows) – Promoters, Publicists, Packagers, etc.

Slide 30

Slide 30 text

© 2015 Assimilation Systems Limited 30/48 Assimilation BOF TONIGHT! Assimilation BOF TONIGHT! ● Get All Your Questions Answered ● TONIGHT! ● 7-8 PM (1900-2000) ● Room E145

Slide 31

Slide 31 text

© 2015 Assimilation Systems Limited 31/48 Resistance Is Futile! Resistance Is Futile! These slides: bit.ly/AssimOSCON15 Mailing List: bit.ly/AssimML @OSSAlanR #assimilation on irc.freenode.net Project Web Site: assimproj.org Company Web Site: assimilationsystems.com Download: assimilationsystems.com/download

Slide 32

Slide 32 text

© 2015 Assimilation Systems Limited 32/48 Minimizing Network Footprint (in roadmap) Minimizing Network Footprint (in roadmap) ● Support diagnosing switch issues ● Minimize network traffic ● Ideal for multi-site arrangements

Slide 33

Slide 33 text

© 2015 Assimilation Systems Limited 33/48 D Discovery / Monitoring Demo iscovery / Monitoring Demo ● Demonstrate basic capabilities – Discovery – Discovery-driven monitoring configuration – Discovery-driven 'tripwire-like' checksums – Monitoring – failures / successes – Host down notification ● No configuration was supplied – everything comes from discovery http://assimilationsystems.com/90_second_demo/

Slide 34

Slide 34 text

© 2015 Assimilation Systems Limited 34/48 Risk Management/Mitigation Risk Management/Mitigation ● Intrusions ● Vulnerable Software ● Licensed Software ● Audit Risk ● Outages ● System management

Slide 35

Slide 35 text

© 2015 Assimilation Systems Limited 35/48 Why a graph database? (Neo4j) Why a graph database? (Neo4j) ● Humans describe systems as graphs ● Dependency & Discovery information: graph ● Speed of graph traversals depends on size of subgraph, not total graph size ● Root cause queries  graph traversals – notoriously slow in relational databases ● Visualization is Natural ● Schema-less design: good for constantly changing heterogeneous environment ● Graph Model === Object Model

Slide 36

Slide 36 text

© 2015 Assimilation Systems Limited 36/48 Monitoring Pros and Cons Monitoring Pros and Cons Pros Simple & Scalable Uniform work distribution No single point of failure Distinguishes switch vs host failure Easy on LAN, WAN Multi-tenant approach Cons Active agents Potential slowness at power-on

Slide 37

Slide 37 text

© 2015 Assimilation Systems Limited 37/48 Sixth Dimension: Graph Schema Sixth Dimension: Graph Schema Two Schema subgraphs ● Client / server dependency ● Switch interconnect

Slide 38

Slide 38 text

© 2015 Assimilation Systems Limited 38/48 "sshd": { "exe": "/usr/sbin/sshd", "cmdline": [ "/usr/sbin/sshd", "-D" ], "uid": "root", "gid": "root", "cwd": "/", "listenaddrs": { "0.0.0.0:22": { "proto": "tcp", "addr": "0.0.0.0", "port": 22 }, sshd sshd Service Service JSON Snippet (from netstat and /proc) JSON Snippet (from netstat and /proc)

Slide 39

Slide 39 text

© 2015 Assimilation Systems Limited 39/48 "ssh": { "exe": "/usr/sbin/ssh", "cmdline": [ "ssh", "servidor" ], "uid": "alanr", "gid": "alanr", "cwd": "/home/alanr/monitor/src", "clientaddrs": { "10.10.10.5:22": { "proto": "tcp", "addr": "10.10.10.5", "port": 22 }, ssh ssh Client Client JSON Snippet(from netstat and /proc) JSON Snippet(from netstat and /proc)

Slide 40

Slide 40 text

© 2015 Assimilation Systems Limited 42/48 First Dimension First Dimension: : Problems Addressed Problems Addressed ● Discovering and maintaining documentation (CMDB) using continuous discovery – Services, Systems, Dependencies, Switches, Interconnects, Configuration ● Monitoring and alerting: services, systems and compliance ● Managing compliance ● Mitigating risk

Slide 41

Slide 41 text

© 2015 Assimilation Systems Limited 43/48 Why Discovery? (DevOps) Why Discovery? (DevOps) ● Documentation: incomplete, incorrect ● Dependencies: unknown ● Planning: Needs accurate data ● Best Practices: Verification needs data ● ITIL CMDB (Configuration Management Data Base) Our Discovery: continuous, low-profile

Slide 42

Slide 42 text

© 2015 Assimilation Systems Limited 44/48 Second Dimension: Second Dimension: Unique Powerful Features Unique Powerful Features 1. Continuous Discovery 2. Discovery: Zero network footprint 3. Centralized graph database 4. We know everything that changes 5. Discover and update dependency information 6. Discovery and monitoring tightly integrated – discovery drives automation

Slide 43

Slide 43 text

© 2015 Assimilation Systems Limited 45/48 (even more) Features... (even more) Features... 7. Discovery and monitoring easily extensible 8. Naturally scalable to > 100K systems 9. Minimal network load 10.Server failures distinguishable from switch failures 11.Best practice and vulnerability alerts 12.Multi-tenant support

Slide 44

Slide 44 text

© 2015 Assimilation Systems Limited 46/48 Third Dimension: Third Dimension: Fully distributed work Fully distributed work Two philosophical underpinnings 1. Monitoring and Discovery are fully distributed 2. Reliable “no news is good news” Only responses to changes are centralized

Slide 45

Slide 45 text

© 2015 Assimilation Systems Limited 47/48 Service Monitoring based on HA Technologies Service Monitoring based on HA Technologies ● Well-proven architecture: – reliable “no news is good news” ● Implements Open Cluster Framework standard (LSB and others – Nagios coming!) ● Each system monitors own services ● Can also start, stop, migrate services

Slide 46

Slide 46 text

© 2015 Assimilation Systems Limited 48/48 How does discovery work? How does discovery work? Nanoprobe scripts perform discovery ● Each discovers one kind of information ● Can take arguments from environment ● Output JSON CMA stores Discovery Information ● JSON stored in Neo4j database ● CMA discovery plugins => graph nodes and relationships

Slide 47

Slide 47 text

© 2015 Assimilation Systems Limited 49/48 A Few Canned Queries A Few Canned Queries allipports get all port/ip/service/hosts allswitchports get switch connections crashed get crashed servers shutdown get gracefully shutdown servers downservices get nonworking services findip get system owning IP findmac get system owning MAC unknownips get unknown IP addresses unmonitored get unmonitored services

Slide 48

Slide 48 text

© 2015 Assimilation Systems Limited 50/48 OS discovery JSON Snippet OS discovery JSON Snippet { "nodename": "alanr-1225B", "operating-system": "GNU/Linux", "machine": "x86_64", "processor": "x86_64", "hardware-platform": "x86_64", "kernel-name": "Linux", "kernel-release": "3.8.0-31-generic", "kernel-version": "#46-Ubuntu SMP ...", "Distributor ID": "Ubuntu", "Description": "Ubuntu 13.04", "Release": "13.04", "Codename": "raring" }