Security Best Practices As Code - Boulder DevOps April 2015

As organizations move to a continuous deployment model, security teams fall further and further behind.

What if you could know immediately that a server was out of compliance, so you could correct it right away, and reduce the window of opportunity for attackers?

This talk describes in detail how the Assimilation System Management Suite is implementing this capability.

Event Link: http://assimilationsystems.com/events/security-best-practices-as-code-boulder-devops/

Transcript

1. Security Best Practices As Code Security Best Practices As Code #AssimProj @OSSAlanR http://assimproj.org/ Alan Robertson <alanr@assimilationsystems.com> Assimilation Systems Limited http://assimilationsystems.com
2. © 2015 Assimilation Systems Limited 2/38 Biography Biography • 35+ years in IT/development – 10 years in system management (SysAdmin) • Founded Linux-HA project - led 1998-2007 – aka “Heartbeat” - now called Pacemaker • Founded Assimilation Project in 2010 • Founded Assimilation Systems Limited in 2013 • Alumnus of Bell Labs, SuSE, IBM
3. © 2015 Assimilation Systems Limited 3/38 Assimilation Project Evolution Assimilation Project Evolution • Inspired by 2 million core computer (cyclops64) • Concerns for extreme scale • Topology aware monitoring • Topology discovery w/out security issues =►Discovery of everything!
4. © 2015 Assimilation Systems Limited 4/38 A 7-dimensional overview A 7-dimensional overview 1.System Management Suite Overview 2.Basic Technology 3.Discovery and Monitoring Demo 4.Best Practice Analyses 5.“Toy” Best Practice Demo 6.Current Status 7.What You Need To Do!
5. © 2015 Assimilation Systems Limited 5/38 Why Why the Assimilation System the Assimilation System Management Suite? Management Suite? • Provides insight and details through a graph-model CMDB • Helps you understand and automate your environment – Reduce Errors – Speed up problem resolution • Reduces Manual Documentation • CMDB-driven configuration => near-zero configuration • Automates Monitoring • Enhances Security • Designed for Extreme Scale
6. © 2015 Assimilation Systems Limited 6/38 What's in the Suite? What's in the Suite? • Graph CMDB • Exception Monitoring • Security Discovery • Network Connections
7. Complexity Complexity “Complexity is the enemy of reliability” • Complexity likely your single biggest problem – Near-zero configuration reduces complexity – Tight service integration reduces complexity – Accurate detailed view improves complexity management
8. © 2015 Assimilation Systems Limited 8/38 Highly Scalable Discovery-Driven Highly Scalable Discovery-Driven Automation Automation Continuous Discovery drives everything • Continuous extensible discovery (CMDB) – systems, switches, services, dependencies – zero network footprint discovery process • Extensible exception monitoring – more than 100K systems • Discovery Drives Best Practice Analyses – Initially concentrating on security • All data goes into central graph CMDB
9. © 2015 Assimilation Systems Limited 9/38 This all sounds unreasonable... This all sounds unreasonable... • Huge scalability without complexity? • Discovery without pings or port scans? Really?
10. © 2015 Assimilation Systems Limited 10/38 S Simple Scalability imple Scalability I can explain how we scale so your grandmother would understand... istockphoto ©bowdenimages
11. © 2015 Assimilation Systems Limited 11/38 Massive Scalability – Massive Scalability – or or “I see dead servers in “I see dead servers in O O(1) time” (1) time” • Adding systems does not increase the monitoring work on any system • Each server monitors 2 (or 4) neighbors • Each server monitors and discovers its own services • Ring repair and alerting is O(n) – but a very small amount of work Current Implementation
12. © 2015 Assimilation Systems Limited 12/38 Minimizing Network Footprint Minimizing Network Footprint (in our roadmap) (in our roadmap) • Support diagnosing switch issues • Minimize network traffic • Ideal for multi-site arrangements
13. © 2015 Assimilation Systems Limited 13/38 D Discovery / Monitoring Demo iscovery / Monitoring Demo • Demonstrate basic capabilities – Discovery – Discovery-driven monitoring configuration – Discovery-driven 'tripwire-like' checksums – Monitoring – failures / successes – Host down notification • No configuration was supplied – everything comes from discovery http://assimilationsystems.com/90_second_demo/
14. © 2015 Assimilation Systems Limited 14/38 Service Dependency Graph Service Dependency Graph
15. © 2015 Assimilation Systems Limited 15/38 Switch Discovery Graph Switch Discovery Graph from LLDP (or CDP) from LLDP (or CDP)
16. © 2015 Assimilation Systems Limited 16/38 B Best Practice Analyses est Practice Analyses This is next major planned capability • Triggered by Discovery Updates – Analysis occurs within seconds of change – No change => No analysis • We can analyze anything discovered • Expect to create alerts and reports
17. © 2015 Assimilation Systems Limited 17/38 Sample Security Best Practices Sample Security Best Practices • Inappropriate services (telnet, etc) • Settings in /proc/sys/ • Security Patch Coverage – OS vendor (RedHat, SuSE, Canonical, etc) – Application (Oracle, IBM, WordPress, etc) • Other OS settings • Common Application Settings • Looking at OpenSCAP best practices FYI: Sharing information (collaborating?) with Lynis project
18. © 2015 Assimilation Systems Limited 18/38 Other Sample Security Features Other Sample Security Features • Discovery of “forgotten” IP addresses • Monitoring of Open Ports and Services • Collection of network-facing app checksums • Nmon profiling of new MAC addresses • Checksum outliers analysis • Security Best Practice Analyses
19. © 2015 Assimilation Systems Limited 19/38 “ “T Toy” Best Practices Demo oy” Best Practices Demo • Demo is of test code in our source tree • Test data was extracted from a real system • This code was written to test feasibility of the approach
20. © 2015 Assimilation Systems Limited 20/38 Sample /proc/sys Rules Sample /proc/sys Rules “BPC-00002-1”: {“rule”: “OR(EQ($kernel.core_uses_pid, 1), NE($kernel.core_pattern, ""))” “url”: “https://trello.com/c/6LOXeyDD” }, “BPC-00003-1”: {“rule”: “EQ($kernel.ctrl-alt-del, 0)”, “url”: “https://trello.com/c/aUmn4WFg”}, “BPC-00006-1”: {“rule”: “EQ($kernel.sysrq, 0)”, “url”: “https://trello.com/c/QSovxhup” },
21. © 2015 Assimilation Systems Limited 21/38 Current Status Current Status • 0.6 (Valentine's Day) release out 14 February 2015 • Moving towards security emphasis • Great unit and system tests • Strongly encrypted communication • Quite a few discovery methods written • Extensible Automated Discovery Triggers • Discovery => Automatic Monitoring + Network-Facing Checksums • Command Line Queries • Licenses: Commercial or GPLv3 • Nagios-compatibility, best practice analysis underway
22. © 2015 Assimilation Systems Limited 22/38 Get Involved! Get Involved! • Early adopters – customers! • Contributors – Testers, Continuous Integration – Best practice experts – Designers – Developers (C,Python, Shell, PowerShell, JavaScript) – Porters (esp Windows) – Promoters, Publicists, Packagers, etc.
23. © 2015 Assimilation Systems Limited 23/38 Resistance Is Futile! Resistance Is Futile! These slides: bit.ly/BoulderDevOps0415 Mailing List: bit.ly/AssimML @OSSAlanR #assimilation on irc.freenode.net Project Web Site: assimproj.org Company Web Site: assimilationsystems.com Download: assimilationsystems.com/download
24. © 2015 Assimilation Systems Limited 24/38 Risk Management/Mitigation Risk Management/Mitigation • Intrusions • Vulnerable Software • Licensed Software • Audit Risk • Outages • System management
25. © 2015 Assimilation Systems Limited 25/38 Why a graph database? (Neo4j) Why a graph database? (Neo4j) • Humans describe systems as graphs • Dependency & Discovery information: graph • Speed of graph traversals depends on size of subgraph, not total graph size • Root cause queries  graph traversals – notoriously slow in relational databases • Visualization is Natural • Schema-less design: good for constantly changing heterogeneous environment • Graph Model === Object Model
26. © 2015 Assimilation Systems Limited 26/38 Monitoring Pros and Cons Monitoring Pros and Cons Pros Simple & Scalable Uniform work distribution No single point of failure Distinguishes switch vs host failure Easy on LAN, WAN Multi-tenant approach Cons Active agents Potential slowness at power-on
27. © 2015 Assimilation Systems Limited 27/38 Sixth Dimension: Sixth Dimension: Graph Schema Graph Schema Two Schema subgraphs • Client / server dependency • Switch interconnect
28. © 2015 Assimilation Systems Limited 28/38 "sshd": { "exe": "/usr/sbin/sshd", "cmdline": [ "/usr/sbin/sshd", "-D" ], "uid": "root", "gid": "root", "cwd": "/", "listenaddrs": { "0.0.0.0:22": { "proto": "tcp", "addr": "0.0.0.0", "port": 22 }, sshd sshd Service Service JSON Snippet JSON Snippet (from netstat and /proc) (from netstat and /proc)
29. © 2015 Assimilation Systems Limited 29/38 "ssh": { "exe": "/usr/sbin/ssh", "cmdline": [ "ssh", "servidor" ], "uid": "alanr", "gid": "alanr", "cwd": "/home/alanr/monitor/src", "clientaddrs": { "10.10.10.5:22": { "proto": "tcp", "addr": "10.10.10.5", "port": 22 }, ssh ssh Client Client JSON Snippet JSON Snippet (from netstat and /proc) (from netstat and /proc)
30. First Dimension First Dimension: : Problems Addressed Problems Addressed • Discovering and maintaining documentation (CMDB) using continuous discovery – Services, Systems, Dependencies, Switches, Interconnects, Configuration • Monitoring and alerting: services, systems and compliance • Managing compliance • Mitigating risk
31. © 2015 Assimilation Systems Limited 33/38 Why Discovery? (DevOps) Why Discovery? (DevOps) • Documentation: incomplete, incorrect • Dependencies: unknown • Planning: Needs accurate data • Best Practices: Verification needs data • ITIL CMDB (Configuration Management Data Base) Our Discovery: continuous, low-profile
32. © 2015 Assimilation Systems Limited 34/38 Second Dimension: Second Dimension: Unique Powerful Features Unique Powerful Features 1. Continuous Discovery 2. Discovery: Zero network footprint 3. Centralized graph database 4. We know everything that changes 5. Discover and update dependency information 6. Discovery and monitoring tightly integrated – discovery drives automation
33. © 2015 Assimilation Systems Limited 35/38 (even more) Features... (even more) Features... 7. Discovery and monitoring easily extensible 8. Naturally scalable to > 100K systems 9. Minimal network load 10.Server failures distinguishable from switch failures 11.Best practice and vulnerability alerts 12.Multi-tenant support
34. © 2015 Assimilation Systems Limited 36/38 Third Dimension: Third Dimension: Fully distributed work Fully distributed work Two philosophical underpinnings 1. Monitoring and Discovery are fully distributed 2. Reliable “no news is good news” Only responses to changes are centralized
35. © 2015 Assimilation Systems Limited 37/38 Service Monitoring based on HA Service Monitoring based on HA Technologies Technologies • Well-proven architecture: – reliable “no news is good news” • Implements Open Cluster Framework standard (LSB and others – Nagios coming!) • Each system monitors own services • Can also start, stop, migrate services
36. © 2015 Assimilation Systems Limited 38/38 How does discovery work? How does discovery work? Nanoprobe scripts perform discovery • Each discovers one kind of information • Can take arguments from environment • Output JSON CMA stores Discovery Information • JSON stored in Neo4j database • CMA discovery plugins => graph nodes and relationships
37. © 2015 Assimilation Systems Limited 39/38 A Few Canned Queries A Few Canned Queries allipports get all port/ip/service/hosts allswitchports get switch connections crashed get crashed servers shutdown get gracefully shutdown servers downservices get nonworking services findip get system owning IP findmac get system owning MAC unknownips get unknown IP addresses unmonitored get unmonitored services
38. © 2015 Assimilation Systems Limited 40/38 OS discovery JSON Snippet OS discovery JSON Snippet { "nodename": "alanr-1225B", "operating-system": "GNU/Linux", "machine": "x86_64", "processor": "x86_64", "hardware-platform": "x86_64", "kernel-name": "Linux", "kernel-release": "3.8.0-31-generic", "kernel-version": "#46-Ubuntu SMP ...", "Distributor ID": "Ubuntu", "Description": "Ubuntu 13.04", "Release": "13.04", "Codename": "raring" }