Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2015 Open Source Monitoring Conference (OSMC) slides

2015 Open Source Monitoring Conference (OSMC) slides

You can find a video recording of this talk here: http://assimilationsystems.com/videos/2015-open-source-monitoring-conference/

Alan Robertson

November 17, 2015

More Decks by Alan Robertson

Other Decks in Technology


  1. Painlessly Discovering and Monitoring Painlessly Discovering and Monitoring Systems, Services

    and Compliance Systems, Services and Compliance #AssimProj @OSSAlanR http://assimproj.org/ Alan Robertson <[email protected]> Assimilation Systems Limited http://assimilationsystems.com
  2. © 2015 Assimilation Systems Limited 2 Biography Biography • 35+

    years in IT/development – 10 years in system management (SysAdmin) • Founded Linux-HA project - led 1998-2007 – aka “Heartbeat” - now called Pacemaker • Founded Assimilation Project in 2010 • Founded Assimilation Systems Limited in 2013 • Alumnus of Bell Labs, SuSE, IBM
  3. © 2015 Assimilation Systems Limited 3 Assimilation Project Evolution Assimilation

    Project Evolution • Inspired by 2 million core computer (cyclops64) • Concerns for extreme scale • Topology aware monitoring • Topology discovery w/out security issues =►Discovery of everything!
  4. © 2015 Assimilation Systems Limited 4 A 6-dimensional overview A

    6-dimensional overview 1.System Management Suite Overview 2.Basic Technology 3.Best Practice Analyses 4.Demo 5.Current Status 6.What You Need To Do!
  5. © 2015 Assimilation Systems Limited 5 Disturbing Trends... Disturbing Trends...

    30% of break-ins come through “lost” systems (Verizon) 90% have had failures of unmonitored services (Turnbull) 80% are unable to stay in compliance (Verizon) 30% only start monitoring only after a problem (Turnbull) 30% of systems doing nothing useful (Koomey)
  6. © 2015 Assimilation Systems Limited 6 More Trends... More Trends...

    • Larger sites have trouble scaling monitoring (Turnbull) • Larger site admins often don’t know dependencies • Documentation is incomplete, out of date, expensive
  7. © 2015 Assimilation Systems Limited 7 You can't manage what

    you can't You can't manage what you can't see... see... We give you X-Ray vision into your infrastructure • Provides insight and details through a graph-model CMDB • Helps you understand and automate your environment – Reduce Errors – Speed up problem resolution • Reduces Manual Documentation • CMDB-driven configuration => near-zero configuration • Automates Monitoring • Enhances Security • Designed for Extreme Scale
  8. © 2015 Assimilation Systems Limited 8 What's in the Suite?

    What's in the Suite? • Graph CMDB • Exception Monitoring • Security Discovery • Network Connections
  9. © 2015 Assimilation Systems Limited 9 Our Unique Value Our

    Unique Value • Security – continuous security compliance • Scalability – scales like nothing else • Complexity Management – reduces and helps you manage complexity
  10. © 2015 Assimilation Systems Limited 10 Complexity Complexity “Complexity is

    the enemy of reliability” • Complexity likely your single biggest problem – Near-zero configuration reduces complexity – Tight service integration reduces complexity – Accurate detailed information provides insights which help manage complexity
  11. © 2015 Assimilation Systems Limited 11 Highly Scalable Discovery-Driven Highly

    Scalable Discovery-Driven Automation Automation Continuous Discovery drives everything • Continuous extensible discovery (CMDB) – systems, switches, services, dependencies – zero network footprint discovery process • Extensible exception monitoring – more than 100K systems • Discovery Drives Best Practice Analyses – Initially concentrating on security • All data goes into central graph CMDB
  12. © 2015 Assimilation Systems Limited 12 This all sounds unreasonable...

    This all sounds unreasonable... • Huge scalability without complexity? • Discovery without pings or port scans? Really?
  13. © 2015 Assimilation Systems Limited 13 S Simple Scalability imple

    Scalability I can explain how we scale so your grandmother would understand... istockphoto ©bowdenimages
  14. © 2015 Assimilation Systems Limited 14 Massive Scalability – Massive

    Scalability – or or “I see dead servers in “I see dead servers in O O(1) time” (1) time” • Adding systems does not increase the monitoring work on any system • Each server monitors 2 (or 4) neighbors • Each server monitors and discovers its own services • Ring repair and alerting is O(n) – but a very small amount of work Current Implementation
  15. © 2015 Assimilation Systems Limited 15 Minimizing Network Footprint Minimizing

    Network Footprint (in our roadmap) (in our roadmap) • Support diagnosing switch issues • Minimize network traffic • Ideal for multi-site arrangements
  16. © 2015 Assimilation Systems Limited 16 Service Monitoring based on

    HA Service Monitoring based on HA Technologies Technologies • Well-proven architecture: – reliable “no news is good news” • Implements Open Cluster Framework standard, LSB and Nagios remote agent APIs • Each system monitors own services • Can also start, stop, migrate services
  17. © 2015 Assimilation Systems Limited 17 How does discovery work?

    How does discovery work? Nanoprobe scripts perform discovery • Each discovers one kind of information • Can take arguments from environment • Output JSON CMA stores Discovery Information • JSON stored in Neo4j database • CMA discovery plugins => graph nodes and relationships
  18. © 2015 Assimilation Systems Limited 18 OS discovery JSON Snippet

    OS discovery JSON Snippet { "nodename": "alanr-1225B", "operating-system": "GNU/Linux", "machine": "x86_64", "processor": "x86_64", "hardware-platform": "x86_64", "kernel-name": "Linux", "kernel-release": "3.8.0-31-generic", "kernel-version": "#46-Ubuntu SMP ...", "Distributor ID": "Ubuntu", "Description": "Ubuntu 13.04", "Release": "13.04", "Codename": "raring" }
  19. © 2015 Assimilation Systems Limited 19 "sshd": { "exe": "/usr/sbin/sshd",

    "cmdline": [ "/usr/sbin/sshd", "-D" ], "uid": "root", "gid": "root", "cwd": "/", "listenaddrs": { "": { "proto": "tcp" } } sshd sshd Service Service JSON Snippet JSON Snippet (from netstat and /proc) (from netstat and /proc)
  20. © 2015 Assimilation Systems Limited 20 "ssh": { "exe": "/usr/sbin/ssh",

    "cmdline": [ "ssh", "servidor" ], "uid": "alanr", "gid": "alanr", "cwd": "/home/alanr/monitor/src", "clientaddrs": {"": {"proto": "tcp"} } ssh ssh Client Client JSON Snippet JSON Snippet (from netstat and /proc) (from netstat and /proc)
  21. © 2015 Assimilation Systems Limited 22 Switch Discovery Graph Switch

    Discovery Graph from LLDP (or CDP) from LLDP (or CDP)
  22. © 2015 Assimilation Systems Limited 23 Why a graph database?

    (Neo4j) Why a graph database? (Neo4j) • Humans describe systems as graphs • Dependency & Discovery information: graph • Speed of graph traversals depends on size of subgraph, not total graph size • Root cause queries  graph traversals – notoriously slow in relational databases • Visualization is Natural • Schema-less design: good for constantly changing heterogeneous environment • Graph Model === Object Model
  23. © 2015 Assimilation Systems Limited 24 A Few Canned Queries

    A Few Canned Queries allipports get all port/ip/service/hosts allswitchports get switch connections crashed get crashed servers shutdown get gracefully shutdown servers downservices get nonworking services findip get system owning IP findmac get system owning MAC unknownips get unknown IP addresses unmonitored get unmonitored services
  24. © 2015 Assimilation Systems Limited 25 B Best Practice Analyses

    est Practice Analyses Under active development • Triggered by Discovery Updates – Analysis occurs within seconds of change – No change => No analysis • We can analyze anything discovered • Expect to create alerts and reports • SIEM integration
  25. © 2015 Assimilation Systems Limited 26 Sample Security Best Practices

    Sample Security Best Practices • Inappropriate services (telnet, etc) • Settings in /proc/sys/ • Security Patch Coverage – OS vendor (RedHat, SuSE, Canonical, etc) – Application (Oracle, IBM, WordPress, etc) • Other OS settings • Common Application Settings • Looking at best practices FYI: Collaborating with Lynis project and Linux Foundation
  26. © 2015 Assimilation Systems Limited 27 Other Sample Security Features

    Other Sample Security Features • Discovery of “forgotten” IP addresses • Monitoring of Open Ports and Services • Collection of network-facing app checksums • Nmon profiling of new MAC addresses • Checksum outliers analysis • Security Best Practice Analyses
  27. © 2015 Assimilation Systems Limited 28 IT Best Practices Project

    IT Best Practices Project ITBestPractices.info • IT-Bestpractices GitHub project • Working on Linux Foundation Sponsorship • Apache 2 License (or similar) • Initial Sources – DISA STIGs – Lynis project – PCI DSS rules – Individual contributions
  28. © 2015 Assimilation Systems Limited 29 IT Best Practices Goals

    IT Best Practices Goals • Make Best Practice rules available in JSON – Curate mechanically-verifiable practices – Human-readable descriptions of issues and remedies – Multiple language support – Not limited to security best practices – Web server under development
  29. © 2015 Assimilation Systems Limited 30 Sample short description Sample

    short description The system must limit the ability of processes to have simultaneous write and execute access to memory.
  30. © 2015 Assimilation Systems Limited 31 Sample long description Sample

    long description ExecShield uses the segmentation feature on all x86 systems to prevent execution in memory higher than a certain address. It writes an address as a limit in the code segment descriptor, to control where code can be executed, on a per-process basis. When the kernel places a process's memory regions such as the stack and heap higher than this address, the hardware prevents execution in that address range.
  31. © 2015 Assimilation Systems Limited 32 Sample Security Rule check

    Sample Security Rule check The status of the "kernel.exec-shield" kernel parameter can be queried by running the following command: $ sysctl kernel.exec-shield $ grep kernel.exec-shield /etc/sysctl.conf The output of the command should indicate a value of "1". If this value is not the default value, investigate how it could have been adjusted at runtime, and verify it is not set improperly in "/etc/sysctl.conf". If the correct value is not returned, this is a finding.
  32. © 2015 Assimilation Systems Limited 33 Assimilation /proc/sys Rule Assimilation

    /proc/sys Rule Disallow executing code on writable pages “nist_V-38597”: {“rule”: “EQ($kernel.exec-shield, 1)”, “category”: “security” }
  33. © 2015 Assimilation Systems Limited 34 Assimilation Networking Rule Assimilation

    Networking Rule Buffer bloat prevention “itbp-0001”: {“rule”: “IN($kernel.core.default_qdisc, fq_codel, codel)”, “category”: “networking” }
  34. © 2015 Assimilation Systems Limited 35 D Discovery / Monitoring

    / Best iscovery / Monitoring / Best Practices Demo Practices Demo • Demonstrate basic capabilities – Discovery-driven monitoring configuration – Discovery-driven 'tripwire-like' checksums – Monitoring – failures / successes – Host down notification – Best Practices • No configuration was supplied – everything comes from discovery http://assimilationsystems.com/90_second_demo/
  35. © 2015 Assimilation Systems Limited 36 Current Status Current Status

    • 1.1.0 release out 3 November 2015 • Way-cool simplified installer! • Continuous Security Monitoring – ~50 best practice rules implemented • Great unit and system tests • Strongly encrypted communication • Quite a few discovery methods written • Extensible Automated Discovery Triggers • Discovery => Automatic Monitoring + Network-Facing Checksums • Compatible with Nagios remote monitoring agent API • REST + Command Line Queries
  36. © 2015 Assimilation Systems Limited 37 Get Involved! Get Involved!

    • Trials! Early Adopters! • Contributors – Testers, Continuous Integration – Best practice experts – Designers – Developers (C, Python, Shell, PowerShell, JavaScript) – Porters (esp Windows) – Promoters, Publicists, Packagers, etc.
  37. © 2015 Assimilation Systems Limited 38 Resistance Is Futile! Resistance

    Is Futile! These slides: bit.ly/DOSUG0915 Mailing List: bit.ly/AssimML @OSSAlanR #assimilation on irc.freenode.net Project Web Site: assimproj.org Company Web Site: assimilationsystems.com Download: assimilationsystems.com/download
  38. © 2015 Assimilation Systems Limited 39 Risk Management/Mitigation Risk Management/Mitigation

    • Intrusions • Vulnerable Software • Licensed Software • Audit Risk • Outages • System management
  39. © 2015 Assimilation Systems Limited 40 Monitoring Pros and Cons

    Monitoring Pros and Cons Pros Simple & Scalable Uniform work distribution No single point of failure Distinguishes switch vs host failure Easy on LAN, WAN Multi-tenant approach Cons Active agents Potential slowness at power-on
  40. © 2015 Assimilation Systems Limited 41 Sixth Dimension: Sixth Dimension:

    Graph Schema Graph Schema Two Schema subgraphs • Client / server dependency • Switch interconnect
  41. First Dimension First Dimension: : Problems Addressed Problems Addressed •

    Discovering and maintaining documentation (CMDB) using continuous discovery – Services, Systems, Dependencies, Switches, Interconnects, Configuration • Monitoring and alerting: services, systems and compliance • Managing compliance • Mitigating risk
  42. © 2015 Assimilation Systems Limited 45 Why Discovery? (DevOps) Why

    Discovery? (DevOps) • Documentation: incomplete, incorrect • Dependencies: unknown • Planning: Needs accurate data • Best Practices: Verification needs data • ITIL CMDB (Configuration Management Data Base) Our Discovery: continuous, low-profile
  43. © 2015 Assimilation Systems Limited 46 Second Dimension: Second Dimension:

    Unique Powerful Features Unique Powerful Features 1. Continuous Discovery 2. Discovery: Zero network footprint 3. Centralized graph database 4. We know everything that changes 5. Discover and update dependency information 6. Discovery and monitoring tightly integrated – discovery drives automation
  44. © 2015 Assimilation Systems Limited 47 (even more) Features... (even

    more) Features... 7. Discovery and monitoring easily extensible 8. Naturally scalable to > 100K systems 9. Minimal network load 10.Server failures distinguishable from switch failures 11.Best practice and vulnerability alerts 12.Multi-tenant support
  45. © 2015 Assimilation Systems Limited 48 Third Dimension: Third Dimension:

    Fully distributed work Fully distributed work Two philosophical underpinnings 1. Monitoring and Discovery are fully distributed 2. Reliable “no news is good news” Only responses to changes are centralized
  46. © 2015 Assimilation Systems Limited 49 Sample /proc/sys Rules Sample

    /proc/sys Rules “BPC-00002-1”: {“rule”: “OR(EQ($kernel.core_uses_pid, 1), NE($kernel.core_pattern, ""))” “url”: “https://trello.com/c/6LOXeyDD” }, “BPC-00003-1”: {“rule”: “EQ($kernel.ctrl-alt-del, 0)”, “url”: “https://trello.com/c/aUmn4WFg”}, “BPC-00006-1”: {“rule”: “EQ($kernel.sysrq, 0)”, “url”: “https://trello.com/c/QSovxhup” },