Tailored SNMP monitoring: Your own SNMP MIB and sub-agent with Python and python-netsnmpagent

SNMP continues to be a essential component in monitoring where the information being made available is structured in so-called Management Information (MIB) modules. The standard net-snmp distribution comes a with a variety of standard MIBs implemented by its snmpd, but sometimes there is the need to make your own information available via SNMP.

Luckily snmpd can be dynamically extended by so-called subagents implementing the AgentX protocol (RFC2747). The net-snmp API however pretty much focusses on the C programming language only, laying the entrance barrier especially for non-developers rather high.

In this talk Pieter will not only demonstrate the creation of a MIB, which, being a text file, is the easier part, but also how easy it is to implement a simple subagent in Python using his python-netsnmpagent module. python-netsnmpagent is a shim wrapper over the net-snmp C API trying to implement just enough abstraction. Licensed under the GPL it is available at https://github.com/pief/python-netsnmpagent as well as PyPI.

Transcript

1. Tailored SNMP monitoring Your own SNMP MIB and sub-agent with Python and python-netsnmpagent OSMC 2018 November 6th, 2018 Pieter Hollants
2. Pieter who? Frankfurt, Germany-based developer (mostly Python) but also... ➔ 3y IT support in Netware/Win 95 times ➔ 9y (Senior) Intern at SUSE Consulting ➔ 4y Linux Systems Engineer at German Air Traffic Control (e.g. automated installations of high availability-systems, hardware standardization) ...and freelancing since over 15y (dev & admin)
3. MIB?
4. You’re down with…?
5. MIB ➔ Management Information Base ➔ Used to monitor and configure devices remotely ➔ Structured as tree with nameless root ➔ Defines entities called Managed Objects ➔ Access protocol “on the wire”: SNMP (Simple Network Management Protocol)
6. MIB modules ➔ MIB modules (“MIBs”) define branches of the tree, can be enterprise-specific or generic, e.g.: ➔ RFC 2863: IF-MIB (network interfaces) ➔ RFC 4022: TCP/IP MIB ➔ CISCOSB-MIB ➔ Defined in text files using ASN.1 subset called SMI (Structure of Management Information), v2: RFC2578 ➔ Definition != Implementation Wikipedia / CC-BY-SA
7. Managed Objects ➔ Uniquely identified by OID (Object Identifier) ➔ OIDs represented as ordered series of numbers (e.g. “.1.3.6.1.4.1”) or ASCII text labels (e.g. “.iso.org.dod. internet.private.enterprise”) ➔ Enterprise-specific OIDs with IANA assigned Enterprise Number ➔ https://www.iana.org/assignments/enterprise-numbers/enterprise-numbers ➔ Two categories of data types: ➔ Scalar objects: ➔ (Un)Signed Integers ➔ Counters, Gauges, Time ticks ➔ Octet Strings/Display Strings, Bit Strings ➔ Network addresses ➔ Object identifiers ➔ Tabular objects (scalar objects grouped in tables)
8. Who’d write their own MIB? ➔ Application programmers wanting to expose internal metrics (e.g. transactions per second) ➔ OEMs (e.g. Nasuni Edge Appliance: https://cdn2.hubspot.net/ hubfs/3316942/Nasuni.com-assets/Support-Docs/Nasuni_Filer_Third- Party_Licensing_Guide.pdf) ➔ End users wishing to unify otherwise spread-out information under a common tree (e.g. vendor-neutral hardware agent used by German Air Traffic Control) ➔ InfraOps (e.g. DNSSEC monitoring: http://www.delaat.net/rp/2014-2015/p38/presentation.pdf) ➔ You? ;)
9. MIB deep dive Not Bernd Erk (Disclaimer: I know nothing about diving.)
10. MIB deep dive (5m) ➔ Formally defines “<MIBNAME>“ as everything between “BEGIN” and “END” ➔ Multiple MIB modules in a text file possible but uncommon ➔ Two dashes (“--”): comment lines
11. MIB deep dive (10m) ➔ Imports definitions (”features”) from existing MIBs, similar to “from … import …” in Python ➔ RFC 2579: SNMPv2-TC, RFC 2580: SNMPv2-CONF ➔ AgentX? We’ll see later...
12. MIB deep dive (15m) ➔ Collects module purpose, contact information, description, revision information ➔ Defined as OID of subtree this MIB handles (“plugs in”) ➔ Must be first item after module header and imports Wrong OID! Why?
13. MIB deep dive (20m) ➔ Associates a name with subtrees in a particular part of the MIB module’s OID tree, similar to naming subdirs ➔ For human convenience, technically not really required ➔ len(OID subtree def) == 2 ➔ “[..] must not be included in an IMPORTS statement”
14. MIB deep dive (25m) ➔ Defines scalar variables in a particular part of the MIB module’s OID tree, similar to naming files ➔ SYNTAX: type of the variable ➔ MAX-ACCESS: read-only/read-write/not-accessible/... ➔ STATUS: current/deprecated/obsolete ➔ DESCRIPTION: for humans and humanoids
15. MIB deep dive (30m) ➔ Defines a “row” as a SEQUENCE of scalar MOs ➔ Simulation of table through N subtrees ~= columns as defined here (holes possible!) ➔ MOs referenced with name and type only ➔ Still need explicit definition...
16. MIB deep dive (32m) ➔ Definitions of columns in a row ➔ MAX-ACCESS not-accessible in practical use for row that will be used as index row (here the only one) ➔ In this example other rows will be read-only
17. MIB deep dive (34m) ➔ Table ~= SEQUENCE OF (SEQUENCE ...) ➔ Both table and row themselves not-accessible ➔ INDEX magically overwrites 1 with values in referenced index column (values = indexes)
18. MIB deep dive (36m) ➔ Table object not-accessible → snmpget = no such object ➔ Same for defined row, e.g. firstTable.1 = no such object ➔ snmpwalk reveals low-level .<table>.<column>.<row> construct ➔ snmptable knows how to interpret tables semantics ➔ firstTableRowIndex values = indexes
19. MIB deep dive (40m) ➔ Always check your MIBs with “smilint” ➔ Part of libsmi, other nice tools such as “smidiff”, “smidump”… ➔ http://www.ibr.cs.tu-bs.de/projects/libsmi/ ➔ Also validate with increased “-lX” severity levels, then learn about MIB concepts you haven’t heard about yet (e.g. conformance groups)
20. Out of breath?
21. Let’s define our own MIB!
22. Let’s define our own MIB! We need something to monitor...
24. Our GT-MIB (1/2) ...we should actually register our own Enterprise Number with IANA! http://pen.iana.org/pen/PenApplication.page
25. Our GT-MIB (2/2) Index column doesn’t necessarily have to be named “...Index”
26. Implementing SNMP ➔ De-facto standard: Net-SNMP suite (formerly called UCD-SNMP), found in all major distros ➔ Current release: 5.8 (July 16th, 2018) ➔ Consists of ➔ Command-line applications such as snmpget, snmpset, snmpwalk, snmptable, snmptranslate... ➔ Extensible agent (software that handles SNMP requests) with built-in support for wide range of MIB modules: snmpd ➔ Daemon for receiving SNMP traps: snmptrapd ➔ Libraries for developing SNMP apps, C/Perl APIs ➔ Tools such as mib2c
27. Implementing MIBs ➔ snmpd: built-in support for number of MIBs, e.g. IF-MIB ➔ Extensible: snmpd as master agent allows for subagents implementing parts of the MIB (MIB modules) ➔ Three alternatives differing in data representation and communication mechanisms: ➔ Proxied SNMP: standard SNMP packet format, full fledged agent unaware of subagent role on non-standard port ➔ SMUX (RFC 1227): standard SNMP packet format, subagent registers with master agent, issues, historical ➔ AgentX (RFC 2741): more compact packet format, advanced functionality, state of the art, enabled with “master agentx” line in snmpd.conf
28. Net-SNMP mib2c (1/2) mib2c currently can’t handle MIBs with both tables and scalars Different approaches to implement tables possible Input: MIB Output: Skeleton C code
29. Net-SNMP mib2c (2/2) Only init callback shown here
30. Net-SNMP mib2c (2/2) Yeah, okay, C code but… isn’t there something easier?
31. Net-SNMP mib2c (2/2) Yeah, okay, C code but… isn’t there something easier?
32. Python & SNMP support ➔ Net-SNMP ships with Python “netsnmp” module ➔ 2500 lines C code that abstract Net-SNMP C API ➔ SNMP client only: get, walk… operations ➔ C API has support for writing agents as well (as seen with mib2c) ➔ Idea: access C API directly from Python using ctypes, imitating agents written in C ➔ Existing python-agentx module on Sourceforge ➔ Design issues (= I didn’t grok it ;) ➔ Orphaned ➔ 2015: pyagentx module on Github, actually implements AgentX network protocol, meanwhile orphaned as well
33. Hello python-netsnmpagent! ➔ Python module I wrote back in 2013 driven by requirements at German Air Traffic Control ➔ LGPLv3 licensed ➔ Source at https://github.com/pief/python-netsnmpagent ➔ Distributable archives on PyPI ➔ RPMs available on Open Build Service ➔ Compatible with Python 2.6, 2.7 and >=3.5 ➔ Tested with net-snmp 5.4.x (SLES11…), 5.7.x, 5.8 ➔ Two extensively commented files ➔ netsnmpapi.py (ctypes stuff for internal use), 317 LOC ➔ netsnmpagent.py (abstraction classes), 839 LOC ➔ No docs but example MIBs/agents included
34. Our gtmib_agent.py (1/3) Used by Net-SNMP e.g. to translate OIDs Default value for column Classes representing data types Only import required Column number Class factory OID this SNMP object handles
35. Our gtmib_agent.py (2/3) Values for index and data columns Connects to master agent – no more object registrations!
36. Our gtmib_agent.py (3/3) Signal handler triggered eg. by ^C Net-SNMP internal packet processing Example in this form not well-suited for real agents. Why?
37. Et voila! Live demo anyone?
38. The fineprint ➔ Currently no support for callback pattern, i.e. data structures must be periodically refreshed ➔ check_and_process() blocks, threading required to do processing and data updating in parallel ➔ Development has slowed somewhat because no active sponsor anymore → your chance? ➔ Some open issues ➔ No support for SNMP traps yet (but MR exists) ➔ Only rudimentary unit tests ➔ It’s Open Source, take your chance!
39. EOT Pieter Hollants pieter@hollants.com pfhllnts pfhllnts@mstdn.io