Zabbix Low-Level Discovery (LLD) from a C module

Using a real-life practical example which we use to monitor vehicles issued with GPS trackers which communicate via MQTT, we will discuss how we implement Zabbix Low-Level Discovery directly from a C module and how the same C module is used to provide up-to-date information from the vehicles to Zabbix items. This basic principle can easily be adapted to provide similar functionility to Internet of Things (IoT) projects

1. Zabbix loadable C modules and Low-Level Discovery (LLD) Jan-Piet Mens OSMC, November 2019 @jpmens
2. @jpmens: consultant, part-time admin, trainer, small-scale fiddler, loves plain text, and things which work. Contributes to Ansible, dreamed up OwnTracks, and chases bugs in open source DNS servers.
3. This is OwnTracks
4. Zabbix cannot, out of the box, support every possible thing you may want to monitor; let's extend it. extensible
5. extending Zabbix system.run[] User Parameters External checks Loadable modules Beware fork(), invalid results: unsupported items
6. system.run[] # agent EnableRemoteCommands=1 LogRemoteCommands=1 76775:20190326:173402.315 Executing command '/usr/local/bin/zdisc.sh one two three' #!/bin/sh echo $# Runs on agent
7. user parameters #!/bin/sh echo 16 exit 0 # agent
 UserParameter=jjolie.cn[*],echo "$1 Jolie" UserParameter=jjolie.uid,/usr/local/bin/uidnumber.sh Runs on agent
8. external checks Runs on server # server
 ExternalScripts=/usr/lib/zabbix/externalscripts
9. loadable modules Written in the C language, compiled, loaded dynamically by server, proxy, and/or agent
10. what can modules do? Basically anything: sockets, System V IPC, files, databases, IoT, sensors, … Beware permissions, context, errors
11. custom module # server, proxy, agent LoadModulePath=/usr/lib/zabbix/agent
 LoadModule=owntracks-module.so
 LoadModule=two.so
12. make install Build inside / outside source tree owntracks-module.so: owntracks-module.c gcc -fPIC -shared -o owntracks-module.so owntracks-module.c install owntracks-module.so /usr/lib/zabbix/agent/
13. module.h --- zabbix-4.0.5/include/module.h 2019-02-25 10:36:39.000000000 +0100 +++ module.h 2019-03-26 15:03:59.000000000 +0100 @@ -20,7 +20,8 @@ #ifndef ZABBIX_MODULE_H #define ZABBIX_MODULE_H -#include "zbxtypes.h" +#include <inttypes.h> +#define zbx_uint64_t uint64_t #define ZBX_MODULE_OK 0 #define ZBX_MODULE_FAIL -1
14. the API int zbx_module_api_version(void) int zbx_module_init(void) ZBX_METRIC *zbx_module_item_list(void) void zbx_module_item_timeout(int timeout) ZBX_HISTORY_WRITE_CBS zbx_module_history_write_cbs(void) int zbx_module_uninit(void)
15. returning a value #define SET_UI64_RESULT(res,val) #define SET_DBL_RESULT(res, val) #define SET_STR_RESULT(res, val) /* xx */ #define SET_TEXT_RESULT(res,val) /* xx */ #define SET_LOG_RESULT(res, val) /* xx */ #define SET_MSG_RESULT(res, val) /* xx */ /* xx: ALWAYS allocate new memory for val! * DON'T USE STATIC OR STACK MEMORY!!! */
16. our C module #include "module.h" /* return the version number of this module interface. */ int zbx_module_api_version(void) { return (ZBX_MODULE_API_VERSION); }
17. initialize /* agent start-up; Zabbix will fail to load if we return ZBX_MODULE_FAIL */ int zbx_module_init(void) { if ((db = db_open()) == NULL) { return (ZBX_MODULE_FAIL); } return (ZBX_MODULE_OK); } /* agent is shutting down */ int zbx_module_uninit(void) { db_close(db); return (ZBX_MODULE_OK); }
18. key to function map static ZBX_METRIC item_keys[] = { /* ITEM.KEY FLAG FUNCTION TEST PARAMETERS */ { "owntracks.ping", 0, owntracks_ping, NULL}, { "owntracks.lld", 0, owntracks_lld, NULL}, { "owntracks.vel", CF_HAVEPARAMS, owntracks_vel, "1234567890012345"}, { NULL } }; /* return list of item keys supported by this module */ ZBX_METRIC *zbx_module_item_list(void) { return (item_keys); }
19. owntracks.ping $ zabbix_get -s 127.0.0.1 -k owntracks.ping 42 /* 42. what else? */ static int owntracks_ping(AGENT_REQUEST *request, AGENT_RESULT *result) { SET_UI64_RESULT(result, 42); return (SYSINFO_RET_OK); }
20. frontend items If our module is loaded into a server, the item type must be Simple Check. If our module is loaded into an agent, the item type must be Zabbix agent or Zabbix agent (active).
21. programatically create item from zabbix_api import ZabbixAPI # https://pypi.org/project/zabbix-api/ import json zapi = ZabbixAPI(server="http://server/zabbix/") zapi.login("admin", "secret") res = zapi.item.create({ "name" : "The pong", "key_" : "owntracks.ping", "hostid" : 10268, "interfaceid" : "11", "type" : 0, # Zabbix agent "value_type" : 3, # numeric, unsigned "delay" : "60s", })
22. owntracks.vel static int owntracks_vel(AGENT_REQUEST *request, AGENT_RESULT *result) { unsigned long retval; if (request->nparam != 1) { SET_MSG_RESULT(result, strdup("Expecting 1 parameter with IMEI")); return (SYSINFO_RET_FAIL); } retval = db_get(db, get_rparam(request, 0)); SET_UI64_RESULT(result, retval); return (SYSINFO_RET_OK); }
23. low-level discovery
24. What is LLD? Low-Level Discovery, widely used, built-in methods* can create items, triggers, graphs, custom LLD, send JSON to Zabbix server LLD rules discover entities, not item values * network interfaces, CPUs, SNMP OIDs, JMX objects, ODBC queries, Windows services, file systems, host interfaces
25. how do we use LLD? (1)
26. how do we use LLD? (2) What does [process] do? • subscribe to MQTT • extract OwnTracks' payload • update device database (store values) => LLD • device not seen or dead, i.e. TTL expired?
 remove from database => LLD • device details (e.g. speed) add to database: owntracks.vel() reads it • spikes trigger Zabbix alerts • 120 lines of Python
27. how do we use LLD? (3) Why LMDB? • in-process • blindingly fast • single writer (our [process]) • multiple readers (functions in our loadable C module) • no external moving parts • multiple language bindings (C, Python, etc…)
28. OwnTracks payloads { "_type": "location", "tst": 1553542948, "lat": 59.373399, "lon": 17.94669, "cog": 280, "alt": 101, "vel": 15, "tid": "xq", "batt": 89, "name": "20-ER-20" } https://owntracks.org/booklet/tech/json/ owntracks/zbx/012549656802107 topic payload
29. defining LLD Define discovery rule. Text item receives JSON. Prototypes create items and triggers.
30. trapper item $ zabbix_sender -z 192.168.1.53 -s AirJP -k my.item -o '{ "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }'
31. LLD JSON format $ zabbix_get -s 127.0.0.1 -k owntracks.lld { "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }
32. owntracks.lld static int owntracks_lld(AGENT_REQUEST *request, AGENT_RESULT *result) { JsonNode *obj = json_mkobject(), *list = json_mkarray(), *o; char *js; while ((o = db_getnext(db)) != NULL) { json_append_element(list, o); } json_append_member(obj, "data", list); if ((js = json_stringify(obj, " ")) == NULL) { SET_MSG_RESULT(result, strdup("Cannot stringify JSON.")); return (SYSINFO_RET_FAIL); } SET_STR_RESULT(result, js); /* Zabbix will free() this */ json_delete(obj); return (SYSINFO_RET_OK); }
33. discovery rule
34. item prototypes { "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }
35. trigger prototypes {airjp:owntracks.vel[{#OT.IMEI}].last()}>90
36. LLD in action
37. latest data
38. stale data auto expiry
39. graph prototypes
40. history export (1) Server only, quite complicated, invoked at server start and then occasionally, badly documented, replaceable with real-time export into JSON files.
41. static void history_cb_integer(const ZBX_HISTORY_INTEGER *history, int history_count) { int n; itemid, val, clock; for (n = 0; n < history_count; n++) { itemid = history[n].itemid; clock = history[n].clock; val = history[n].value; ... } } ZBX_HISTORY_WRITE_CBS zbx_module_history_write_cbs(void) { static ZBX_HISTORY_WRITE_CBS my_callbacks = { NULL, /* my_history_float_cb */ history_cb_integer, /* my_history_integer_cb */ history_cb_string, /* my_history_string_cb */ NULL, /* my_history_text_cb */ NULL, /* my_history_log_cb */ }; return (my_callbacks); }
42. history export (2) { "host": "airjp", "groups": [ "Linux servers" ], "applications": [], "itemid": 28530, "name": "jjolie.cn", "clock": 1553636580, "ns": 518541363, "value": "Jane Jolie" } { "host": "Zabbix server", "groups": [ "Zabbix servers" ], "applications": [ "Filesystems" ], "itemid": 28506, "name": "Free disk space on / (percentage)", "clock": 1553636586, "ns": 523296311, "value": 70.684192 }
43. low-level recipe write and compile module, edit configuration, restart agent, server, or proxy, create items, test, enjoy
44. buywhy.gif C functions are fast. LLD is highly flexible. Together this is a powerful combination. Unfortunately the API has no support for reading zabbix.conf
 Documentation is minimal.
45. zabbix.com @zabbix