Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Zabbix Low-Level Discovery (LLD) from a C module

Jan-Piet Mens
November 05, 2019

Zabbix Low-Level Discovery (LLD) from a C module

Using a real-life practical example which we use to monitor vehicles issued with GPS trackers which communicate via MQTT, we will discuss how we implement Zabbix Low-Level Discovery directly from a C module and how the same C module is used to provide up-to-date information from the vehicles to Zabbix items. This basic principle can easily be adapted to provide similar functionility to Internet of Things (IoT) projects

Jan-Piet Mens

November 05, 2019
Tweet

More Decks by Jan-Piet Mens

Other Decks in Technology

Transcript

  1. Zabbix loadable C modules and Low-Level Discovery (LLD) Jan-Piet Mens

    OSMC, November 2019 @jpmens
  2. @jpmens: consultant, part-time admin, trainer, small-scale fiddler, loves plain text,

    and things which work. Contributes to Ansible, dreamed up OwnTracks, and chases bugs in open source DNS servers.
  3. This is OwnTracks

  4. Zabbix cannot, out of the box, support every possible thing

    you may want to monitor; let's extend it. extensible
  5. extending Zabbix system.run[] User Parameters External checks Loadable modules Beware

    fork(), invalid results: unsupported items
  6. system.run[] # agent EnableRemoteCommands=1 LogRemoteCommands=1 76775:20190326:173402.315 Executing command '/usr/local/bin/zdisc.sh one

    two three' #!/bin/sh echo $# Runs on agent
  7. user parameters #!/bin/sh echo 16 exit 0 # agent
 UserParameter=jjolie.cn[*],echo

    "$1 Jolie" UserParameter=jjolie.uid,/usr/local/bin/uidnumber.sh Runs on agent
  8. external checks Runs on server # server
 ExternalScripts=/usr/lib/zabbix/externalscripts

  9. loadable modules Written in the C language, compiled, loaded dynamically

    by server, proxy, and/or agent
  10. what can modules do? Basically anything: sockets, System V IPC,

    files, databases, IoT, sensors, … Beware permissions, context, errors
  11. custom module # server, proxy, agent LoadModulePath=/usr/lib/zabbix/agent
 LoadModule=owntracks-module.so
 LoadModule=two.so

  12. make install Build inside / outside source tree owntracks-module.so: owntracks-module.c

    gcc -fPIC -shared -o owntracks-module.so owntracks-module.c install owntracks-module.so /usr/lib/zabbix/agent/
  13. module.h --- zabbix-4.0.5/include/module.h 2019-02-25 10:36:39.000000000 +0100 +++ module.h 2019-03-26 15:03:59.000000000

    +0100 @@ -20,7 +20,8 @@ #ifndef ZABBIX_MODULE_H #define ZABBIX_MODULE_H -#include "zbxtypes.h" +#include <inttypes.h> +#define zbx_uint64_t uint64_t #define ZBX_MODULE_OK 0 #define ZBX_MODULE_FAIL -1
  14. the API int zbx_module_api_version(void) int zbx_module_init(void) ZBX_METRIC *zbx_module_item_list(void) void zbx_module_item_timeout(int

    timeout) ZBX_HISTORY_WRITE_CBS zbx_module_history_write_cbs(void) int zbx_module_uninit(void)
  15. returning a value #define SET_UI64_RESULT(res,val) #define SET_DBL_RESULT(res, val) #define SET_STR_RESULT(res,

    val) /* xx */ #define SET_TEXT_RESULT(res,val) /* xx */ #define SET_LOG_RESULT(res, val) /* xx */ #define SET_MSG_RESULT(res, val) /* xx */ /* xx: ALWAYS allocate new memory for val! * DON'T USE STATIC OR STACK MEMORY!!! */
  16. our C module #include "module.h" /* return the version number

    of this module interface. */ int zbx_module_api_version(void) { return (ZBX_MODULE_API_VERSION); }
  17. initialize /* agent start-up; Zabbix will fail to load if

    we return ZBX_MODULE_FAIL */ int zbx_module_init(void) { if ((db = db_open()) == NULL) { return (ZBX_MODULE_FAIL); } return (ZBX_MODULE_OK); } /* agent is shutting down */ int zbx_module_uninit(void) { db_close(db); return (ZBX_MODULE_OK); }
  18. key to function map static ZBX_METRIC item_keys[] = { /*

    ITEM.KEY FLAG FUNCTION TEST PARAMETERS */ { "owntracks.ping", 0, owntracks_ping, NULL}, { "owntracks.lld", 0, owntracks_lld, NULL}, { "owntracks.vel", CF_HAVEPARAMS, owntracks_vel, "1234567890012345"}, { NULL } }; /* return list of item keys supported by this module */ ZBX_METRIC *zbx_module_item_list(void) { return (item_keys); }
  19. owntracks.ping $ zabbix_get -s 127.0.0.1 -k owntracks.ping 42 /* 42.

    what else? */ static int owntracks_ping(AGENT_REQUEST *request, AGENT_RESULT *result) { SET_UI64_RESULT(result, 42); return (SYSINFO_RET_OK); }
  20. frontend items If our module is loaded into a server,

    the item type must be Simple Check. If our module is loaded into an agent, the item type must be Zabbix agent or Zabbix agent (active).
  21. programatically create item from zabbix_api import ZabbixAPI # https://pypi.org/project/zabbix-api/ import

    json zapi = ZabbixAPI(server="http://server/zabbix/") zapi.login("admin", "secret") res = zapi.item.create({ "name" : "The pong", "key_" : "owntracks.ping", "hostid" : 10268, "interfaceid" : "11", "type" : 0, # Zabbix agent "value_type" : 3, # numeric, unsigned "delay" : "60s", })
  22. owntracks.vel static int owntracks_vel(AGENT_REQUEST *request, AGENT_RESULT *result) { unsigned long

    retval; if (request->nparam != 1) { SET_MSG_RESULT(result, strdup("Expecting 1 parameter with IMEI")); return (SYSINFO_RET_FAIL); } retval = db_get(db, get_rparam(request, 0)); SET_UI64_RESULT(result, retval); return (SYSINFO_RET_OK); }
  23. low-level discovery

  24. What is LLD? Low-Level Discovery, widely used, built-in methods* can

    create items, triggers, graphs, custom LLD, send JSON to Zabbix server LLD rules discover entities, not item values * network interfaces, CPUs, SNMP OIDs, JMX objects, ODBC queries, Windows services, file systems, host interfaces
  25. how do we use LLD? (1)

  26. how do we use LLD? (2) What does [process] do?

    • subscribe to MQTT • extract OwnTracks' payload • update device database (store values) => LLD • device not seen or dead, i.e. TTL expired?
 remove from database => LLD • device details (e.g. speed) add to database: owntracks.vel() reads it • spikes trigger Zabbix alerts • 120 lines of Python
  27. how do we use LLD? (3) Why LMDB? • in-process

    • blindingly fast • single writer (our [process]) • multiple readers (functions in our loadable C module) • no external moving parts • multiple language bindings (C, Python, etc…)
  28. OwnTracks payloads { "_type": "location", "tst": 1553542948, "lat": 59.373399, "lon":

    17.94669, "cog": 280, "alt": 101, "vel": 15, "tid": "xq", "batt": 89, "name": "20-ER-20" } https://owntracks.org/booklet/tech/json/ owntracks/zbx/012549656802107 topic payload
  29. defining LLD Define discovery rule. Text item receives JSON. Prototypes

    create items and triggers.
  30. trapper item $ zabbix_sender -z 192.168.1.53 -s AirJP -k my.item

    -o '{ "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }'
  31. LLD JSON format $ zabbix_get -s 127.0.0.1 -k owntracks.lld {

    "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }
  32. owntracks.lld static int owntracks_lld(AGENT_REQUEST *request, AGENT_RESULT *result) { JsonNode *obj

    = json_mkobject(), *list = json_mkarray(), *o; char *js; while ((o = db_getnext(db)) != NULL) { json_append_element(list, o); } json_append_member(obj, "data", list); if ((js = json_stringify(obj, " ")) == NULL) { SET_MSG_RESULT(result, strdup("Cannot stringify JSON.")); return (SYSINFO_RET_FAIL); } SET_STR_RESULT(result, js); /* Zabbix will free() this */ json_delete(obj); return (SYSINFO_RET_OK); }
  33. discovery rule

  34. item prototypes { "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20"

    } ] }
  35. trigger prototypes {airjp:owntracks.vel[{#OT.IMEI}].last()}>90

  36. LLD in action

  37. latest data

  38. stale data auto expiry

  39. graph prototypes

  40. history export (1) Server only, quite complicated, invoked at server

    start and then occasionally, badly documented, replaceable with real-time export into JSON files.
  41. static void history_cb_integer(const ZBX_HISTORY_INTEGER *history, int history_count) { int n;

    itemid, val, clock; for (n = 0; n < history_count; n++) { itemid = history[n].itemid; clock = history[n].clock; val = history[n].value; ... } } ZBX_HISTORY_WRITE_CBS zbx_module_history_write_cbs(void) { static ZBX_HISTORY_WRITE_CBS my_callbacks = { NULL, /* my_history_float_cb */ history_cb_integer, /* my_history_integer_cb */ history_cb_string, /* my_history_string_cb */ NULL, /* my_history_text_cb */ NULL, /* my_history_log_cb */ }; return (my_callbacks); }
  42. history export (2) { "host": "airjp", "groups": [ "Linux servers"

    ], "applications": [], "itemid": 28530, "name": "jjolie.cn", "clock": 1553636580, "ns": 518541363, "value": "Jane Jolie" } { "host": "Zabbix server", "groups": [ "Zabbix servers" ], "applications": [ "Filesystems" ], "itemid": 28506, "name": "Free disk space on / (percentage)", "clock": 1553636586, "ns": 523296311, "value": 70.684192 }
  43. low-level recipe write and compile module, edit configuration, restart agent,

    server, or proxy, create items, test, enjoy
  44. buywhy.gif C functions are fast. LLD is highly flexible. Together

    this is a powerful combination. Unfortunately the API has no support for reading zabbix.conf
 Documentation is minimal.
  45. zabbix.com @zabbix