Python as a tool to boost productivity in (electronic) product and system development. by Johan Hartman

The general take away from this talk will be to motivate the use of Python not only as the implementation language to develop your application, but also to use Python to develop your own automation tools that fit your development process. By doing this, you will be able to implement faster, more accurate, have a better tested result and along the way derive many benefits that you won’t foresee when you start.

Although my talk will speak from an embedded engineering point of view and not an IT or web development point of view, I belief the principles applies generally. Over many years and through the life cycle of many products I have evolved a development methodology that uses Python as the language of choice to develop my own tools for code generation (for ‘C’ and Python), software and firmware test automation, systems compliance testing and hardware manufacturing test automation.

My process is that I start a project by defining XML object definitions for all instances of configuration, data or communication objects. From the XML, on the embedded firmware side, I generate ‘C’ structures, enum’s and prototype function bodies to use in my coding. On the PC or server side, I generate Python data object definitions from the XML. These are imported into Python programs that are written to operate on the object definitions, giving me free reuse of all the interfacing, testing, visualization etc. code that I have developed in the past.

Transcript

1. Python as a tool to boost productivity in (electronic) product and system development by Johan Hartman. PyConZA2018 Johan Hartman 1 of 36
2. Introduction Linkedin: https://www.linkedin.com/in/johan-hartman-pretoria/ Personal web page: https://jeghartman.wixsite.com/electronicsengineer PyConZA2018 Johan Hartman 2 of 36
3. My career started here, Me, playing with / in dirt. PyConZA2018 Johan Hartman 3 of 36
4. Still playing with dirt, maybe a cleaner form – Silicon Because over the years we have figured out how to ‘dope’ Silicon to get it to do math and other interesting stuff. Attribution: https://siliconpr0n.org/ map/atmel/atmega8/single/ atmel_atmega8_mz.jpg PyConZA2018 Johan Hartman 4 of 36
5. Since its the graveyard shift, lets take a quick picture zoom into a chip. Attribution: http://siliconpr0n.org/archive/doku.php?id=anon:ncr:81489 PyConZA2018 Johan Hartman 5 of 36
6. Let’s zoom in on the embedded processor. Attribution: http://siliconpr0n.org/archive/doku.php?id=anon:ncr:81489 PyConZA2018 Johan Hartman 6 of 36
7. XRAY of the same Chip Attribution: http://siliconpr0n.org/archive/doku.php?id=anon:ncr:81489 PyConZA2018 Johan Hartman 7 of 36
8. Microscope Photo of a portion of the chip silicon. Attribution: http://siliconpr0n.org/archive/doku.php?id=anon:ncr:81489 PyConZA2018 Johan Hartman 8 of 36
9. This is what I spent 80% of my time doing. PyConZA2018 Johan Hartman 9 of 36
10. Simplified firmware design methodology. Model View Controller. PyConZA2018 Johan Hartman 10 of 36 ADC SENSOR INPUT Interrupt Action 1 Action 2 Controllers View Model OBJECTS
11. Object definitions in XML <msgobject name="msgcfg_a2d_cal" msgID="5" dir="RxTx" instances="1" setup="False" description="ADC control-conversion structure"> <field type="enum" name="adc_cal" display="dec" edit="True" units="index" elements="1" defval="0" minval="0" maxval="7" description="ADC channel instance" options="'ADC_TEMP','ADC_3V3','ADC_VIN','ADC_4V8','ADC_Vpos','ADC_NTC','ADC_VBATT'"/> <field type="enum" name="valid" display="dec" edit="True" units="enum" elements="1" defval="0" minval="0" maxval="2" description="Indicate table validity" options="'False', 'True', 'Defaults'"/> <field type="int8u" name="chanidx" display="dec" edit="False" units="index" elements="1" defval="0" minval="0" maxval="255" description="ADC channel input index 0..15"/> <field type="str" name="units" display="str" edit="True" units="Text" elements="2" defval="mV" minval="0" maxval="2" description="channel value units" options="'mV'"/> <field type="int16u" name="adc_raw_low" display="dec" edit="True" units="counts" elements="1" defval="0" minval="0" maxval="65535" description="ADC raw conversion low point"/> <field type="int16u" name="adc_cal_low" display="dec" edit="True" units="mVolt" elements="1" defval="0" minval="0" maxval="65535" description="ADC calibrated low point"/> <field type="int16u" name="adc_raw_high" display="dec" edit="True" units="counts" elements="1" defval="0" minval="0" maxval="65535" description="ADC raw conversion high point"/> <field type="int16u" name="adc_cal_high" display="dec" edit="True" units="mVolt" elements="1" defval="0" minval="0" maxval="65535" description="ADC calibrated high point"/> <field type="int16u" name="crc" display="hex" edit="False" units="none" elements="1" defval="0" minval="0" maxval="65535" description="CRC of object"/> </msgobject> PyConZA2018 Johan Hartman 11 of 36
12. ‘C’ namespace definitions, ie Enum’s typedef enum _E_ADC_CAL { ADC_CAL_ADC_TEMP, ADC_CAL_ADC_3V3, ADC_CAL_ADC_VIN, ADC_CAL_ADC_4V8, ADC_CAL_ADC_VPOS, ADC_CAL_ADC_NTC, ADC_CAL_ADC_VBATT, } T_E_ADC_CAL; PyConZA2018 Johan Hartman 12 of 36
13. ‘C’ namespace definitions, Structure //----- PROT_FRAME_MSGCFG_A2D_CAL 5 //----- ADC control-conversion structure typedef struct __attribute__((__packed__)) _MSGCFG_A2D_CAL { int8u adc_cal; // ADC channel instance int8u valid; // Indicate table validity int8u chanidx; // ADC channel input index 0..15 int8u units[2]; // channel value units int16u adc_raw_low; // ADC raw conversion low point int16u adc_cal_low; // ADC calibrated low point int16u adc_raw_high; // ADC raw conversion high point int16u adc_cal_high; // ADC calibrated high point int16u crc; // CRC of object } T_MSGCFG_A2D_CAL, *pT_MSGCFG_A2D_CAL; PyConZA2018 Johan Hartman 13 of 36
14. ‘C’ function prototype definitions, ie Messages extern int8u FRAME_Rx_frame_msgcfg_a2d_cal (int8u* data, int8u length); // MsgID = 5 extern int8u FRAME_Tx_frame_msgcfg_a2d_cal (int8u* data, int8u length); // MsgID = 5 ‘C’ function template definition, ie Message receive ‘C’ function template definition, ie Message transmit PyConZA2018 Johan Hartman 14 of 36
15. //------------------------------------------------------------------------------------- int8u FRAME_Rx_frame_msgcfg_a2d_cal (int8u* data, int8u length) { // MsgID = 5 int8u status; pT_MSGCFG_A2D_CAL pmsg; status = FALSE; pmsg = (pT_MSGCFG_A2D_CAL)(data); //. = pmsg->adc_cal; // ADC channel instance //. = pmsg->valid; // Indicate table validity //. = pmsg->chanidx; // ADC channel input index 0..15 //. = pmsg->units[2]; // channel value units //. = pmsg->adc_raw_low; // ADC raw conversion low point //. = pmsg->adc_cal_low; // ADC calibrated low point //. = pmsg->adc_raw_high; // ADC raw conversion high point //. = pmsg->adc_cal_high; // ADC calibrated high point //. = pmsg->crc; // CRC of object if (length == sizeof(T_MSGCFG_A2D_CAL)) { // message length correct } else { ERR_Sprint_P(ERR_LVL_DEBUG, PSTR("::msg illegal length")); } PyConZA2018 Johan Hartman 15 of 36
16. return status; } PyConZA2018 Johan Hartman 16 of 36
17. //------------------------------------------------------------------------------------- int8u FRAME_Tx_frame_msgcfg_a2d_cal (int8u* data, int8u length) { // MsgID = 5 int8u status; pT_MEMP_BLOCK pmemblock; pT_MSGCFG_A2D_CAL pmsg; status = FALSE; pmemblock = NULL; //pmsg = (pT_MSGCFG_A2D_CAL)(data); //pmsg->adc_cal = .; // ADC channel instance //pmsg->valid = .; // Indicate table validity //pmsg->chanidx = .; // ADC channel input index 0..15 //pmsg->units[2] = .; // channel value units //pmsg->adc_raw_low = .; // ADC raw conversion low point //pmsg->adc_cal_low = .; // ADC calibrated low point //pmsg->adc_raw_high = .; // ADC raw conversion high point //pmsg->adc_cal_high = .; // ADC calibrated high point //pmsg->crc = .; // CRC of object if (length > sizeof(T_MSGCFG_A2D_CAL)) { // data length Incorrect ERR_Sprint_P(ERR_LVL_ERROR, PSTR("::Abort, data length")); return FALSE; } PyConZA2018 Johan Hartman 17 of 36
18. else { // Get memblock //pmemblock = MEMP_Get_Block( (length..), MEMP_TASKID_..); pmemblock = NULL; if (pmemblock != NULL) { // Build msg // status = TRUE; } } return status; } PyConZA2018 Johan Hartman 18 of 36
19. Python Object definition #----- object name = PROT_FRAME::MSGCFG_A2D_CAL 5 {'protocol_name' : 'FRAME', 'protocol_ID' : '1', 'message_name' : 'msgcfg_a2d_cal', 'message_id' : '5', 'message_direction' : 'RxTx', # From embedded device point of view 'elements' : [ [ 0, " enum", " adc_cal", "dec", 1, 0, 7, "index", "ADC channel instance", ('ADC_TEMP','ADC_3V3','ADC_VIN','ADC_4V8','ADC_Vpos','ADC_NTC','ADC_VBATT'), True], [ 0, " enum", " valid", "dec", 1, 0, 2, "enum", "Indicate table validity", ('False', 'True', 'Defaults'), True], [ 0, " int8u", " chanidx", "dec", 1, 0, 255, "index", "ADC channel input index 0..15", None, False], [" mV", " str", " units", "str", 2, 0, 2, "Text", "channel value units", ('mV'), True], [ 0, " int16u", " adc_raw_low", "dec", 1, 0, 65535, "counts", "ADC raw conversion low point", None, True], [ 0, " int16u", " adc_cal_low", "dec", 1, 0, 65535, "mVolt", "ADC calibrated low point", None, True], [ 0, " int16u", " adc_raw_high", "dec", 1, 0, 65535, "counts", "ADC raw conversion high point", None, True], [ 0, " int16u", " adc_cal_high", "dec", 1, 0, 65535, "mVolt", "ADC calibrated high point", None, True], [ 0, " int16u", " crc", "hex", 1, 0, 65535, "none", "CRC of object", None, False], PyConZA2018 Johan Hartman 19 of 36
20. ], # end of elements 'element_count' : 9, 'format_str' : '<BBB2sHHHHH', 'rx_time' : None, # store last used instance in a list here 'rx_instance' : None, # store last used instance in a list here 'disp_instance' : None }, # store last display manipulated instance in a list here #-----end of object name = PROT_FRAME::MSGCFG_A2D_CAL 5 PyConZA2018 Johan Hartman 20 of 36
21. Auto Generated Python Object Display Screen PyConZA2018 Johan Hartman 21 of 36
22. Auto Generated Display Screen – Dropdown Lists PyConZA2018 Johan Hartman 22 of 36
23. Auto Generated Display Screen – Named Bitfields PyConZA2018 Johan Hartman 23 of 36
24. Auto Generated Display Screen – Setting configuration PyConZA2018 Johan Hartman 24 of 36
25. Message Encoding – Message Logger PyConZA2018 Johan Hartman 25 of 36
26. Message Encoding – Binary Data on Serial Port PyConZA2018 Johan Hartman 26 of 36
27. Draw Graphs with MatPlotLib PyConZA2018 Johan Hartman 27 of 36
28. Run a TCP Server Using the same message En/Decoder Using Asyncio PyConZA2018 Johan Hartman 28 of 36
29. Standard Program Configuration PyConZA2018 Johan Hartman 29 of 36
30. Large Projects – Save tons of work. PyConZA2018 Johan Hartman 30 of 36
31. Scripting – Message select PyConZA2018 Johan Hartman 31 of 36
32. Scripting – Message Setup PyConZA2018 Johan Hartman 32 of 36
33. Scripting – Action Setup PyConZA2018 Johan Hartman 33 of 36
34. Software / Factory Testing PyConZA2018 Johan Hartman 34 of 36
35. In Summary • Which parts of your work is repetitive and structured? • Database tables • Data capturing screens for the database tables • Report generation from the database tables • Web site generator Some of these things are basically the same thing over and over again? Can you generate some, or a lot of the code from some form of data definition? I want to encourage you: Build your own tools to speed up your work, simplify your work, make your final product more robust against human coding errors. PyConZA2018 Johan Hartman 35 of 36
36. PyConZA2018 Johan Hartman 36 of 36