Python as a tool to boost productivity in (electronic) product and system development. by Johan Hartman

7b0645f018c0bddc8ce3900ccc3ba70c?s=47 Pycon ZA
October 12, 2018

Python as a tool to boost productivity in (electronic) product and system development. by Johan Hartman

The general take away from this talk will be to motivate the use of Python not only as the implementation language to develop your application, but also to use Python to develop your own automation tools that fit your development process. By doing this, you will be able to implement faster, more accurate, have a better tested result and along the way derive many benefits that you won’t foresee when you start.

Although my talk will speak from an embedded engineering point of view and not an IT or web development point of view, I belief the principles applies generally. Over many years and through the life cycle of many products I have evolved a development methodology that uses Python as the language of choice to develop my own tools for code generation (for ‘C’ and Python), software and firmware test automation, systems compliance testing and hardware manufacturing test automation.

My process is that I start a project by defining XML object definitions for all instances of configuration, data or communication objects. From the XML, on the embedded firmware side, I generate ‘C’ structures, enum’s and prototype function bodies to use in my coding. On the PC or server side, I generate Python data object definitions from the XML. These are imported into Python programs that are written to operate on the object definitions, giving me free reuse of all the interfacing, testing, visualization etc. code that I have developed in the past.

7b0645f018c0bddc8ce3900ccc3ba70c?s=128

Pycon ZA

October 12, 2018
Tweet

Transcript

  1. Python as a tool to boost productivity in (electronic) product

    and system development by Johan Hartman. PyConZA2018 Johan Hartman 1 of 36
  2. Introduction Linkedin: https://www.linkedin.com/in/johan-hartman-pretoria/ Personal web page: https://jeghartman.wixsite.com/electronicsengineer PyConZA2018 Johan Hartman

    2 of 36
  3. My career started here, Me, playing with / in dirt.

    PyConZA2018 Johan Hartman 3 of 36
  4. Still playing with dirt, maybe a cleaner form – Silicon

    Because over the years we have figured out how to ‘dope’ Silicon to get it to do math and other interesting stuff. Attribution: https://siliconpr0n.org/ map/atmel/atmega8/single/ atmel_atmega8_mz.jpg PyConZA2018 Johan Hartman 4 of 36
  5. Since its the graveyard shift, lets take a quick picture

    zoom into a chip. Attribution: http://siliconpr0n.org/archive/doku.php?id=anon:ncr:81489 PyConZA2018 Johan Hartman 5 of 36
  6. Let’s zoom in on the embedded processor. Attribution: http://siliconpr0n.org/archive/doku.php?id=anon:ncr:81489 PyConZA2018

    Johan Hartman 6 of 36
  7. XRAY of the same Chip Attribution: http://siliconpr0n.org/archive/doku.php?id=anon:ncr:81489 PyConZA2018 Johan Hartman

    7 of 36
  8. Microscope Photo of a portion of the chip silicon. Attribution:

    http://siliconpr0n.org/archive/doku.php?id=anon:ncr:81489 PyConZA2018 Johan Hartman 8 of 36
  9. This is what I spent 80% of my time doing.

    PyConZA2018 Johan Hartman 9 of 36
  10. Simplified firmware design methodology. Model View Controller. PyConZA2018 Johan Hartman

    10 of 36 ADC SENSOR INPUT Interrupt Action 1 Action 2 Controllers View Model OBJECTS
  11. Object definitions in XML <msgobject name="msgcfg_a2d_cal" msgID="5" dir="RxTx" instances="1" setup="False"

    description="ADC control-conversion structure"> <field type="enum" name="adc_cal" display="dec" edit="True" units="index" elements="1" defval="0" minval="0" maxval="7" description="ADC channel instance" options="'ADC_TEMP','ADC_3V3','ADC_VIN','ADC_4V8','ADC_Vpos','ADC_NTC','ADC_VBATT'"/> <field type="enum" name="valid" display="dec" edit="True" units="enum" elements="1" defval="0" minval="0" maxval="2" description="Indicate table validity" options="'False', 'True', 'Defaults'"/> <field type="int8u" name="chanidx" display="dec" edit="False" units="index" elements="1" defval="0" minval="0" maxval="255" description="ADC channel input index 0..15"/> <field type="str" name="units" display="str" edit="True" units="Text" elements="2" defval="mV" minval="0" maxval="2" description="channel value units" options="'mV'"/> <field type="int16u" name="adc_raw_low" display="dec" edit="True" units="counts" elements="1" defval="0" minval="0" maxval="65535" description="ADC raw conversion low point"/> <field type="int16u" name="adc_cal_low" display="dec" edit="True" units="mVolt" elements="1" defval="0" minval="0" maxval="65535" description="ADC calibrated low point"/> <field type="int16u" name="adc_raw_high" display="dec" edit="True" units="counts" elements="1" defval="0" minval="0" maxval="65535" description="ADC raw conversion high point"/> <field type="int16u" name="adc_cal_high" display="dec" edit="True" units="mVolt" elements="1" defval="0" minval="0" maxval="65535" description="ADC calibrated high point"/> <field type="int16u" name="crc" display="hex" edit="False" units="none" elements="1" defval="0" minval="0" maxval="65535" description="CRC of object"/> </msgobject> PyConZA2018 Johan Hartman 11 of 36
  12. ‘C’ namespace definitions, ie Enum’s typedef enum _E_ADC_CAL { ADC_CAL_ADC_TEMP,

    ADC_CAL_ADC_3V3, ADC_CAL_ADC_VIN, ADC_CAL_ADC_4V8, ADC_CAL_ADC_VPOS, ADC_CAL_ADC_NTC, ADC_CAL_ADC_VBATT, } T_E_ADC_CAL; PyConZA2018 Johan Hartman 12 of 36
  13. ‘C’ namespace definitions, Structure //----- PROT_FRAME_MSGCFG_A2D_CAL 5 //----- ADC control-conversion

    structure typedef struct __attribute__((__packed__)) _MSGCFG_A2D_CAL { int8u adc_cal; // ADC channel instance int8u valid; // Indicate table validity int8u chanidx; // ADC channel input index 0..15 int8u units[2]; // channel value units int16u adc_raw_low; // ADC raw conversion low point int16u adc_cal_low; // ADC calibrated low point int16u adc_raw_high; // ADC raw conversion high point int16u adc_cal_high; // ADC calibrated high point int16u crc; // CRC of object } T_MSGCFG_A2D_CAL, *pT_MSGCFG_A2D_CAL; PyConZA2018 Johan Hartman 13 of 36
  14. ‘C’ function prototype definitions, ie Messages extern int8u FRAME_Rx_frame_msgcfg_a2d_cal (int8u*

    data, int8u length); // MsgID = 5 extern int8u FRAME_Tx_frame_msgcfg_a2d_cal (int8u* data, int8u length); // MsgID = 5 ‘C’ function template definition, ie Message receive ‘C’ function template definition, ie Message transmit PyConZA2018 Johan Hartman 14 of 36
  15. //------------------------------------------------------------------------------------- int8u FRAME_Rx_frame_msgcfg_a2d_cal (int8u* data, int8u length) { // MsgID

    = 5 int8u status; pT_MSGCFG_A2D_CAL pmsg; status = FALSE; pmsg = (pT_MSGCFG_A2D_CAL)(data); //. = pmsg->adc_cal; // ADC channel instance //. = pmsg->valid; // Indicate table validity //. = pmsg->chanidx; // ADC channel input index 0..15 //. = pmsg->units[2]; // channel value units //. = pmsg->adc_raw_low; // ADC raw conversion low point //. = pmsg->adc_cal_low; // ADC calibrated low point //. = pmsg->adc_raw_high; // ADC raw conversion high point //. = pmsg->adc_cal_high; // ADC calibrated high point //. = pmsg->crc; // CRC of object if (length == sizeof(T_MSGCFG_A2D_CAL)) { // message length correct } else { ERR_Sprint_P(ERR_LVL_DEBUG, PSTR("::msg illegal length")); } PyConZA2018 Johan Hartman 15 of 36
  16. return status; } PyConZA2018 Johan Hartman 16 of 36

  17. //------------------------------------------------------------------------------------- int8u FRAME_Tx_frame_msgcfg_a2d_cal (int8u* data, int8u length) { // MsgID

    = 5 int8u status; pT_MEMP_BLOCK pmemblock; pT_MSGCFG_A2D_CAL pmsg; status = FALSE; pmemblock = NULL; //pmsg = (pT_MSGCFG_A2D_CAL)(data); //pmsg->adc_cal = .; // ADC channel instance //pmsg->valid = .; // Indicate table validity //pmsg->chanidx = .; // ADC channel input index 0..15 //pmsg->units[2] = .; // channel value units //pmsg->adc_raw_low = .; // ADC raw conversion low point //pmsg->adc_cal_low = .; // ADC calibrated low point //pmsg->adc_raw_high = .; // ADC raw conversion high point //pmsg->adc_cal_high = .; // ADC calibrated high point //pmsg->crc = .; // CRC of object if (length > sizeof(T_MSGCFG_A2D_CAL)) { // data length Incorrect ERR_Sprint_P(ERR_LVL_ERROR, PSTR("::Abort, data length")); return FALSE; } PyConZA2018 Johan Hartman 17 of 36
  18. else { // Get memblock //pmemblock = MEMP_Get_Block( (length..), MEMP_TASKID_..);

    pmemblock = NULL; if (pmemblock != NULL) { // Build msg // status = TRUE; } } return status; } PyConZA2018 Johan Hartman 18 of 36
  19. Python Object definition #----- object name = PROT_FRAME::MSGCFG_A2D_CAL 5 {'protocol_name'

    : 'FRAME', 'protocol_ID' : '1', 'message_name' : 'msgcfg_a2d_cal', 'message_id' : '5', 'message_direction' : 'RxTx', # From embedded device point of view 'elements' : [ [ 0, " enum", " adc_cal", "dec", 1, 0, 7, "index", "ADC channel instance", ('ADC_TEMP','ADC_3V3','ADC_VIN','ADC_4V8','ADC_Vpos','ADC_NTC','ADC_VBATT'), True], [ 0, " enum", " valid", "dec", 1, 0, 2, "enum", "Indicate table validity", ('False', 'True', 'Defaults'), True], [ 0, " int8u", " chanidx", "dec", 1, 0, 255, "index", "ADC channel input index 0..15", None, False], [" mV", " str", " units", "str", 2, 0, 2, "Text", "channel value units", ('mV'), True], [ 0, " int16u", " adc_raw_low", "dec", 1, 0, 65535, "counts", "ADC raw conversion low point", None, True], [ 0, " int16u", " adc_cal_low", "dec", 1, 0, 65535, "mVolt", "ADC calibrated low point", None, True], [ 0, " int16u", " adc_raw_high", "dec", 1, 0, 65535, "counts", "ADC raw conversion high point", None, True], [ 0, " int16u", " adc_cal_high", "dec", 1, 0, 65535, "mVolt", "ADC calibrated high point", None, True], [ 0, " int16u", " crc", "hex", 1, 0, 65535, "none", "CRC of object", None, False], PyConZA2018 Johan Hartman 19 of 36
  20. ], # end of elements 'element_count' : 9, 'format_str' :

    '<BBB2sHHHHH', 'rx_time' : None, # store last used instance in a list here 'rx_instance' : None, # store last used instance in a list here 'disp_instance' : None }, # store last display manipulated instance in a list here #-----end of object name = PROT_FRAME::MSGCFG_A2D_CAL 5 PyConZA2018 Johan Hartman 20 of 36
  21. Auto Generated Python Object Display Screen PyConZA2018 Johan Hartman 21

    of 36
  22. Auto Generated Display Screen – Dropdown Lists PyConZA2018 Johan Hartman

    22 of 36
  23. Auto Generated Display Screen – Named Bitfields PyConZA2018 Johan Hartman

    23 of 36
  24. Auto Generated Display Screen – Setting configuration PyConZA2018 Johan Hartman

    24 of 36
  25. Message Encoding – Message Logger PyConZA2018 Johan Hartman 25 of

    36
  26. Message Encoding – Binary Data on Serial Port PyConZA2018 Johan

    Hartman 26 of 36
  27. Draw Graphs with MatPlotLib PyConZA2018 Johan Hartman 27 of 36

  28. Run a TCP Server Using the same message En/Decoder Using

    Asyncio PyConZA2018 Johan Hartman 28 of 36
  29. Standard Program Configuration PyConZA2018 Johan Hartman 29 of 36

  30. Large Projects – Save tons of work. PyConZA2018 Johan Hartman

    30 of 36
  31. Scripting – Message select PyConZA2018 Johan Hartman 31 of 36

  32. Scripting – Message Setup PyConZA2018 Johan Hartman 32 of 36

  33. Scripting – Action Setup PyConZA2018 Johan Hartman 33 of 36

  34. Software / Factory Testing PyConZA2018 Johan Hartman 34 of 36

  35. In Summary • Which parts of your work is repetitive

    and structured? • Database tables • Data capturing screens for the database tables • Report generation from the database tables • Web site generator Some of these things are basically the same thing over and over again? Can you generate some, or a lot of the code from some form of data definition? I want to encourage you: Build your own tools to speed up your work, simplify your work, make your final product more robust against human coding errors. PyConZA2018 Johan Hartman 35 of 36
  36. PyConZA2018 Johan Hartman 36 of 36