Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Test Execution Automation Framework for Embedded Systems

misawa-san
December 11, 2021

Introduction to Test Execution Automation Framework for Embedded Systems

misawa-san

December 11, 2021
Tweet

More Decks by misawa-san

Other Decks in Programming

Transcript

  1. Embedded Systems
    CI Research Working Group
    presents
    Introduction to
    Test Execution Automation Framework
    for Embedded Systems
    Embedded Systems CI Research Working Group
    Hiroshi Misawa
    1
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    December 11, #STAC2021

    View Slide

  2. 2
    ・Hiroshi Misawa
    ・born in Osaka, Japan
    ・Embedded Software Engineer.
    ・semiconductor company to
    automotive component manufacturer.
    ・JASST(Japan Symposium on Software Testing)
    Tokai's executive committee.
    ・Embedded Systems CI
    Research Working Group
    ・Recent interests
    AWS,Docker,CI
    ・Hobby
    ・Making toys
    ・Ring Fit Adventure(Lv 330)
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    December 11, #STAC2021
    Self introduction

    View Slide

  3. 3
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    December 11, #STAC2021
    Issue#1 Must test while integrated with hardware.
    implementation
    SYS Requirement definition
    SW Requirement definition
    SW architect design
    SW detailed design
    SYS architect design
    SW unit testing
    SW integration testing
    SW system testing
    Sys integration testing
    Sys system testing
    Typical issue: Embedded Systems Testing
    2)only
    software
    1)integration
    of hardware
    Issue#2 Want to test software first, eliminate bugs,
    but don't have general-purpose tools.

    View Slide

  4. 4
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    December 11, #STAC2021
    Issue#3 Difficult to reuse the software test environment.
    Because input & output are different.
    System Under Test
    Test environment
    (SILS)
    sensor actuator
    micom
    Software
    Test environment
    (HILS)
    System Under Test
    (integration of hardware)
    sensor actuator
    System Under Test
    Test environment
    (PILS/S-PILS)
    sensor actuator
    actual simulation
    【legend】
    Test environment 1
    (The HILS test environment)
    Test environment 2
    (The SILS test environment)
    inout=hardware I/F
    (pin1=2.5V, etc)
    inout=software I/F
    (val1=250, etc)
    inout=software I/F
    (val1=250, etc)
    Issue of Testing First in
    Embedded systems
    Test environment 2
    (The SILS test environment)
    1
    1

    View Slide

  5. Issues & Solutions
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 5
    《Issues》
    - Must test while integrated with hardware.
    - Want to test software first, eliminate bugs, but don't have general-purpose tools.
    - Difficult to reuse the software test environment. Because input & output are different.
    《Solutions》
    - Create a SILS environment using open source.
    - To make test scenario to be common in different test environments.
    - Implements(Libraries) hide difference between test environments.

    View Slide

  6. Exterior Lighting System
    Exterior Lighting System
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 6
    environment
    light
    ENGIN
    START
    Ignition
    state
    ECU A ECU B
    voltage
    System Under Test
    1. Parking light is active at night.
    2. Start engine.
    3. From parking light to passing light.
    《test scenario》
    Photoelectric
    sensor
    light
    parking light
    passing light
    light
    light
    com
    (I2C)

    human

    View Slide

  7. Exterior Lighting System
    7
    December 11, #STAC2021 7
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    Exterior
    Lighting ECU
    HILS test environment of Exterior
    Lighting System
    Front
    Spotlights ECU
    com(I2C)
    Photoelectric
    sensor
    actual actual
    actual
    simulation
    Ignition state
    simulation
    parking light
    actual
    passing light
    actual
    light
    voltage
    light
    light
    Voltage
    Voltage
    PWM
    PWM
    simulation(human)
    simulation(human)
    actual
    Robot Framework
    HIL equipment
    test scenario
    Libraries(for HILS)
    environment
    Light
    Photoelectric sensor
    Photoelectric sensor

    View Slide

  8. 1
    1
    Exterior Lighting System
    December 11, #STAC2021
    Exterior Lighting ECU
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    Front Spotlights ECU
    ROS
    TCP/IP
    Photoelectric
    sensor
    simulation
    environment light
    Ignition state
    simulation
    parking light
    passing light
    Photoelectric sensor
    numerical
    value(light)
    num
    (voltage)
    num
    (light)
    num
    (voltage)
    simulation(human)
    Library
    Library
    sim
    Library
    sim
    stub
    source
    code
    actual sim
    stub
    sim
    sim
    stub
    actual sim
    stub
    Library
    Library
    simulation
    Library
    Library
    num
    (voltage)
    source
    code
    num (pwm)
    docker on Amazon Web Service
    wrapper wrapper
    Robot Framework
    test scenario
    Libraries(for SILS)
    num (pwm)
    SIL equipment
    8
    SILS test environment of Exterior
    Lighting System
    simulation
    simulation(human)
    Photoelectric sensor
    num
    (light)

    View Slide

  9. Note: Wrapper
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 9
    wrapper
    Robot
    Framework
    Exterior
    Lighting ECU
    Software
    System
    under test
    Front
    Spotlights
    ECU
    Software
    ECU
    execution
    control
    server
    ECU
    execution
    control
    server
    docker container
    docker container
    docker engine (container type virtualization software)
    Wrapper
    Wrapper
    docker container
    test scenario
    (python)
    read/write
    each ECU’s
    variables
    ECU’s
    communication,
    simulated by
    ROS(Robot OS)

    View Slide

  10. Note: Wrapper
    December 11, #STAC2021 © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    10
    ECU execution control server
    Flask
    (Web server)
    Python
    test scenario
    (python)
    ECU execution control server
    Flask
    (Web server)
    Python
    Exterior
    Lighting
    ECU
    Software
    (Clang)
    Front
    Spotlights
    ECU
    Software
    (Clang)
    SharedMemory
    SharedMemory
    Wrapper
    Wrapper

    View Slide

  11. DEMO VIDEO (HILS test environment)
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 11
    https://www.youtube.com/channel/UCMS_fCjRiB2XjakCR5fGLGQ

    View Slide

  12. December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 12
    https://www.youtube.com/channel/UCMS_fCjRiB2XjakCR5fGLGQ
    DEMO VIDEO (SILS test environment)

    View Slide

  13. pros & cons
    《pros》
    - Can test source code early on embedded systems. (Quality)
    - No need actual test environment. (Quality)
    - Can scale-out test environment. (Quality)
    - Can reuse test scenario. (Cost)
    - May reduce making test environment cost. (Cost)
    《cons》
    - So expensive to use for embedded software engineers.
    - Difficult to maintain libraries.
    - Cannot be used for real-time performance testing.
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 13

    View Slide

  14. Test report (SILS test environment)
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 14
    Test Scenario
    (common to HILS & SILS)

    View Slide

  15. December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 15
    テストケース
    It was not executed
    because of abnormal
    processing.
    Test report (SILS test environment)

    View Slide

  16. How to get this tool
    You can get the complete environment
    with the test case used in the demo from the URL below.
    Test Execution Automation Framework for Embedded Systems
    https://github.com/misawa-san/ExteriorLightSystem
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 16

    View Slide

  17. How to run SILS test environment
    Run the Docker container.
    ■procedure
    1. git clone https://github.com/misawa-san/ExteriorLightSystem.git
    2. cd ExteriorLightSystem
    3. sudo chmod -R 777 .
    4. docker-compose -f docker-compose.yml up -d --build
    5. docker-compose -f docker-compose.yml up -d --build
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 17

    View Slide

  18. Note:
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 18
    SILS test environment
    HILS test environment
    keyword-driven
    testing
    To make test scenario to be
    common in different test
    environments.
    Test Scenario
    (common to HILS & SILS)

    View Slide

  19. Participants wanted
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 19
    Embedded Systems CI Research Working Group is conducting
    research continuous-integration/testing/deployment
    for Embedded Systems.
    If you are interested, please join us.
    It’s held once a month, mainly on Sundays from 13:30 to 17:00.
    Please contact us at the following email address.
    [email protected]

    View Slide

  20. Thank you for your kind attention
    20
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    December 11, #STAC2021

    View Slide