Introduction to Test Execution Automation Framework for Embedded Systems

Transcript

1. Embedded Systems
   CI Research Working Group
   presents
   Introduction to
   Test Execution Automation Framework
   for Embedded Systems
   Embedded Systems CI Research Working Group
   Hiroshi Misawa
   1
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved
   December 11, #STAC2021
   

   View Slide

2. 2
   ・Hiroshi Misawa
   ・born in Osaka, Japan
   ・Embedded Software Engineer.
   ・semiconductor company to
   automotive component manufacturer.
   ・JASST(Japan Symposium on Software Testing)
   Tokai's executive committee.
   ・Embedded Systems CI
   Research Working Group
   ・Recent interests
   AWS,Docker,CI
   ・Hobby
   ・Making toys
   ・Ring Fit Adventure(Lv 330)
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved
   December 11, #STAC2021
   Self introduction
   

   View Slide

3. 3
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved
   December 11, #STAC2021
   Issue#1 Must test while integrated with hardware.
   implementation
   SYS Requirement definition
   SW Requirement definition
   SW architect design
   SW detailed design
   SYS architect design
   SW unit testing
   SW integration testing
   SW system testing
   Sys integration testing
   Sys system testing
   Typical issue: Embedded Systems Testing
   2)only
   software
   1)integration
   of hardware
   Issue#2 Want to test software first, eliminate bugs,
   but don't have general-purpose tools.
   

   View Slide

4. 4
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved
   December 11, #STAC2021
   Issue#3 Difficult to reuse the software test environment.
   Because input & output are different.
   System Under Test
   Test environment
   (SILS)
   sensor actuator
   micom
   Software
   Test environment
   (HILS）
   System Under Test
   （integration of hardware）
   sensor actuator
   System Under Test
   Test environment
   (PILS/S-PILS)
   sensor actuator
   actual simulation
   【legend】
   Test environment 1
   (The HILS test environment)
   Test environment 2
   (The SILS test environment)
   inout＝hardware I/F
   （pin１＝2.5V, etc）
   inout＝software I/F
   （val１＝250, etc）
   inout＝software I/F
   （val１＝250, etc）
   Issue of Testing First in
   Embedded systems
   Test environment 2
   (The SILS test environment)
   1
   1
   

   View Slide

5. Issues & Solutions
   December 11, #STAC2021
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved 5
   《Issues》
   - Must test while integrated with hardware.
   - Want to test software first, eliminate bugs, but don't have general-purpose tools.
   - Difficult to reuse the software test environment. Because input & output are different.
   《Solutions》
   - Create a SILS environment using open source.
   - To make test scenario to be common in different test environments.
   - Implements(Libraries) hide difference between test environments.
   

   View Slide

6. Exterior Lighting System
   Exterior Lighting System
   December 11, #STAC2021
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved 6
   environment
   light
   ENGIN
   START
   Ignition
   state
   ECU A ECU B
   voltage
   System Under Test
   1. Parking light is active at night.
   2. Start engine.
   3. From parking light to passing light.
   《test scenario》
   Photoelectric
   sensor
   light
   parking light
   passing light
   light
   light
   com
   （I2C）
   ２
   human
   

   View Slide

7. Exterior Lighting System
   7
   December 11, #STAC2021 7
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved
   Exterior
   Lighting ECU
   HILS test environment of Exterior
   Lighting System
   Front
   Spotlights ECU
   com（I2C）
   Photoelectric
   sensor
   actual actual
   actual
   simulation
   Ignition state
   simulation
   parking light
   actual
   passing light
   actual
   light
   voltage
   light
   light
   Voltage
   Voltage
   PWM
   PWM
   simulation(human)
   simulation(human)
   actual
   Robot Framework
   HIL equipment
   test scenario
   Libraries(for HILS)
   environment
   Light
   Photoelectric sensor
   Photoelectric sensor
   

   View Slide

8. 1
   1
   Exterior Lighting System
   December 11, #STAC2021
   Exterior Lighting ECU
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved
   Front Spotlights ECU
   ROS
   TCP/IP
   Photoelectric
   sensor
   simulation
   environment light
   Ignition state
   simulation
   parking light
   passing light
   Photoelectric sensor
   numerical
   value(light)
   num
   （voltage)
   num
   (light)
   num
   （voltage)
   simulation(human)
   Library
   Library
   sim
   Library
   sim
   stub
   source
   code
   actual sim
   stub
   sim
   sim
   stub
   actual sim
   stub
   Library
   Library
   simulation
   Library
   Library
   num
   (voltage)
   source
   code
   num (pwm)
   docker on Amazon Web Service
   wrapper wrapper
   Robot Framework
   test scenario
   Libraries(for SILS)
   num (pwm）
   SIL equipment
   8
   SILS test environment of Exterior
   Lighting System
   simulation
   simulation(human)
   Photoelectric sensor
   num
   (light)
   

   View Slide

9. Note: Wrapper
   December 11, #STAC2021
   © 2021 Embedded Systems CI Research Working
   Group All rights reserved 9
   wrapper
   Robot
   Framework
   Exterior
   Lighting ECU
   Software
   System
   under test
   Front
   Spotlights
   ECU
   Software
   ECU
   execution
   control
   server
   ECU
   execution
   control
   server
   docker container
   docker container
   docker engine (container type virtualization software)
   Wrapper
   Wrapper
   docker container
   test scenario
   (python)
   read/write
   each ECU’s
   variables
   ECU’s
   communication,
   simulated by
   ROS(Robot OS)
   

   View Slide

10. Note: Wrapper
    December 11, #STAC2021 © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    10
    ECU execution control server
    Flask
    (Web server)
    Python
    test scenario
    (python)
    ECU execution control server
    Flask
    (Web server)
    Python
    Exterior
    Lighting
    ECU
    Software
    (Clang)
    Front
    Spotlights
    ECU
    Software
    (Clang)
    SharedMemory
    SharedMemory
    Wrapper
    Wrapper
    

    View Slide

11. DEMO VIDEO (HILS test environment)
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 11
    https://www.youtube.com/channel/UCMS_fCjRiB2XjakCR5fGLGQ
    

    View Slide

12. December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 12
    https://www.youtube.com/channel/UCMS_fCjRiB2XjakCR5fGLGQ
    DEMO VIDEO (SILS test environment)
    

    View Slide

13. pros & cons
    《pros》
    - Can test source code early on embedded systems. (Quality）
    - No need actual test environment. （Quality）
    - Can scale-out test environment. （Quality）
    - Can reuse test scenario. （Cost）
    - May reduce making test environment cost. (Cost)
    《cons》
    - So expensive to use for embedded software engineers.
    - Difficult to maintain libraries.
    - Cannot be used for real-time performance testing.
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 13
    

    View Slide

14. Test report (SILS test environment)
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 14
    Test Scenario
    (common to HILS & SILS)
    

    View Slide

15. December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 15
    テストケース
    It was not executed
    because of abnormal
    processing.
    Test report (SILS test environment)
    

    View Slide

16. How to get this tool
    You can get the complete environment
    with the test case used in the demo from the URL below.
    Test Execution Automation Framework for Embedded Systems
    https://github.com/misawa-san/ExteriorLightSystem
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 16
    

    View Slide

17. How to run SILS test environment
    Run the Docker container.
    ■procedure
    1. git clone https://github.com/misawa-san/ExteriorLightSystem.git
    2. cd ExteriorLightSystem
    3. sudo chmod -R 777 .
    4. docker-compose -f docker-compose.yml up -d --build
    5. docker-compose -f docker-compose.yml up -d --build
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 17
    

    View Slide

18. Note:
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 18
    SILS test environment
    HILS test environment
    keyword-driven
    testing
    To make test scenario to be
    common in different test
    environments.
    Test Scenario
    (common to HILS & SILS)
    

    View Slide

19. Participants wanted
    December 11, #STAC2021
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved 19
    Embedded Systems CI Research Working Group is conducting
    research continuous-integration/testing/deployment
    for Embedded Systems.
    If you are interested, please join us.
    It’s held once a month, mainly on Sundays from 13:30 to 17:00.
    Please contact us at the following email address.
    aster-embci@qualab.jp
    

    View Slide

20. Thank you for your kind attention
    20
    © 2021 Embedded Systems CI Research Working
    Group All rights reserved
    December 11, #STAC2021
    

    View Slide