Introduction to Test Execution Automation Framework for Embedded Systems

Transcript

1. Embedded Systems CI Research Working Group presents Introduction to Test Execution Automation Framework for Embedded Systems Embedded Systems CI Research Working Group Hiroshi Misawa 1 © 2021 Embedded Systems CI Research Working Group All rights reserved December 11, #STAC2021
2. 2 ・Hiroshi Misawa ・born in Osaka, Japan ・Embedded Software Engineer. ・semiconductor company to automotive component manufacturer. ・JASST(Japan Symposium on Software Testing) Tokai's executive committee. ・Embedded Systems CI Research Working Group ・Recent interests AWS,Docker,CI ・Hobby ・Making toys ・Ring Fit Adventure(Lv 330) © 2021 Embedded Systems CI Research Working Group All rights reserved December 11, #STAC2021 Self introduction
3. 3 © 2021 Embedded Systems CI Research Working Group All rights reserved December 11, #STAC2021 Issue#1 Must test while integrated with hardware. implementation SYS Requirement definition SW Requirement definition SW architect design SW detailed design SYS architect design SW unit testing SW integration testing SW system testing Sys integration testing Sys system testing Typical issue: Embedded Systems Testing 2)only software 1)integration of hardware Issue#2 Want to test software first, eliminate bugs, but don't have general-purpose tools.
4. 4 © 2021 Embedded Systems CI Research Working Group All rights reserved December 11, #STAC2021 Issue#3 Difficult to reuse the software test environment. Because input & output are different. System Under Test Test environment (SILS) sensor actuator micom Software Test environment (HILS） System Under Test （integration of hardware） sensor actuator System Under Test Test environment (PILS/S-PILS) sensor actuator actual simulation 【legend】 Test environment 1 (The HILS test environment) Test environment 2 (The SILS test environment) inout＝hardware I/F （pin１＝2.5V, etc） inout＝software I/F （val１＝250, etc） inout＝software I/F （val１＝250, etc） Issue of Testing First in Embedded systems Test environment 2 (The SILS test environment) 1 1
5. Issues & Solutions December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 5 《Issues》 - Must test while integrated with hardware. - Want to test software first, eliminate bugs, but don't have general-purpose tools. - Difficult to reuse the software test environment. Because input & output are different. 《Solutions》 - Create a SILS environment using open source. - To make test scenario to be common in different test environments. - Implements(Libraries) hide difference between test environments.
6. Exterior Lighting System Exterior Lighting System December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 6 environment light ENGIN START Ignition state ECU A ECU B voltage System Under Test 1. Parking light is active at night. 2. Start engine. 3. From parking light to passing light. 《test scenario》 Photoelectric sensor light parking light passing light light light com （I2C） ２ human
7. Exterior Lighting System 7 December 11, #STAC2021 7 © 2021 Embedded Systems CI Research Working Group All rights reserved Exterior Lighting ECU HILS test environment of Exterior Lighting System Front Spotlights ECU com（I2C） Photoelectric sensor actual actual actual simulation Ignition state simulation parking light actual passing light actual light voltage light light Voltage Voltage PWM PWM simulation(human) simulation(human) actual Robot Framework HIL equipment test scenario Libraries(for HILS) environment Light Photoelectric sensor Photoelectric sensor
8. 1 1 Exterior Lighting System December 11, #STAC2021 Exterior Lighting ECU © 2021 Embedded Systems CI Research Working Group All rights reserved Front Spotlights ECU ROS TCP/IP Photoelectric sensor simulation environment light Ignition state simulation parking light passing light Photoelectric sensor numerical value(light) num （voltage) num (light) num （voltage) simulation(human) Library Library sim Library sim stub source code actual sim stub sim sim stub actual sim stub Library Library simulation Library Library num (voltage) source code num (pwm) docker on Amazon Web Service wrapper wrapper Robot Framework test scenario Libraries(for SILS) num (pwm） SIL equipment 8 SILS test environment of Exterior Lighting System simulation simulation(human) Photoelectric sensor num (light)
9. Note: Wrapper December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 9 wrapper Robot Framework Exterior Lighting ECU Software System under test Front Spotlights ECU Software ECU execution control server ECU execution control server docker container docker container docker engine (container type virtualization software) Wrapper Wrapper docker container test scenario (python) read/write each ECU’s variables ECU’s communication, simulated by ROS(Robot OS)
10. Note: Wrapper December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 10 ECU execution control server Flask (Web server) Python test scenario (python) ECU execution control server Flask (Web server) Python Exterior Lighting ECU Software (Clang) Front Spotlights ECU Software (Clang) SharedMemory SharedMemory Wrapper Wrapper
11. DEMO VIDEO (HILS test environment) December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 11 https://www.youtube.com/channel/UCMS_fCjRiB2XjakCR5fGLGQ
12. December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 12 https://www.youtube.com/channel/UCMS_fCjRiB2XjakCR5fGLGQ DEMO VIDEO (SILS test environment)
13. pros & cons 《pros》 - Can test source code early on embedded systems. (Quality） - No need actual test environment. （Quality） - Can scale-out test environment. （Quality） - Can reuse test scenario. （Cost） - May reduce making test environment cost. (Cost) 《cons》 - So expensive to use for embedded software engineers. - Difficult to maintain libraries. - Cannot be used for real-time performance testing. December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 13
14. Test report (SILS test environment) December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 14 Test Scenario (common to HILS & SILS)
15. December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 15 テストケース It was not executed because of abnormal processing. Test report (SILS test environment)
16. How to get this tool You can get the complete environment with the test case used in the demo from the URL below. Test Execution Automation Framework for Embedded Systems https://github.com/misawa-san/ExteriorLightSystem December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 16
17. How to run SILS test environment Run the Docker container. ▪procedure 1. git clone https://github.com/misawa-san/ExteriorLightSystem.git 2. cd ExteriorLightSystem 3. sudo chmod -R 777 . 4. docker-compose -f docker-compose.yml up -d --build 5. docker-compose -f docker-compose.yml up -d --build December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 17
18. Note: December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 18 SILS test environment HILS test environment keyword-driven testing To make test scenario to be common in different test environments. Test Scenario (common to HILS & SILS)
19. Participants wanted December 11, #STAC2021 © 2021 Embedded Systems CI Research Working Group All rights reserved 19 Embedded Systems CI Research Working Group is conducting research continuous-integration/testing/deployment for Embedded Systems. If you are interested, please join us. It’s held once a month, mainly on Sundays from 13:30 to 17:00. Please contact us at the following email address. aster-embci@qualab.jp
20. Thank you for your kind attention 20 © 2021 Embedded Systems CI Research Working Group All rights reserved December 11, #STAC2021