A4Q World Congress: 2021 IEEE Autonomous Driving AI Test Challenge

Elena Treshcheva - IEEE AV AI Test Challenge Organizing Committee,
Researcher, Exactpro

2021 IEEE Autonomous Driving AI Test Challenge
15.04.2021
--
To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. 2021 IEEE Autonomous Driving AI Test Challenge Elena Treshcheva IEEE AV AI Test Challenge Organizing Committee, Researcher, Exactpro
2. http://av-test-challenge.org/ 2021 IEEE Autonomous Driving AI Test Challenge
3. - WHY both industry and academia need initiatives at the confluence of AI and software testing - WHAT the 2021 IEEE Autonomous Driving AI Test Challenge is all about - HOW one can explore the knowledge domain of testing complex intelligent systems Overview
4. WHY: AI and Software Testing as a new field of studies Today’s context: - rapid advancements in AI-based technologies, active development of intelligent systems - research on trustworthiness, safety and reliability of AI - professional certifications and educational programs Indicators of the new area being established
5. WHY: AI and Software Testing as a new field of studies
6. AI systems as specific type of the system under test: - Non-deterministic - Non-testable with traditional methods AI to support software testing: - Oracle problem: telling right from wrong cannot be delegated to the algorithms - Limited applicability - not to testing per se, but rather to surrounding tasks A4Q AI and Software Testing Foundation Syllabus WHY: AI and Software Testing as a new field of studies
7. WHY: AI and Software Testing as a new field of studies Quality characteristics of AI-based systems: - Ability to learn: the capacity of the system to learn from use for the system itself, or data and events it is exposed to - Ability to generalize: the ability of the system to apply to different and previously unseen scenarios - Trustworthiness: the degree to which the system is trusted by stakeholders A4Q AI and Software Testing Foundation Syllabus
8. WHY: AI and Software Testing as a new field of studies Quality characteristics of AI-based systems: - Ability to learn: the capacity of the system to learn from use for the system itself, or data and events it is exposed to - Ability to generalize: the ability of the system to apply to different and previously unseen scenarios - Trustworthiness: the degree to which the system is trusted by stakeholders A4Q AI and Software Testing Foundation Syllabus
9. WHY: AI and Software Testing as a new field of studies Quality characteristics of AI-based systems: - Ability to learn: the capacity of the system to learn from use for the system itself, or data and events it is exposed to - Ability to generalize: the ability of the system to apply to different and previously unseen scenarios - Trustworthiness: the degree to which the system is trusted by stakeholders A4Q AI and Software Testing Foundation Syllabus
10. http://av-test-challenge.org/ WHAT: IEEE Autonomous Driving AI Test Challenge
11. http://ieeeaitests.com/ The Third IEEE International Conference on Artificial Intelligence Testing
12. http://www.ieeeaitests.com/html/callForPapers.html AI Test 2021 - Call for Papers: - Regular (8 pages) and Short papers (2 pages) - AI Testing In Practice (8 pages) - Tool Demo Track (4 pages) The Third IEEE International Conference on Artificial Intelligence Testing
13. http://www.ieeeaitests.com/html/callForPapers.html AI Test 2021 - Topics of Interest: - Testing AI applications - Applications of AI techniques to software testing - Data quality checking for AI applications The Third IEEE International Conference on Artificial Intelligence Testing
14. http://av-test-challenge.org/ WHAT: IEEE Autonomous Driving AI Test Challenge
15. Autonomous Driving: Testing by simulation
16. Autonomous Driving: Testing by simulation
17. Autonomous Driving: Testing by simulation
18. Autonomous Driving: Testing by simulation
19. Autonomous Driving: Testing by simulation
20. Autonomous Driving: Testing by simulation
21. Autonomous Driving: Testing by simulation
22. Autonomous Driving: Testing by simulation
23. https://arxiv.org/pdf/2005.03778.pdf Rong et al. (2020) - LGSVL Simulator: A high fidelity simulator for autonomous driving
24. https://arxiv.org/pdf/2005.03778.pdf Rong et al. (2020) - LGSVL Simulator: A high fidelity simulator for autonomous driving
25. https://arxiv.org/pdf/2005.03778.pdf Rong et al. (2020) - LGSVL Simulator: A high fidelity simulator for autonomous driving
26. Baidu Apollo Simulator
27. Tutorial: End-to-end video tutorial of LGSVL simulator with Apollo driving Recordings and slides from the AV AI Test Challenge training sessions HOW: More on Software testing and AI
28. Software testing for complex intelligent systems and autonomous vehicles - Playlist on the Exactpro YouTube channel Software Testing for Complex Intelligent Systems and Autonomous Vehicles Online Course HOW: More on Software testing and AI
29. Online course from Exactpro Software Testing for Complex Intelligent Systems and Autonomous Vehicles Online Course - an introduction to AI-based software, risks associated with it and the necessity of its comprehensive testing - general knowledge of the software testing basics - an ability to use testing methods on real-life tasks - hands-on experience of using test tools and configuring a test environment - the skill of writing test cases, including automated test cases, and executing them - an ability to analyse and interpret test run results
30. 1. Autonomous driving and software testing 2. Introduction to self-driving cars 3. Application programming interface 4. Foundations of software testing 5. Build Software to Test Software 6. Reliability and safety 7. Software defects and metrics 8. Software negligence and testing coverage Online course from Exactpro Theoretical module Online course on the Exactpro YouTube channel
31. 1. Launching the LGSVL Simulator 2. Setting up a Python IDE 3. Unit Testing 4. Behavior Driven Development 5. Data and Keyword Driven Testing 6. Data Analysis and Reporting 7. Running Apollo 8. Modeling Test Scenarios Online course from Exactpro Practical module Online course on the Exactpro YouTube channel
32. Takeaways - WHY: we need initiatives at the confluence of AI and software testing to work out new software testing techniques addressing sophisticated AI systems - WHAT: IEEE Autonomous Driving AI Test Challenge and AI Test Conference serve as a forum for discussion and development of new approaches to software testing - HOW: We definitely need to prepare software testers to work with intelligent systems. There are resources providing knowledge and training in the intersection of AI and Software Testing domains