A4Q World Congress: 2021 IEEE Autonomous Driving AI Test Challenge

Elena Treshcheva - IEEE AV AI Test Challenge Organizing Committee,
Researcher, Exactpro

2021 IEEE Autonomous Driving AI Test Challenge
15.04.2021
--
To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. 2021 IEEE Autonomous Driving
   AI Test Challenge
   Elena Treshcheva
   IEEE AV AI Test Challenge Organizing Committee,
   Researcher, Exactpro
   

   View Slide

2. http://av-test-challenge.org/
   2021 IEEE Autonomous Driving AI Test Challenge
   

   View Slide

3. - WHY both industry and academia need initiatives at the
   confluence of AI and software testing
   - WHAT the 2021 IEEE Autonomous Driving AI Test
   Challenge is all about
   - HOW one can explore the knowledge domain of testing
   complex intelligent systems
   Overview
   

   View Slide

4. WHY: AI and Software Testing as a new field of studies
   Today’s context:
   - rapid advancements in AI-based technologies,
   active development of intelligent systems
   - research on trustworthiness, safety and reliability
   of AI
   - professional certifications and educational
   programs
   Indicators of the new area being established
   

   View Slide

5. WHY: AI and Software Testing as a new field of studies
   

   View Slide

6. AI systems as specific type of the system
   under test:
   - Non-deterministic
   - Non-testable with traditional methods
   AI to support software testing:
   - Oracle problem: telling right from wrong cannot be delegated to the algorithms
   - Limited applicability - not to testing per se, but rather to surrounding tasks
   A4Q AI and Software Testing Foundation Syllabus
   WHY: AI and Software Testing as a new field of studies
   

   View Slide

7. WHY: AI and Software Testing as a new field of studies
   Quality characteristics of AI-based systems:
   - Ability to learn: the capacity of the system to learn from use for the
   system itself, or data and events it is exposed to
   - Ability to generalize: the ability of the system to apply to different
   and previously unseen scenarios
   - Trustworthiness: the degree to which the system is trusted by
   stakeholders
   A4Q AI and Software Testing Foundation Syllabus
   

   View Slide

8. WHY: AI and Software Testing as a new field of studies
   Quality characteristics of AI-based systems:
   - Ability to learn: the capacity of the system to learn from use for the
   system itself, or data and events it is exposed to
   - Ability to generalize: the ability of the system to apply to different
   and previously unseen scenarios
   - Trustworthiness: the degree to which the system is trusted by
   stakeholders
   A4Q AI and Software Testing Foundation Syllabus
   

   View Slide

9. WHY: AI and Software Testing as a new field of studies
   Quality characteristics of AI-based systems:
   - Ability to learn: the capacity of the system to learn from use for the
   system itself, or data and events it is exposed to
   - Ability to generalize: the ability of the system to apply to different
   and previously unseen scenarios
   - Trustworthiness: the degree to which the system is trusted by
   stakeholders
   A4Q AI and Software Testing Foundation Syllabus
   

   View Slide

10. http://av-test-challenge.org/
    WHAT: IEEE Autonomous Driving AI Test Challenge
    

    View Slide

11. http://ieeeaitests.com/
    The Third IEEE International Conference on Artificial
    Intelligence Testing
    

    View Slide

12. http://www.ieeeaitests.com/html/callForPapers.html
    AI Test 2021 - Call for Papers:
    - Regular (8 pages) and Short papers (2
    pages)
    - AI Testing In Practice (8 pages)
    - Tool Demo Track (4 pages)
    The Third IEEE International Conference on Artificial
    Intelligence Testing
    

    View Slide

13. http://www.ieeeaitests.com/html/callForPapers.html
    AI Test 2021 - Topics of Interest:
    - Testing AI applications
    - Applications of AI techniques to software testing
    - Data quality checking for AI applications
    The Third IEEE International Conference on Artificial
    Intelligence Testing
    

    View Slide

14. http://av-test-challenge.org/
    WHAT: IEEE Autonomous Driving AI Test Challenge
    

    View Slide

15. Autonomous Driving: Testing by simulation
    

    View Slide

16. Autonomous Driving: Testing by simulation
    

    View Slide

17. Autonomous Driving: Testing by simulation
    

    View Slide

18. Autonomous Driving: Testing by simulation
    

    View Slide

19. Autonomous Driving: Testing by simulation
    

    View Slide

20. Autonomous Driving: Testing by simulation
    

    View Slide

21. Autonomous Driving: Testing by simulation
    

    View Slide

22. Autonomous Driving: Testing by simulation
    

    View Slide

23. https://arxiv.org/pdf/2005.03778.pdf
    Rong et al. (2020) - LGSVL Simulator: A high fidelity
    simulator for autonomous driving
    

    View Slide

24. https://arxiv.org/pdf/2005.03778.pdf
    Rong et al. (2020) - LGSVL Simulator: A high fidelity
    simulator for autonomous driving
    

    View Slide

25. https://arxiv.org/pdf/2005.03778.pdf
    Rong et al. (2020) - LGSVL Simulator: A high fidelity
    simulator for autonomous driving
    

    View Slide

26. Baidu Apollo Simulator
    

    View Slide

27. Tutorial: End-to-end video tutorial of LGSVL simulator with Apollo driving
    Recordings and slides from the AV AI Test Challenge training sessions
    HOW: More on Software testing and AI
    

    View Slide

28. Software testing for complex intelligent systems and autonomous vehicles -
    Playlist on the Exactpro YouTube channel
    Software Testing for Complex
    Intelligent Systems and
    Autonomous Vehicles
    Online Course
    HOW: More on Software testing and AI
    

    View Slide

29. Online course from Exactpro
    Software Testing for Complex
    Intelligent Systems and
    Autonomous Vehicles
    Online Course
    - an introduction to AI-based software, risks associated with it
    and the necessity of its comprehensive testing
    - general knowledge of the software testing basics
    - an ability to use testing methods on real-life tasks
    - hands-on experience of using test tools and configuring a
    test environment
    - the skill of writing test cases, including automated test
    cases, and executing them
    - an ability to analyse and interpret test run results
    

    View Slide

30. 1. Autonomous driving and software testing
    2. Introduction to self-driving cars
    3. Application programming interface
    4. Foundations of software testing
    5. Build Software to Test Software
    6. Reliability and safety
    7. Software defects and metrics
    8. Software negligence and testing coverage
    Online course from Exactpro
    Theoretical
    module
    Online course on the Exactpro
    YouTube channel
    

    View Slide

31. 1. Launching the LGSVL Simulator
    2. Setting up a Python IDE
    3. Unit Testing
    4. Behavior Driven Development
    5. Data and Keyword Driven Testing
    6. Data Analysis and Reporting
    7. Running Apollo
    8. Modeling Test Scenarios
    Online course from Exactpro
    Practical
    module
    Online course on the Exactpro
    YouTube channel
    

    View Slide

32. Takeaways
    - WHY: we need initiatives at the confluence of AI and software testing to
    work out new software testing techniques addressing sophisticated AI
    systems
    - WHAT: IEEE Autonomous Driving AI Test Challenge and AI Test
    Conference serve as a forum for discussion and development of new
    approaches to software testing
    - HOW: We definitely need to prepare software testers to work with
    intelligent systems. There are resources providing knowledge and
    training in the intersection of AI and Software Testing domains
    

    View Slide

33. View Slide