A Quality Model for Trustworthy AI Systems: Data, Code and Models

Short Introduction (Karnowski & Fadely, 2016) • Software engineer (MSc Computer Science) since 2002 • Software quality consultant since 2004 • Lecturer Fontys ICT since 2012 • 2016 PhD “Quality of JIT Requirements” (Computer Science) • 2019 Senior researcher AI Engineering “How to build trustworthy AI systems?” https://fontysblogt.nl/author/petraheck/

AI Engineering “the use of scientific principles, tools, and techniques of machine learning and traditional software engineering to design and build complex computing systems” Burkov, A. (2020). Machine learning engineering. True Positive Inc. (Harikrishna, 2018) Farah, D. (2020). The modern MLOps blueprint.

AI-Enabled System = Data + Model + Code https://martinfowler.com/articles/cd4ml.html

Quality of Software Products: ISO 25000 Standard https://iso25000.com/index.php/en/iso-25000-standards

Quality of AI-Enabled Systems: ISO 25000 + ? ML testing properties: • correctness • overfitting degree • fairness • interpretability • robustness • security • data privacy • efficiency Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2020). Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering. EU (2019). Ethics guidelines for trustworthy AI.

No content

FlowerPower Wildflower Monitoring Platform https://www.portfoliofontysict.nl/en/march-2022/flowerpower-makes-biodiversity-measurable-with-ai-image-analysis/

Self-learning vs Rule-based: FlowerPower AI Model https://livebook.manning.com/book/deep-learning-with-javascript/chapter-1/v-3/

For AI systems we train machine learning models to learn rules from data • Model correctness: learn the correct rules, also for unseen data • Reproducibility: traceable learning process • Model robustness: be resilient against perturbed inputs, such as adversarial attacks

Example AI-related Quality Requirements for FlowerPower • Correctness: o The system shall only detect flowers in there blooming phase that are at least 50% visible and with a flower head size of at least 1cm. o The system shall function with different camera types with a resolution higher than 15Mpx. o The system shall function with photos taken in different open landscape types under different weather conditions.

Training Data Selection vs Human Impact

Difficult to check large datasets from different sources • Privacy: not uncover personal properties • Fairness: good representation of the real population

Example AI-related Quality Requirements for FlowerPower • Privacy: o The system shall remove or obfuscate person-related metadata from the photos. o The system shall protect the exact location of rare and vulnerable wildflowers. • Fairness: o The system shall have a configurable parameter removing annotations (as well as the associated images on which these annotations occur) of underrepresented species.

Human-centered AI Biology Researcher AI Engineer Policy Maker Citizen Scientist

Automate/support decision making • Collaboration effectiveness: help, not hinder • Explainability: clear on which grounds (aka interpretability and transparency) • Controllability: steer the outcome or decision of the AI system if needed • Human autonomy: the user should not feel threatened

Example AI-related Quality Requirements for FlowerPower • Collaboration effectiveness: The system shall have a batch mode to aggregate wildflower species counts over multiple photos. • Explainability: The system shall provide saliency maps. • Controllability: o The user shall be able to add or correct annotations o The user shall be able to request a retraining of the model with an updated or new dataset. o The AI engineer shall be able to upload a new object detection model and compare its metrics with the performance metrics of previous models.

No content

From Quality Model to Quality Toolbox • Model = dictionary for defining AI-related quality requirements • Industry case studies with students and applied researchers • “How to build trustworthy AI systems?” • Data, model, and code • Best practices, tools, techniques • Prevention, detection, mitigation • Toolbox and education material for applied university and industry https://fontysblogt.nl/author/petraheck/

Thank you! Questions? [email protected]