Exploratory Unit Testing with and for GenAI

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

© 2025 CGI Inc. 4 @maaretp @[email protected] o Who Am I? Guessing with power to accept Testing my program or the tool? Note: …pillaging digital content without consent, compensation and attribution Even if legal not ethical.

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

© 2025 CGI Inc. 15 Part I. Review for correctness and conciseness Part II. Input -> Output Part III. Rules of behavior boundaries Part IIII. Coverage Part IIIII. Sampling vs wide nets (approvals) Part VI. Properties Developer intent

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

© 2025 CGI Inc. 25 Part 1. Be the resident expert. Ask around. Part II. Rules. More rules. Part III. Find better experts. Part IIII. Disagreeing with boundaries. Part IIIII. Oracles. Part VI. Find better oracles. Part VII. No user would do what users would do. Domain

Slide 26

Slide 26 text

© 2025 CGI Inc. 26 Part 1. Dependencies. Part II. Interruptions. Both software and hardware. Part III. People. “People are not pure functions; they have all sorts of interesting side effects.” - Engineering Management for the Rest of Us, Sarah Drasner … nor are pure functions if you grow the boundary of what might fail. Environment

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2025 CGI Inc. 28 Answer key to some of the bugs 1…3999 is not the right boundary. 4 is IIII and 5 is IIIII and 50 is XXXXX with right choice of domain. Entirely different rules from classic to concise to simplified. Fractions and zero do exist. Infinite loops, miscalculation at boundaries. Unimplemented boundary checking. Reuse without license. 28

Slide 29

Slide 29 text

© 2025 CGI Inc. 29 Stakeholders happy, even delighted –Quality Information Good Team’s Output –Quality Information Less than Good Team’s Output –Quality Information Results Gap Surprise! Results Gap on a Team that thinks Testers == Testing Pick up the pizza boxes… ”Find (some of) What Others May Have Missed”

Slide 30

Slide 30 text

© 2025 CGI Inc. 30 A majority of the production failures (77%) can be reproduced by a unit test. https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf Through https://www.slideshare.net/Kevlin/the-error-of-our-ways

Slide 31

Slide 31 text

© 2025 CGI Inc. 31 A vocab walkthrough From Arrange-Act- Assert to Do- Verify Structure From Asserts to Approvals Timing of expected From handcrafted examples to generated Tools to do more From hard to soft Decision of fail handling Tools ideology Pandora’s box with opportunities

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

© 2025 CGI Inc. 35 Insights you can act on Founded in 1976, CGI is among the largest IT and business consulting services firms in the world. We are insights-driven and outcomes-based to help accelerate returns on your investments. Across hundreds of locations worldwide, we provide comprehensive, scalable and sustainable IT and business consulting services that are informed globally and delivered locally. cgi.com