Slide 1

Slide 1 text

Towards a usable defect prediction tool Vladimir Kovalenko JetBrains Crossbreeding machine learning and heuristics

Slide 2

Slide 2 text

Defect Prediction • Common goal: identify defect-prone entities in advance • Why? • QA • Resource allocation (testing, review, etc)

Slide 3

Slide 3 text

Previous research • Academia • Microsoft Research • Google case study

Slide 4

Slide 4 text

Research papers (common points) • ML defect prediction models work in general • No universal model: projects are too different • Code metrics as features improve prediction quality • Typical precision/recall ~0.7

Slide 5

Slide 5 text

Google case study • Collaborated with researchers to introduce defect prediction in internal code review system • Came up with a heuristic model Google Bug Prediction Score (Time Weighted Risk)

Slide 6

Slide 6 text

Tools • No defect prediction tools known to be used in industry • Why? • Too low accuracy • Too much effort to set up

Slide 7

Slide 7 text

Tool usability criteria • Language independent • “entity” >= file • Little or no effort to set up • no plain supervised learning • Near real-time • Easy to use: VCS agnostic, etc • Accurate!

Slide 8

Slide 8 text

Implementation • CI server plugin • Only use VCS metrics • Automatic bugfix changes detection (heuristics) • Processing: detect bug-introducing changes • ML classifier: Naive Bayes / Decision Tree • Take prediction top, not absolute values • Automatic quality evaluation

Slide 9

Slide 9 text

Features • Local change frequency • Number of authors • File age • Number of affecting commits • Google Score

Slide 10

Slide 10 text

Quality Evaluation • Bug tracker integration: find bugfix changes • Quality metric: fraction of files from model predictions affected by bugfix changes in the future

Slide 11

Slide 11 text

Result samples Project A, 2 years Project B, 1 year

Slide 12

Slide 12 text

Conclusions • It is possible combine learning and heuristic approaches to get the best of two worlds • The accuracy is still not good enough • No wonder no prediction tools are widely used yet

Slide 13

Slide 13 text

Thank you! [email protected]