Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Re-thinking Regression Testing
Search
Mario Gonzalez Macedo
March 21, 2014
Research
1
120
Re-thinking Regression Testing
Talk I gave at Mountain West Ruby Conf 2014 about mutation testing
Mario Gonzalez Macedo
March 21, 2014
Tweet
Share
Other Decks in Research
See All in Research
Rの機械学習フレームワークの紹介〜tidymodelsを中心に〜 / machine_learning_with_r2024
s_uryu
0
240
メタ動画データセットによる動作認識の現状と可能性
yuyay
0
210
SANER 2019 Most Influential Paper Talk
tsantalis
1
120
Alternative Photographic Processes Reimagined: The Role of Digital Technology in Revitalizing Classic Printing Techniques【SIGGRAPH Asia 2023】
toremolo72
0
440
MegaParticles: GPUを利用したStein Particle Filterによる点群6自由度姿勢推定
koide3
1
550
How to Perform Manual Classification for Deep Learning Using CloudCompare
kentaitakura
0
670
F0に基づいて伸縮された画像文字からの音声合成 [ASJ2024春]
nehi0615
0
120
時系列解析と疫学
kingqwert
2
940
継続的な研究費獲得のための考え方
moda0
2
440
精神疾患患者のアクティビティデータを利用したリハビリテーションのためのシステムに関する研究
comfortdesignlab
0
150
First Authorに俺はなるっ!! IROS’23 CCC2023 FY
shota_nishiyama
0
180
HP (Hitto Point: 筆頭ポイント)
tanichu
0
750
Featured
See All Featured
Fantastic passwords and where to find them - at NoRuKo
philnash
39
2.5k
A Philosophy of Restraint
colly
197
16k
KATA
mclloyd
16
12k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
4 Signs Your Business is Dying
shpigford
176
21k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
8
1.3k
RailsConf 2023
tenderlove
8
550
Documentation Writing (for coders)
carmenintech
60
4k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
6.9k
VelocityConf: Rendering Performance Case Studies
addyosmani
321
23k
Building Effective Engineering Teams - LeadDev
addyosmani
31
1.9k
Principles of Awesome APIs and How to Build Them.
keavy
121
16k
Transcript
Re-thinking regression testing Mario González Macedo @maramono ortask.com
http://www.softwaretestpro.com/Item/5020/Minor-Risk-Patches/Cartoon
regression code coverage test redundancy ortask.com
ortask.com
regression code coverage test redundancy ortask.com
mutation testing
…vs code coverage
ortask.com
ortask.com
ortask.com
ortask.com
http://www.softwaretestpro.com/Item/5020/Minor-Risk-Patches/Cartoon
ortask.com
ortask.com
mutation score
(0, 1) worst mutation-adequate
ok… but what?
a type of fault injection
no extra code
change the code
mutate existing source code changes to tests: none
None
None
score… how?
mutants total killed mutants score =
zombies total killed zombies score =
ninjas total killed ninjas score =
how many mutations?
one mutation?
two mutations?
100 mutations?
k = mutation qty
assumption
assumption (a valid hypothesis)
competent programmer hypothesis
competent programmer hypothesis Some software bugs are caused by syntactically
small errors
competent programmer hypothesis k = mutation qty
competent programmer hypothesis k = 1
k
if k == 1 then first-order
if k == 2 then second-order
k-order mutant k mutations
k-order for simplicity easy to communicate
if k == 1 then low-order or simply first-order
if k > 1 then high-order
high-order… but why?
competent programmer hypothesis
competent programmer hypothesis coupling effect
coupling effect Other software bugs are caused by interactions of
small errors
coupling effect 1 < k < 6
why it works
for regression analysis one developer on fixes and features competent
programmer hypothesis
for regression analysis many contributors on fixes and features competent
programmer hypothesis coupling effect
so… why better?
test redundancy mutation score correlation: code coverage & test redundancy
ortask.com
ortask.com
mutation score test redundancy mutation score correlation: code coverage &
ortask.com
code coverage correlation: likelihood of bugs & test redundancy ortask.com
http://www.sustainablebabysteps.com/inflating-tires.html
http://willyoubemyyokoono.blogspot.com/2012_09_01_archive.html
correlation: mutation score & test redundancy code coverage ortask.com
how to apply it in practice
heckle mutant muJava Jester NinjaTurtles
heckle: obsolete as of Ruby 1.9 mutant: did not use
(rc) muJava Jester NinjaTurtles
https://github.com/ortask/mutator research-strength fast cross-language
painless mutation analysis a few guidelines
guideline 1: passing tests http://vector-magz.com/logo/quality-control-logo-item-4/
guideline 2: mutants 50 low-order experiment with high-order
guideline 3: cycle mutate score improve test
guideline 4: coupling low coupling high coupling detection
Thank you! Mario González Macedo @maramono ortask.com
Other mutation testing sources http://slid.es/markusschirp/mutation-testing https://github.com/mbj/mutant/tree/master/l ib/mutant/mutator/node https://github.com/seattlerb/heckle http://vimeo.com/10863662
Other sources http://vimeo.com/1683910