Testing is Overrated

A look at the problems with automated testing, and other techniques that can be used to improve software quality, such as user experience testing and code reviews.

Delivered at RubyFringe in Toronto.

For more references and more details, see this talk's page on my website:

Transcript

1. Testing is overrated Luke Francl
2. Test::Unit RSpec Shoulda TDD BDD unit tests behaviors test-first test-along Selenium story runner Watir rcov green bar coverage test cases fixtures mock stub Mocha object mother autotest fuzz random miniunit
3. All you need is tests
4. Extensive research
5. testing is hard
6. total_withholdings = 0 employees.each do |employee| if employee.government_retirement_withheld < MAX_GOVT_RETIREMENT government_retirement = compute_government_retirement(employee) end company_retirement = 0 if employee.wants_retirement && eligible_for_retirement(employee) company_retirement = get_retirement(employee) end gross_pay = compute_gross_pay(employee) personal_retirement = 0 if eligible_for_personal_retirement(employee) personal_retirement = personal_retirement_contribution(employee, company_retirement, gross_pay) end withholding = compute_withholding(employee) net_pay = gross_pay - withholding - company_retirement - government_retirement - personal_retirement pay_employee(employee, net_pay) total_withholdings = total_withholdings + withholding total_government_retirement = total_government_retirement + government_retirement total_retirement = total_retirement + company_retirement end save_pay_records(total_withholdings, total_government_retirement, total_retirement) 1 6 17
7. code coverage Code Coverage
8. def test_last_day_items_are_privacy_scoped_for_non_friends non_friend = create_user story = stories(:learning_no) story.published_at = 10.minutes.ago story.save! story = stories(:aaron_private_story) story.published_at = 5.minutes.ago story.save! items_for_non_friend = accounts(:quentin_and_aaron).last_day_items assert_privacy_status(items_for_non_friend, "Public") end You can’t test code that’s not there
9. Tests have bugs
10. def test_critical_functionality begin ... Bunch of stuff to exercise code ... # Commented out by Luke to fix test failure # assert "Some important assert", condition rescue # Don't let anything fail this test! end end
11. Developer testing isn’t very good at finding defects
12. Unit testing 50% 25% 75% Code reviews Code inspections Prototyping System test 100% 0% Defect Detection Rates of Selected Techniques
13. Unit tests Code reviews User testing Manual testing * Set overlap completely fabricated
14. Complements to developer testing
15. Manual testing
17. code review kitty is not pleased with your code
18. Growing better developers
19. Usability testing
20. The ultimate
21. From Don’t Make Me Think by Steve Krug
22. Don’t put all your eggs in one basket...
23. ...or you’ll end up as roadkill
24. Thanks! (You can yell at me over drinks.)
25. Jan Tik audreyjm529 aussiegall jpctalbot Flowizm Stuck in Customs GAV01 Andrew Dowsett hans.gerwitz