How PostgreSQL is tested

How do we know PostgreSQL works? What do "regression tests", "isolation tests", "TAP tests" do? What does the build farm do? What is the community's role in testing? What are the strengths and weaknesses of the testing methodologies and tools and how can we improve them?

I invite all existing hackers to join me so we can share our experiences and visions. For users who are not yet hackers, this presentation is a great way to learn about important development tools and practices of the PostgreSQL hacker community.

Transcript

1. How PostgreSQL is tested Peter Eisentraut peter@eisentraut.org @petereisentraut
2. POSTGRES 4.2
3. src/test/regress/
4. sql/ results/ expected/
5. make installcheck
6. make check 7.0 install before test parallel
7. pg_regress
8. Problems can only test SQL-level things must be deterministic plan differences platform differences configuration differences changes in external dependencies inverts TDD approach
9. Module tests 6.5: mb tests 7.1: intarray, cube, seg ...
10. ECPG tests 8.2
11. Build farm http://buildfarm.postgresql.org/ 8.0 ... 8.3 ...
12. Why test?
13. The perfect patch code documentation tests commit message
14. make check-world 9.0
15. Isolation tests 9.1
16. pg_upgrade tests 9.2
17. libpq (URL) tests 9.2
18. TAP/Perl tests 9.4: frontends 9.5: SSL 9.6: replication
19. Testing zoo pg_regress ecpg isolationtester pg_upgrade tests libpq tests TAP/Perl
20. Other testing Debian packaging QA teams users
21. Coverity, scan-build, Valgrind, ... see PGCon 2014
22. Test coverage make coverage-clean make check make coverage-html http://coverage.postgresql.org/ http://blog.2ndquadrant.com/code-coverage/ http://pgci.eisentraut.org/jenkins/job/postgresql_master_coverage/
23. Gaps pg_dump pg_upgrade more details pg_upgrade cross-version other clients/tools indexes vacuum crash recovery
24. More gaps configurations protocol access control replication archiving
25. Even more gaps performance packaging extensions
26. Test infrastructure goals test discovery test result reporting test performance running tests partially tests that don't run everywhere