A primer on testing database-driven applications

Interested in learning more about this topic? Read the overview of my research to learn more: https://www.gregorykapfhammer.com/research/

Transcript

1. A Primer on Testing Database-Driven Applications Gregory M. Kapfhammer Department of Computer Science Allegheny College 3rd Biennial Intramural Faculty Conference May 18 - 19, 2004 Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 1/23
2. Outline Introduction and Motivation Testing Challenges Database-Driven Applications A Unified Representation Test Adequacy Criteria Test Suite Execution Test Coverage Monitoring Conclusions and Resources Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 2/23
3. Invocation Thus spake the master, “Any program, no matter how small, contains bugs.” The novice did not believe the master’s words. “What if the program were so small that it performed a single function?” The master thoughtfully responded, “Such a program would have no meaning. But, if such a one existed, the operating system would fail eventually, producing a bug.” Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 3/23
4. Invocation But the novice was not satisfied. “What if the operating system did not fail?” The master responded, “There is no operating system that does not fail. But if such a one existed, the hardware would fail eventually, producing a bug.” The novice still was not satisfied. “What if the hardware did not fail?” Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 4/23
5. Invocation The master gave a great sigh. “There is no hardware that does not fail. But if such a one existed, the user would want the program to do something different, and this too is a bug.” A program without bugs would be an absurdity, a nonesuch. If there were a program without any bugs then the world would cease to exist. Geoffrey James The Zen of Programming (Adaptation) Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 5/23
6. Life of a Program: Compilation P Source Code P Byte Code Compiler The programming language compiler produces a representation of a program that can be executed Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 6/23
7. Life of a Program: Execution P Virtual Machine Byte Code Input a 5 print ... exit Final Result: 45 Output The virtual machine executes the program’s byte codes Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 7/23
8. Testing Challenges I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra, 1968 Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 8/23
9. Where Bugs Live Machine P Virtual File System System Operating Database Byte Code Execution Environment Input a 5 print ... exit Final Result: 45 Output Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 9/23
10. Motivation The Risks Digest, Volume 22, Issue 64, 2003 Jeppesen reports airspace boundary problems About 350 airspace boundaries contained in Jeppesen NavData are incorrect, the FAA has warned. The error occurred at Jeppesen after a software upgrade when information was pulled from a database containing 20,000 airspace boundaries worldwide for the March NavData update, which takes effect March 20. Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 10/23
11. More Testing Challenges Should consider the environment in which software applications execute Must test a program and its interaction with a database Database-driven application’s state space is well-structured, but infinite (Chays et al.) Need to show program does not violate database integrity, where integrity = consistency + validity (Motro) Must locate program and database coupling points that vary in granularity Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 11/23
12. Database-Driven Applications P m i m j Dl Dk R R2 1 E F G H A B C D I R3 J K L Program P interacts with two relational databases Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 12/23
13. Database Interaction Levels P Database Level 1 n D D A program can interact with a database at different levels of granularity Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 13/23
14. Database Interaction Levels UserInfo user_name 4 acct_lock 1 Brian Zorman 2 Robert Roos 3 card_number pin_number Geoffrey Arnold 0 0 0 0 32142 41601 45322 56471 Marcus Bittman Relation Level P D1 Dn Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 13/23
15. Database Interaction Levels UserInfo user_name 4 acct_lock 1 Brian Zorman 2 Robert Roos 3 card_number pin_number Geoffrey Arnold 0 0 0 0 32142 41601 45322 56471 Marcus Bittman Record Level P n D1 D Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 13/23
16. Database Interaction Levels UserInfo 4 acct_lock 1 Brian Zorman 2 Robert Roos 3 card_number pin_number 0 0 0 0 32142 41601 45322 56471 user_name Attribute Level Marcus Bittman Geoffrey Arnold P D1 Dn Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 13/23
17. Database Interaction Levels UserInfo 4 acct_lock 1 Brian Zorman 2 Robert Roos 3 card_number pin_number Geoffrey Arnold 0 0 0 0 32142 41601 45322 56471 user_name Attribute Value Level Marcus Bittman P D1 n D Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 13/23
18. Database Interaction Points select A1 , A2 , . . . , Aq from r1 , r2 , . . . , rm where Q delete from r where Q insert into r(A1 , A2 , . . . , Aq ) values(v1 , v2 , . . . , vq ) update r set Al = F(Al ) where Q Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 14/23
19. Refined Database-Driven Application P m i m j R R2 1 E F G H A B C D l D k D where set J = 500 update L < 1000 R3 select 1 * from R R2 from ) select where A < ( avg(G) I R3 J K L Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 15/23
20. Data Flow Information Interaction point: ‘‘UPDATE UserInfo SET acct lock=1 WHERE card number=’’ + card number + ‘‘;’’; Database Level: define(BankDB) Attribute Level: define(acct_lock) and use(card_number) Data flow information varies with respect to the granularity of the database interaction Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 16/23
21. Test Adequacy Criteria all−attribute−value−DUs all−record−DUs all−attribute−DUs all−relation−DUs all−database−DUs Database interaction association (DIA) involves the def and use of a database entity DIAs can be located in the DICFG with data flow analysis all-database-DUs requires tests to exercise all DIAs for all of the accessed databases Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 17/23
22. Generating Test Requirements Database Seeder Database (P) Test Adequacy Criterion (C) System Under Test Test Case Specification Relational Schema Requirements Test Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 18/23
23. Tracking Covered DIGs and DIAs DB P DIGr m i m j DIGs DIG # 1 DEF USE { ... } { ... } q { ... } { ... } 1 1 2 2 TEST { ... } { ... } COV? DIG Coverage Table DIA coverage can be tracked by recording which DIGs within a DICFG were executed during testing Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 19/23
24. Types of Test Suites T T ∆ 1 e m 1 ∆ 1 e ∆ 0 m e ∆e−1 Independent ∆ m T1 1 0 ∆1 ε−1 ∆ Tε ε m ε ∆ e ∆ e m Te ∆e Partially Independent T T ∆ 1 e m 1 ∆ 1 e ∆ 0 m e ∆e−1 Non-restricted – p. 20/23
25. Calculating Adequacy m i mj m i DIA <def(e1), use(e1)> <def(e2), use(e2)> COV? <def(e3), use(e3)> <def(e4), use(e4)> Test Requirements DIA COV? <def(e5), use(e5)> <def(e6), use(e6)> <def(e7), use(e7)> <def(e8), use(e8)> <def(e9), use(e9)> <def(e10), use(e10)> Test Requirements m j Tf cov(mi ) = 2 4 cov(mj ) = 4 6 cov(Tf ) = 6 10 – p. 21/23
26. Conclusions Software testing is hard, especially when interaction with the application’s environment is considered Must test the program’s interaction with the database Many challenges associated with (1) unified program representation, (2) test adequacy criteria, (3) test coverage monitoring, (4) test suite execution Unique family of test adequacy criteria to ensure that test suites detect violations of database validity and completeness Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 22/23
27. Resources Gregory M. Kapfhammer and Mary Lou Soffa. A Family of Test Adequacy Criteria for Database-Driven Applications. In Proceedings of the ACM SIGSOFT International Conference on the Foundations of Software Engineering, 2003. Gregory M. Kapfhammer. Software Testing. CRC Press Computer Science Handbook. June, 2004. http://cs.allegheny.edu/˜gkapfham/research/diatoms/ Database drIven Application T esting tOol ModuleS, Allegheny Faculty Conference, May 18, 2004 – p. 23/23