Writing Tests for PHP Source - php[tek] 2017

This talk was given at php[tek] 2017.

This isn’t a talk about adding tests to your PHP codebase, but adding tests to the PHP language itself. And the best part is, all the tests are written in PHP, so you don’t even have to dust off your old C book from college. In this talk, I will show you how to find untested parts of the PHP source code, how to write a test, and how to submit your tests to PHP internals. Just think, even without any C programming, you’ll be able to call yourself an internals contributor.

"TDD is dead": http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html
PHP source code on GitHub: https://github.com/php/php-src
runt-tests.php source code: https://github.com/php/php-src/blob/master/run-tests.php
PHP test naming conventions: http://qa.php.net/write-test.php#naming-conventions
Full list of section names: http://qa.php.net/phpt_details.php
PHP source code coverage: http://gcov.php.net/
ext/readline docs: http://php.net/readline
run-tests refactor audit: https://github.com/SammyK/run-tests
Internals discussion on run-tests refactor: https://externals.io/thread/888

Transcript

1. joind.in/talk/6c2ad @SammyK #phptek For PHP Source Writing Tests Sammy Kaye Powers 2017-05-25 PHP[TEK] 2017
2. joind.in/talk/6c2ad @SammyK #phptek Slides Get the joind.in/talk/6c2ad
3. Thanks to our Sponsors!
4. joind.in/talk/6c2ad @SammyK #phptek PHP Internals This is a talk
5. Write more TESTS!
6. joind.in/talk/6c2ad @SammyK #phptek Unit tests?
7. joind.in/talk/6c2ad @SammyK #phptek @dhh “TDD is dead.” http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html
8. joind.in/talk/6c2ad @SammyK #phptek
9. joind.in/talk/6c2ad @SammyK #phptek Integration tests? *Sort of
10. joind.in/talk/6c2ad @SammyK #phptek Acceptance tests?
11. joind.in/talk/6c2ad @SammyK #phptek Black-box Testing
12. joind.in/talk/6c2ad @SammyK #phptek Testing …you could probably argue that it is Integration
13. joind.in/talk/6c2ad @SammyK #phptek Testing …but it’s more like System
14. joind.in/talk/6c2ad @SammyK #phptek Black-box Testing? What is this
15. joind.in/talk/6c2ad @SammyK #phptek
16. joind.in/talk/6c2ad @SammyK #phptek What is in there?
17. joind.in/talk/6c2ad @SammyK #phptek Input Expected Output ✓
18. joind.in/talk/6c2ad @SammyK #phptek run-tests.php
19. joind.in/talk/6c2ad @SammyK #phptek .phpt Expected Hello World Output Hello World
20. joind.in/talk/6c2ad @SammyK #phptek run-tests.php Development
21. joind.in/talk/6c2ad @SammyK #phptek run-tests.php
25. joind.in/talk/6c2ad @SammyK #phptek
26. joind.in/talk/6c2ad @SammyK #phptek
27. joind.in/talk/6c2ad @SammyK #phptek run-tests.php Development
28. Legacy Code
29. joind.in/talk/6c2ad @SammyK #phptek Files
30. joind.in/talk/6c2ad @SammyK #phptek /ext/{name}/tests /Zend/tests /sapi/{name}/tests Extensions SAPI’s Zend Engine
32. joind.in/talk/6c2ad @SammyK #phptek Input Output
33. joind.in/talk/6c2ad @SammyK #phptek Running run-tests with
34. joind.in/talk/6c2ad @SammyK #phptek 1 source code Clone the
35. joind.in/talk/6c2ad @SammyK #phptek $ git clone \ git@github.com:php/php-src.git
37. joind.in/talk/6c2ad @SammyK #phptek 2source Compile from
39. joind.in/talk/6c2ad @SammyK #phptek $ ./buildconf
41. joind.in/talk/6c2ad @SammyK #phptek $ ./configure
42. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --disable-all
43. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --enable-debug \ --enable-phpdbg \ --enable-cli \ --enable-cgi \ --with-zlib
45. joind.in/talk/6c2ad @SammyK #phptek $ make
47. joind.in/talk/6c2ad @SammyK #phptek
48. $ sapi/cli/php --version
49. joind.in/talk/6c2ad @SammyK #phptek 3tests Run the
50. joind.in/talk/6c2ad @SammyK #phptek Two Ways to run tests
51. joind.in/talk/6c2ad @SammyK #phptek run-tests.php Run the with… directly
53. joind.in/talk/6c2ad @SammyK #phptek Run the with… the Makefile Recommended!
54. Recommended!
55. Recommended!
56. joind.in/talk/6c2ad @SammyK #phptek Let’s Make a test!
57. joind.in/talk/6c2ad @SammyK #phptek test-echo.phpt Save as… php Run from
59. joind.in/talk/6c2ad @SammyK #phptek Test it! $ make test TESTS=test-echo.phpt
61. joind.in/talk/6c2ad @SammyK #phptek Fail When tests
62. joind.in/talk/6c2ad @SammyK #phptek Input Output Expected
65. joind.in/talk/6c2ad @SammyK #phptek Output Examining the !==
66. Files test-echo.out test-echo.exp test-echo.diff test-echo.log Output
67. joind.in/talk/6c2ad @SammyK #phptek $ less test-echo.out Hello world Script output
68. joind.in/talk/6c2ad @SammyK #phptek $ less test-echo.exp Hello Jane! --EXPECT--
69. joind.in/talk/6c2ad @SammyK #phptek $ less test-echo.diff 001+ Hello world 001- Hello Jane!
70. joind.in/talk/6c2ad @SammyK #phptek $ less test-echo.log ---- EXPECTED OUTPUT Hello Jane! ---- ACTUAL OUTPUT Hello world ---- FAILED
71. Files test-echo.php test-echo.sh Executable
72. joind.in/talk/6c2ad @SammyK #phptek test-echo.php --FILE--
74. test-echo.sh
75. joind.in/talk/6c2ad @SammyK #phptek Example A more complex
79. joind.in/talk/6c2ad @SammyK #phptek Fails So say it
82. $ less test-dt.diff 001+ Boy you know I'm from Europe/Oslo 001- Boy you know I'm from America/Chicago
83. test-dt.php Executable!
85. test-dt.sh
88. joind.in/talk/6c2ad @SammyK #phptek Debugging $ gdb … failed tests with
91. #ShamelessPlug Tomorrow @ 9:30 am in Main Ballroom Dance The An Intro To Step Debugging in
92. joind.in/talk/6c2ad @SammyK #phptek Conventions Naming
93. joind.in/talk/6c2ad @SammyK #phptek {function-name}_basic.phpt {function-name}_error.phpt {function-name}_variation.phpt {ext-name}{number}.phpt bug{bug-id}.phpt http://qa.php.net/write-test.php#naming-conventions
94. joind.in/talk/6c2ad @SammyK #phptek Contribute How-to
95. joind.in/talk/6c2ad @SammyK #phptek Tests! Add some
96. gcov.php.net
97. PHP_FUNCTION
98. readline.c
99. joind.in/talk/6c2ad @SammyK #phptek ? readline()
100. joind.in/talk/6c2ad @SammyK #phptek php.net/readline
102. joind.in/talk/6c2ad @SammyK #phptek Let’s Make a test!
103. joind.in/talk/6c2ad @SammyK #phptek
104. joind.in/talk/6c2ad @SammyK #phptek for ext/readline Find the ext/readline/tests
106. joind.in/talk/6c2ad @SammyK #phptek Save as… ext/readline/tests/readline_basic.phpt
108. joind.in/talk/6c2ad @SammyK #phptek $ less ext/readline/tests/readline_basic.out Fatal error: Uncaught Error: Call to undefined function readline() in /Users/SammyK/Developer/php-src/ ext/readline/tests/ readline_basic.php:2
109. joind.in/talk/6c2ad @SammyK #phptek php.net/readline
110. joind.in/talk/6c2ad @SammyK #phptek
112. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --enable-debug \ --enable-phpdbg \ --enable-cli \ --enable-cgi \ --with-zlib
113. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --enable-debug \ --enable-phpdbg \ --enable-cli \ --enable-cgi \ --with-zlib \ --with-readline
114. joind.in/talk/6c2ad @SammyK #phptek OR
115. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --enable-debug \ --enable-phpdbg \ --enable-cli \ --enable-cgi \ --with-zlib \ --with-libedit
117. joind.in/talk/6c2ad @SammyK #phptek $ less ext/readline/tests/readline_basic.out You said:
118. joind.in/talk/6c2ad @SammyK #phptek readline_basic.sh Let’s try
120. joind.in/talk/6c2ad @SammyK #phptek
122. Now send a pull request
123. joind.in/talk/6c2ad @SammyK #phptek internals contributor You’re an
124. joind.in/talk/6c2ad @SammyK #phptek Shhh…. It’s a secret
125. Legacy Code
126. refactor github.com/SammyK/run-tests run-tests
127. refactor run-tests
128. wishlist * Concurrency * Unit tests * Colored output run-tests
129. joind.in/talk/6c2ad @SammyK #phptek Code with me tonight! Main Ballroom during community night after +
130. joind.in/talk/6c2ad @SammyK #phptek Pull Request to php-src yield by midnight ;
131. THANKS! SAMMY KAYE POWERS @SammyK SammyK.me Host of @PHPRoundtable @ChiPHPUG West Coast Swing Hire me! :) /talk/6c2ad