Writing Tests for PHP Source - php[tek] 2017

Writing Tests for PHP Source - php[tek] 2017

This talk was given at php[tek] 2017.

This isn’t a talk about adding tests to your PHP codebase, but adding tests to the PHP language itself. And the best part is, all the tests are written in PHP, so you don’t even have to dust off your old C book from college. In this talk, I will show you how to find untested parts of the PHP source code, how to write a test, and how to submit your tests to PHP internals. Just think, even without any C programming, you’ll be able to call yourself an internals contributor.

"TDD is dead": http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html
PHP source code on GitHub: https://github.com/php/php-src
runt-tests.php source code: https://github.com/php/php-src/blob/master/run-tests.php
PHP test naming conventions: http://qa.php.net/write-test.php#naming-conventions
Full list of section names: http://qa.php.net/phpt_details.php
PHP source code coverage: http://gcov.php.net/
ext/readline docs: http://php.net/readline
run-tests refactor audit: https://github.com/SammyK/run-tests
Internals discussion on run-tests refactor: https://externals.io/thread/888

8c090cc1ccd623a146ddd9159b1bf7e2?s=128

Sammy Kaye Powers

May 25, 2017
Tweet

Transcript

  1. joind.in/talk/6c2ad @SammyK #phptek For PHP Source Writing Tests Sammy Kaye

    Powers 2017-05-25 PHP[TEK] 2017
  2. joind.in/talk/6c2ad @SammyK #phptek Slides Get the joind.in/talk/6c2ad

  3. Thanks to our Sponsors!

  4. joind.in/talk/6c2ad @SammyK #phptek PHP Internals This is a talk

  5. Write more TESTS!

  6. joind.in/talk/6c2ad @SammyK #phptek Unit tests?

  7. joind.in/talk/6c2ad @SammyK #phptek @dhh “TDD is dead.” http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html

  8. joind.in/talk/6c2ad @SammyK #phptek

  9. joind.in/talk/6c2ad @SammyK #phptek Integration tests? *Sort of

  10. joind.in/talk/6c2ad @SammyK #phptek Acceptance tests?

  11. joind.in/talk/6c2ad @SammyK #phptek Black-box Testing

  12. joind.in/talk/6c2ad @SammyK #phptek Testing …you could probably argue that it

    is Integration
  13. joind.in/talk/6c2ad @SammyK #phptek Testing …but it’s more like System

  14. joind.in/talk/6c2ad @SammyK #phptek Black-box Testing? What is this

  15. joind.in/talk/6c2ad @SammyK #phptek

  16. joind.in/talk/6c2ad @SammyK #phptek What is in there?

  17. joind.in/talk/6c2ad @SammyK #phptek Input Expected Output ✓

  18. joind.in/talk/6c2ad @SammyK #phptek run-tests.php

  19. joind.in/talk/6c2ad @SammyK #phptek .phpt Expected Hello World Output Hello World

  20. joind.in/talk/6c2ad @SammyK #phptek run-tests.php Development

  21. joind.in/talk/6c2ad @SammyK #phptek run-tests.php

  22. None
  23. None
  24. None
  25. joind.in/talk/6c2ad @SammyK #phptek

  26. joind.in/talk/6c2ad @SammyK #phptek

  27. joind.in/talk/6c2ad @SammyK #phptek run-tests.php Development

  28. Legacy Code

  29. joind.in/talk/6c2ad @SammyK #phptek Files

  30. joind.in/talk/6c2ad @SammyK #phptek /ext/{name}/tests /Zend/tests /sapi/{name}/tests Extensions SAPI’s Zend Engine

  31. None
  32. joind.in/talk/6c2ad @SammyK #phptek Input Output

  33. joind.in/talk/6c2ad @SammyK #phptek Running run-tests with

  34. joind.in/talk/6c2ad @SammyK #phptek 1 source code Clone the

  35. joind.in/talk/6c2ad @SammyK #phptek $ git clone \ git@github.com:php/php-src.git

  36. None
  37. joind.in/talk/6c2ad @SammyK #phptek 2source Compile from

  38. None
  39. joind.in/talk/6c2ad @SammyK #phptek $ ./buildconf

  40. None
  41. joind.in/talk/6c2ad @SammyK #phptek $ ./configure

  42. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --disable-all

  43. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --enable-debug \ --enable-phpdbg \

    --enable-cli \ --enable-cgi \ --with-zlib
  44. None
  45. joind.in/talk/6c2ad @SammyK #phptek $ make

  46. None
  47. joind.in/talk/6c2ad @SammyK #phptek

  48. $ sapi/cli/php --version

  49. joind.in/talk/6c2ad @SammyK #phptek 3tests Run the

  50. joind.in/talk/6c2ad @SammyK #phptek Two Ways to run tests

  51. joind.in/talk/6c2ad @SammyK #phptek run-tests.php Run the with… directly

  52. None
  53. joind.in/talk/6c2ad @SammyK #phptek Run the with… the Makefile Recommended!

  54. Recommended!

  55. Recommended!

  56. joind.in/talk/6c2ad @SammyK #phptek Let’s Make a test!

  57. joind.in/talk/6c2ad @SammyK #phptek test-echo.phpt Save as… php Run from

  58. None
  59. joind.in/talk/6c2ad @SammyK #phptek Test it! $ make test TESTS=test-echo.phpt

  60. None
  61. joind.in/talk/6c2ad @SammyK #phptek Fail When tests

  62. joind.in/talk/6c2ad @SammyK #phptek Input Output Expected

  63. None
  64. None
  65. joind.in/talk/6c2ad @SammyK #phptek Output Examining the !==

  66. Files test-echo.out test-echo.exp test-echo.diff test-echo.log Output

  67. joind.in/talk/6c2ad @SammyK #phptek $ less test-echo.out Hello world Script output

  68. joind.in/talk/6c2ad @SammyK #phptek $ less test-echo.exp Hello Jane! --EXPECT--

  69. joind.in/talk/6c2ad @SammyK #phptek $ less test-echo.diff 001+ Hello world 001-

    Hello Jane!
  70. joind.in/talk/6c2ad @SammyK #phptek $ less test-echo.log ---- EXPECTED OUTPUT Hello

    Jane! ---- ACTUAL OUTPUT Hello world ---- FAILED
  71. Files test-echo.php test-echo.sh Executable

  72. joind.in/talk/6c2ad @SammyK #phptek test-echo.php --FILE--

  73. None
  74. test-echo.sh

  75. joind.in/talk/6c2ad @SammyK #phptek Example A more complex

  76. None
  77. None
  78. None
  79. joind.in/talk/6c2ad @SammyK #phptek Fails So say it

  80. None
  81. None
  82. $ less test-dt.diff 001+ Boy you know I'm from Europe/Oslo

    001- Boy you know I'm from America/Chicago
  83. test-dt.php Executable!

  84. None
  85. test-dt.sh

  86. None
  87. None
  88. joind.in/talk/6c2ad @SammyK #phptek Debugging $ gdb … failed tests with

  89. None
  90. None
  91. #ShamelessPlug Tomorrow @ 9:30 am in Main Ballroom Dance The

    An Intro To Step Debugging in
  92. joind.in/talk/6c2ad @SammyK #phptek Conventions Naming

  93. joind.in/talk/6c2ad @SammyK #phptek {function-name}_basic.phpt {function-name}_error.phpt {function-name}_variation.phpt {ext-name}{number}.phpt bug{bug-id}.phpt http://qa.php.net/write-test.php#naming-conventions

  94. joind.in/talk/6c2ad @SammyK #phptek Contribute How-to

  95. joind.in/talk/6c2ad @SammyK #phptek Tests! Add some

  96. gcov.php.net

  97. PHP_FUNCTION

  98. readline.c

  99. joind.in/talk/6c2ad @SammyK #phptek ? readline()

  100. joind.in/talk/6c2ad @SammyK #phptek php.net/readline

  101. None
  102. joind.in/talk/6c2ad @SammyK #phptek Let’s Make a test!

  103. joind.in/talk/6c2ad @SammyK #phptek

  104. joind.in/talk/6c2ad @SammyK #phptek for ext/readline Find the ext/readline/tests

  105. None
  106. joind.in/talk/6c2ad @SammyK #phptek Save as… ext/readline/tests/readline_basic.phpt

  107. None
  108. joind.in/talk/6c2ad @SammyK #phptek $ less ext/readline/tests/readline_basic.out Fatal error: Uncaught Error:

    Call to undefined function readline() in /Users/SammyK/Developer/php-src/ ext/readline/tests/ readline_basic.php:2
  109. joind.in/talk/6c2ad @SammyK #phptek php.net/readline

  110. joind.in/talk/6c2ad @SammyK #phptek

  111. None
  112. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --enable-debug \ --enable-phpdbg \

    --enable-cli \ --enable-cgi \ --with-zlib
  113. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --enable-debug \ --enable-phpdbg \

    --enable-cli \ --enable-cgi \ --with-zlib \ --with-readline
  114. joind.in/talk/6c2ad @SammyK #phptek OR

  115. joind.in/talk/6c2ad @SammyK #phptek $ ./configure \ --enable-debug \ --enable-phpdbg \

    --enable-cli \ --enable-cgi \ --with-zlib \ --with-libedit
  116. None
  117. joind.in/talk/6c2ad @SammyK #phptek $ less ext/readline/tests/readline_basic.out You said:

  118. joind.in/talk/6c2ad @SammyK #phptek readline_basic.sh Let’s try

  119. None
  120. joind.in/talk/6c2ad @SammyK #phptek

  121. None
  122. Now send a pull request

  123. joind.in/talk/6c2ad @SammyK #phptek internals contributor You’re an

  124. joind.in/talk/6c2ad @SammyK #phptek Shhh…. It’s a secret

  125. Legacy Code

  126. refactor github.com/SammyK/run-tests run-tests

  127. refactor run-tests

  128. wishlist * Concurrency * Unit tests * Colored output run-tests

  129. joind.in/talk/6c2ad @SammyK #phptek Code with me tonight! Main Ballroom during

    community night after +
  130. joind.in/talk/6c2ad @SammyK #phptek Pull Request to php-src yield by midnight

    ;
  131. THANKS! SAMMY KAYE POWERS @SammyK SammyK.me Host of @PHPRoundtable @ChiPHPUG

    West Coast Swing Hire me! :) /talk/6c2ad