Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Writing Tests for PHP Source - php[tek] 2017

Writing Tests for PHP Source - php[tek] 2017

This talk was given at php[tek] 2017.

This isn’t a talk about adding tests to your PHP codebase, but adding tests to the PHP language itself. And the best part is, all the tests are written in PHP, so you don’t even have to dust off your old C book from college. In this talk, I will show you how to find untested parts of the PHP source code, how to write a test, and how to submit your tests to PHP internals. Just think, even without any C programming, you’ll be able to call yourself an internals contributor.

"TDD is dead": http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html
PHP source code on GitHub: https://github.com/php/php-src
runt-tests.php source code: https://github.com/php/php-src/blob/master/run-tests.php
PHP test naming conventions: http://qa.php.net/write-test.php#naming-conventions
Full list of section names: http://qa.php.net/phpt_details.php
PHP source code coverage: http://gcov.php.net/
ext/readline docs: http://php.net/readline
run-tests refactor audit: https://github.com/SammyK/run-tests
Internals discussion on run-tests refactor: https://externals.io/thread/888

Sammy Kaye Powers

May 25, 2017
Tweet

More Decks by Sammy Kaye Powers

Other Decks in Programming

Transcript

  1. joind.in/talk/6c2ad
    @SammyK #phptek
    For PHP Source
    Writing Tests
    Sammy Kaye Powers
    2017-05-25
    PHP[TEK] 2017

    View Slide

  2. joind.in/talk/6c2ad
    @SammyK #phptek
    Slides
    Get the
    joind.in/talk/6c2ad

    View Slide

  3. Thanks to our
    Sponsors!

    View Slide

  4. joind.in/talk/6c2ad
    @SammyK #phptek
    PHP Internals
    This is a
    talk

    View Slide

  5. Write more
    TESTS!

    View Slide

  6. joind.in/talk/6c2ad
    @SammyK #phptek
    Unit tests?

    View Slide

  7. joind.in/talk/6c2ad
    @SammyK #phptek
    @dhh
    “TDD is dead.”
    http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html

    View Slide

  8. joind.in/talk/6c2ad
    @SammyK #phptek

    View Slide

  9. joind.in/talk/6c2ad
    @SammyK #phptek
    Integration tests?
    *Sort of

    View Slide

  10. joind.in/talk/6c2ad
    @SammyK #phptek
    Acceptance tests?

    View Slide

  11. joind.in/talk/6c2ad
    @SammyK #phptek
    Black-box
    Testing

    View Slide

  12. joind.in/talk/6c2ad
    @SammyK #phptek
    Testing
    …you could probably argue that it is
    Integration

    View Slide

  13. joind.in/talk/6c2ad
    @SammyK #phptek
    Testing
    …but it’s more like
    System

    View Slide

  14. joind.in/talk/6c2ad
    @SammyK #phptek
    Black-box
    Testing?
    What is this

    View Slide

  15. joind.in/talk/6c2ad
    @SammyK #phptek

    View Slide

  16. joind.in/talk/6c2ad
    @SammyK #phptek
    What is
    in there?

    View Slide

  17. joind.in/talk/6c2ad
    @SammyK #phptek
    Input
    Expected
    Output

    View Slide

  18. joind.in/talk/6c2ad
    @SammyK #phptek
    run-tests.php

    View Slide

  19. joind.in/talk/6c2ad
    @SammyK #phptek
    .phpt
    Expected
    Hello
    World
    Output
    Hello
    World

    View Slide

  20. joind.in/talk/6c2ad
    @SammyK #phptek
    run-tests.php
    Development

    View Slide

  21. joind.in/talk/6c2ad
    @SammyK #phptek
    run-tests.php

    View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. joind.in/talk/6c2ad
    @SammyK #phptek

    View Slide

  26. joind.in/talk/6c2ad
    @SammyK #phptek

    View Slide

  27. joind.in/talk/6c2ad
    @SammyK #phptek
    run-tests.php
    Development

    View Slide

  28. Legacy
    Code

    View Slide

  29. joind.in/talk/6c2ad
    @SammyK #phptek
    Files

    View Slide

  30. joind.in/talk/6c2ad
    @SammyK #phptek
    /ext/{name}/tests
    /Zend/tests /sapi/{name}/tests
    Extensions
    SAPI’s
    Zend Engine

    View Slide

  31. View Slide

  32. joind.in/talk/6c2ad
    @SammyK #phptek
    Input Output

    View Slide

  33. joind.in/talk/6c2ad
    @SammyK #phptek
    Running
    run-tests
    with

    View Slide

  34. joind.in/talk/6c2ad
    @SammyK #phptek
    1 source code
    Clone the

    View Slide

  35. joind.in/talk/6c2ad
    @SammyK #phptek
    $ git clone \
    [email protected]:php/php-src.git

    View Slide

  36. View Slide

  37. joind.in/talk/6c2ad
    @SammyK #phptek
    2source
    Compile from

    View Slide

  38. View Slide

  39. joind.in/talk/6c2ad
    @SammyK #phptek
    $ ./buildconf

    View Slide

  40. View Slide

  41. joind.in/talk/6c2ad
    @SammyK #phptek
    $ ./configure

    View Slide

  42. joind.in/talk/6c2ad
    @SammyK #phptek
    $ ./configure \
    --disable-all

    View Slide

  43. joind.in/talk/6c2ad
    @SammyK #phptek
    $ ./configure \
    --enable-debug \
    --enable-phpdbg \
    --enable-cli \
    --enable-cgi \
    --with-zlib

    View Slide

  44. View Slide

  45. joind.in/talk/6c2ad
    @SammyK #phptek
    $ make

    View Slide

  46. View Slide

  47. joind.in/talk/6c2ad
    @SammyK #phptek

    View Slide

  48. $ sapi/cli/php --version

    View Slide

  49. joind.in/talk/6c2ad
    @SammyK #phptek
    3tests
    Run the

    View Slide

  50. joind.in/talk/6c2ad
    @SammyK #phptek
    Two Ways
    to run
    tests

    View Slide

  51. joind.in/talk/6c2ad
    @SammyK #phptek
    run-tests.php
    Run the with…
    directly

    View Slide

  52. View Slide

  53. joind.in/talk/6c2ad
    @SammyK #phptek
    Run the with…
    the Makefile
    Recommended!

    View Slide

  54. Recommended!

    View Slide

  55. Recommended!

    View Slide

  56. joind.in/talk/6c2ad
    @SammyK #phptek
    Let’s
    Make a test!

    View Slide

  57. joind.in/talk/6c2ad
    @SammyK #phptek
    test-echo.phpt
    Save as…
    php
    Run from

    View Slide

  58. View Slide

  59. joind.in/talk/6c2ad
    @SammyK #phptek
    Test it!
    $ make test TESTS=test-echo.phpt

    View Slide

  60. View Slide

  61. joind.in/talk/6c2ad
    @SammyK #phptek
    Fail
    When tests

    View Slide

  62. joind.in/talk/6c2ad
    @SammyK #phptek
    Input Output
    Expected

    View Slide

  63. View Slide

  64. View Slide

  65. joind.in/talk/6c2ad
    @SammyK #phptek
    Output
    Examining the
    !==

    View Slide

  66. Files
    test-echo.out
    test-echo.exp
    test-echo.diff
    test-echo.log
    Output

    View Slide

  67. joind.in/talk/6c2ad
    @SammyK #phptek
    $ less test-echo.out
    Hello world
    Script output

    View Slide

  68. joind.in/talk/6c2ad
    @SammyK #phptek
    $ less test-echo.exp
    Hello Jane!
    --EXPECT--

    View Slide

  69. joind.in/talk/6c2ad
    @SammyK #phptek
    $ less test-echo.diff
    001+ Hello world
    001- Hello Jane!

    View Slide

  70. joind.in/talk/6c2ad
    @SammyK #phptek
    $ less test-echo.log
    ---- EXPECTED OUTPUT
    Hello Jane!
    ---- ACTUAL OUTPUT
    Hello world
    ---- FAILED

    View Slide

  71. Files
    test-echo.php
    test-echo.sh
    Executable

    View Slide

  72. joind.in/talk/6c2ad
    @SammyK #phptek
    test-echo.php
    --FILE--

    View Slide

  73. View Slide

  74. test-echo.sh

    View Slide

  75. joind.in/talk/6c2ad
    @SammyK #phptek
    Example
    A more complex

    View Slide

  76. View Slide

  77. View Slide

  78. View Slide

  79. joind.in/talk/6c2ad
    @SammyK #phptek
    Fails
    So say it

    View Slide

  80. View Slide

  81. View Slide

  82. $ less test-dt.diff
    001+ Boy you know I'm
    from Europe/Oslo
    001- Boy you know I'm
    from America/Chicago

    View Slide

  83. test-dt.php
    Executable!

    View Slide

  84. View Slide

  85. test-dt.sh

    View Slide

  86. View Slide

  87. View Slide

  88. joind.in/talk/6c2ad
    @SammyK #phptek
    Debugging
    $ gdb …
    failed tests with

    View Slide

  89. View Slide

  90. View Slide

  91. #ShamelessPlug
    Tomorrow @ 9:30 am
    in Main Ballroom
    Dance
    The
    An Intro To
    Step Debugging in

    View Slide

  92. joind.in/talk/6c2ad
    @SammyK #phptek
    Conventions
    Naming

    View Slide

  93. joind.in/talk/6c2ad
    @SammyK #phptek
    {function-name}_basic.phpt
    {function-name}_error.phpt
    {function-name}_variation.phpt
    {ext-name}{number}.phpt
    bug{bug-id}.phpt
    http://qa.php.net/write-test.php#naming-conventions

    View Slide

  94. joind.in/talk/6c2ad
    @SammyK #phptek
    Contribute
    How-to

    View Slide

  95. joind.in/talk/6c2ad
    @SammyK #phptek
    Tests!
    Add some

    View Slide

  96. gcov.php.net

    View Slide

  97. PHP_FUNCTION

    View Slide

  98. readline.c

    View Slide

  99. joind.in/talk/6c2ad
    @SammyK #phptek
    ?
    readline()

    View Slide

  100. joind.in/talk/6c2ad
    @SammyK #phptek
    php.net/readline

    View Slide

  101. View Slide

  102. joind.in/talk/6c2ad
    @SammyK #phptek
    Let’s
    Make a test!

    View Slide

  103. joind.in/talk/6c2ad
    @SammyK #phptek

    View Slide

  104. joind.in/talk/6c2ad
    @SammyK #phptek
    for ext/readline
    Find the
    ext/readline/tests

    View Slide

  105. View Slide

  106. joind.in/talk/6c2ad
    @SammyK #phptek
    Save as…
    ext/readline/tests/readline_basic.phpt

    View Slide

  107. View Slide

  108. joind.in/talk/6c2ad
    @SammyK #phptek
    $ less ext/readline/tests/readline_basic.out
    Fatal error: Uncaught Error: Call
    to undefined function readline() in
    /Users/SammyK/Developer/php-src/
    ext/readline/tests/
    readline_basic.php:2

    View Slide

  109. joind.in/talk/6c2ad
    @SammyK #phptek
    php.net/readline

    View Slide

  110. joind.in/talk/6c2ad
    @SammyK #phptek

    View Slide

  111. View Slide

  112. joind.in/talk/6c2ad
    @SammyK #phptek
    $ ./configure \
    --enable-debug \
    --enable-phpdbg \
    --enable-cli \
    --enable-cgi \
    --with-zlib

    View Slide

  113. joind.in/talk/6c2ad
    @SammyK #phptek
    $ ./configure \
    --enable-debug \
    --enable-phpdbg \
    --enable-cli \
    --enable-cgi \
    --with-zlib \
    --with-readline

    View Slide

  114. joind.in/talk/6c2ad
    @SammyK #phptek
    OR

    View Slide

  115. joind.in/talk/6c2ad
    @SammyK #phptek
    $ ./configure \
    --enable-debug \
    --enable-phpdbg \
    --enable-cli \
    --enable-cgi \
    --with-zlib \
    --with-libedit

    View Slide

  116. View Slide

  117. joind.in/talk/6c2ad
    @SammyK #phptek
    $ less ext/readline/tests/readline_basic.out
    You said:

    View Slide

  118. joind.in/talk/6c2ad
    @SammyK #phptek
    readline_basic.sh
    Let’s try

    View Slide

  119. View Slide

  120. joind.in/talk/6c2ad
    @SammyK #phptek

    View Slide

  121. View Slide

  122. Now send a
    pull request

    View Slide

  123. joind.in/talk/6c2ad
    @SammyK #phptek
    internals contributor
    You’re an

    View Slide

  124. joind.in/talk/6c2ad
    @SammyK #phptek
    Shhh….
    It’s a secret

    View Slide

  125. Legacy
    Code

    View Slide

  126. refactor
    github.com/SammyK/run-tests
    run-tests

    View Slide

  127. refactor
    run-tests

    View Slide

  128. wishlist
    * Concurrency
    * Unit tests
    * Colored output
    run-tests

    View Slide

  129. joind.in/talk/6c2ad
    @SammyK #phptek
    Code
    with me tonight!
    Main Ballroom
    during community night
    after
    +

    View Slide

  130. joind.in/talk/6c2ad
    @SammyK #phptek
    Pull
    Request
    to php-src
    yield
    by midnight
    ;

    View Slide

  131. THANKS!
    SAMMY KAYE POWERS
    @SammyK
    SammyK.me
    Host of @PHPRoundtable
    @ChiPHPUG
    West Coast Swing
    Hire me! :)
    /talk/6c2ad

    View Slide