Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BALTO, ONE TEST OUTPUT FORMAT TO UNITE THEM ALL

410e3353165c33043ab69be7fc366428?s=47 Boris Feld
February 01, 2020

BALTO, ONE TEST OUTPUT FORMAT TO UNITE THEM ALL

Since several years, software quality tools have evolved, CI systems are more and more scalable, there are more testing libraries than ever and they are more mature than ever and we have seen the rise of new tools to improve the quality of code we craft.

Unfortunately, most of our CI system still launch a script and check the return code, most of the testing libraries don't allow to select finely which tests to launch and most of CI advanced innovations, parallel running, and remote execution, are not available to developers on their workstation.

Each language community has its own set of tools, libraries, and command-line and visual interfaces increasing the effort for developers to learn or learn again how to write, run and debug tests in each language.

How to improve the situation? In this talk, I will present one of my project LITF (https://github.com/Lothiraldan/litf) a new protocol for test running and test output as well as BALTO (https://github.com/lothiraldan/balto), a test orchestrator using this new format. Thanks to this new format, BALTO can execute several test suites in different languages, remotely on a Kubernetes cluster and all in parallel. In any case, this is the goal of the stable version.

410e3353165c33043ab69be7fc366428?s=128

Boris Feld

February 01, 2020
Tweet

Transcript

  1. BALTO, ONE TEST OUTPUT FORMAT TO UNITE BALTO, ONE TEST

    OUTPUT FORMAT TO UNITE THEM ALL THEM ALL Boris Feld - Fosdem 2020
  2. PRESENTATION PRESENTATION

  3. BORIS FELD BORIS FELD Python and testing fan. Former Mercurial

    reviewer. https://www.comet.ml/ @lothiraldan
  4. POLL: HOW MANY WRITE UNIT-TESTS? POLL: HOW MANY WRITE UNIT-TESTS?

  5. POLL: HOW MANY ONLY RUN UNIT-TESTS? POLL: HOW MANY ONLY

    RUN UNIT-TESTS?
  6. WE ARE NOT ALONE WE ARE NOT ALONE 2017 2018

    2019 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% Launching tests Writing tests => More than 70% write and run unit tests. Source: Jetbrains Developer Ecosystem Survey
  7. I DO BOTH I DO BOTH

  8. I WRITE TESTS I WRITE TESTS Mainly in Python but

    in JS too.
  9. I KNOW HOW TO WRITE TESTS + RUN I KNOW

    HOW TO WRITE TESTS + RUN THEM THEM I know how to run all tests, A single test le, A single test, Check why my new test is not running, Know where to nd the needed data on the output, Know how to debug a failing test ef ciently.
  10. PYTEST INTERFACE PYTEST INTERFACE ============================= test session starts ============================== platform

    linux2 -- Python 2.7.15, pytest-3.1.2, py-1.5.3, pluggy- 0.4.0 rootdir: /home/lothiraldan/Labo/presentations/balto_revolution_fr/src, inifile: collected 2 items test.py F. =================================== FAILURES =================================== ________________________ TestStringMethods.test_isupper ________________________ self = <test.TestStringMethods testMethod=test_isupper> def test_isupper(self): self.assertTrue('FOO'.isupper()) > self.assertTrue('Foo'.isupper()) E AssertionError: False is not true
  11. JEST INTERFACE JEST INTERFACE FAIL src/store.test.js • We can select

    and deselect a test leaf expect(received).toEqual(expected) Expected value to equal: {"files": [], "suites": [], "tests": ["my_test_dir::my_test_file::my_test_1"]} Received: {"my_suite": {"files": [], "nodeids": ["my_test_dir::my_test_file::my_test_1"]}} Difference: - Expected + Received Object { + "my_suite": Object { "files": Array [], - "suites": Array [], - "tests": Array [
  12. CI EXPERT CI EXPERT But I’m also the CI expert,

    so: I have to understand failures on test suites in other languages. I have to nd the needed data on the outputs. I have to be able to run the failing tests locally or in another server. All with an unfamiliar language, tool, and interface.
  13. WE NEED SOME COMMON TOOLING! WE NEED SOME COMMON TOOLING!

  14. WHAT DO WE WANT IN AN INTERFACE? WHAT DO WE

    WANT IN AN INTERFACE?
  15. MY CHRISTMAS LIST MY CHRISTMAS LIST Color Progress bar Relaunch

    failed tests only Launch speci c test(s) Web interface
  16. BALTO BALTO

  17. BALTO KESAKO? BALTO KESAKO? Balto is a Language Independent Test

    Orchestrator
  18. COLOR COLOR

  19. PROGRESS BAR PROGRESS BAR

  20. RELAUNCH FAILED TESTS ONLY RELAUNCH FAILED TESTS ONLY

  21. LAUNCH SPECIFIC TEST(S) LAUNCH SPECIFIC TEST(S)

  22. BALTO DEMO BALTO DEMO [Demo live]

  23. BALTO 101 BALTO 101

  24. THE SECRET THE SECRET The secret of Balto is that

    it is reading the stdout of its subprocesses.
  25. THE OTHER PART OF THE SECRET THE OTHER PART OF

    THE SECRET But the plugin, server, and UI need to understand each other.
  26. TO UNDERSTAND EACH OTHER TO UNDERSTAND EACH OTHER They need

    a common language.
  27. TEST OUTPUT FORMATS TEST OUTPUT FORMATS

  28. TEST OUTPUT FORMATS TEST OUTPUT FORMATS Junit.xml Tap Mozlog Subunit

  29. JUNIT.XML JUNIT.XML Based on XML One le generated at the

    end of the build Format tied to Junit => Non-streamable
  30. TAP TAP Famous in the Perl community Simple, hard to

    extend Format tied to TAP Perl => Need an independent parser in Python and JS
  31. MOZLOG MOZLOG Format used internally at Mozilla 1 message at

    the begging and another at the end of each test => Readers need to keep some state, 1 test == 1 message is easier for the consumer
  32. SUBUNIT SUBUNIT Closest to the design of LITF Is a

    binary streaming format with streaming lters capability An effort has been made to add the missing pieces into subunit. => No input eld
  33. MY CHRISTMAS LIST MY CHRISTMAS LIST Easy to write and

    read. Streamable so we do not need to wait for the end of the run. Format de ned outside of an implementation.
  34. A NEW FORMAT A NEW FORMAT

  35. WHAT DO WE NEED? WHAT DO WE NEED? Test name,

    Test status (failed, passed, skipped…), Error message in case of error.
  36. EXAMPLE EXAMPLE { "test_name": "test_success", "outcome": "passed" } { "test_name":

    "test_failed", "outcome": "failed", "error": { "humanrepr": "def test_fails():\n> assert False\nE assert False\n\ntest_func.py:9: AssertionError" } }
  37. MISSING PIECES MISSING PIECES { "test_number": 6, "_type": "session_start" }

    { "failed": 3, "total_duration": 0.03, "_type": "session_end", "skipped": 0, "error": 0, "passed": 3 }
  38. MORE DATA MORE DATA Timings Log messages Stdout and stderr

    Texte and image diff File, line and more
  39. LAUNCH A SPECIFIC TEST? LAUNCH A SPECIFIC TEST?

  40. INPUT VS OUTPUT INPUT VS OUTPUT We also need a

    new input format: { "collect_only": true } { "files": ["file1.py", "file2.py"], "nodes": ["file1.py::test_func"] }
  41. HOW TO GET A UNIQUE IDENTIFIER? HOW TO GET A

    UNIQUE IDENTIFIER? Let’s add a new output eld: { "test_name": "test_success", "outcome": "passed", "_id": "test_func.py::test_success" }
  42. FULL EXAMPLE FULL EXAMPLE { "test_name": "test_failed", "_id": "test_func.py::test_success", "outcome":

    "failed", "error": { "humanrepr": "def test_fails():\n> assert False\nE assert False\n\ntest_func.py:9: AssertionError" } }
  43. LITF LITF

  44. THE FORMAT 101 THE FORMAT 101 The Language Independent Test

    Format is de ned in its repository. https://github.com/Lothiraldan/litf
  45. MESSAGE DEFINITION MESSAGE DEFINITION Each message is a JSON-Schema le

    which de nes mandatory and required elds.
  46. HELPER SCRIPT HELPER SCRIPT The LITF repository contains a script

    to validate a LITF stream or le.
  47. MISSING DATA MISSING DATA Version number Binary data

  48. STATUS STATUS Beta, works for me™

  49. MORE LANGUAGES MORE LANGUAGES JS PHP Rust Java

  50. OTHER STREAM SOURCES OTHER STREAM SOURCES SSH Docker container

  51. ASYNC FOR THE WIN ASYNC FOR THE WIN As Balto

    is using the asynchronous capabilities of Python, we could run several test suites in parallel. Or even several test suites in parallel, written in several different languages on a distance server using Docker?
  52. ARCHITECTURE ARCHITECTURE Balto architecture Balto Pytest-Litf LITF over Subprocess Mercurial-Litf

    LITF over SSH Jest-Litf LITF over Docker Pytest Mercurial Jest
  53. CONCLUSION CONCLUSION

  54. A NEW PROTOCOL A NEW PROTOCOL Similar to HTTP or

    LSP, it’s hopefully a foundation that could be used for building tomorrow tools.
  55. EXAMPLES EXAMPLES You want a curses tool to run your

    tests? You can! You want a detailed report on the test timings? You can!
  56. INSTALL BALTO INSTALL BALTO Install pipx on your system Install

    Balto using pipx: pipx install balto Install the LITF plugin for you test-runner in your environment: pip install pytest-litf or npm install jest-litf Enjoy
  57. HOW YOU CAN HELP HOW YOU CAN HELP Feedback about

    LITF, Create new LITF producer plugin, Create new LITF consumer, Use Balto or speak about it.
  58. CONTRIBUTE CONTRIBUTE Balto and LITF are open-source: https://github.com/lothiraldan/litf https://github.com/lothiraldan/balto

  59. QUESTIONS? QUESTIONS?