BALTO, ONE TEST OUTPUT FORMAT TO UNITE THEM ALL

Since several years, software quality tools have evolved, CI systems are more and more scalable, there are more testing libraries than ever and they are more mature than ever and we have seen the rise of new tools to improve the quality of code we craft.

Unfortunately, most of our CI system still launch a script and check the return code, most of the testing libraries don't allow to select finely which tests to launch and most of CI advanced innovations, parallel running, and remote execution, are not available to developers on their workstation.

Each language community has its own set of tools, libraries, and command-line and visual interfaces increasing the effort for developers to learn or learn again how to write, run and debug tests in each language.

How to improve the situation? In this talk, I will present one of my project LITF (https://github.com/Lothiraldan/litf) a new protocol for test running and test output as well as BALTO (https://github.com/lothiraldan/balto), a test orchestrator using this new format. Thanks to this new format, BALTO can execute several test suites in different languages, remotely on a Kubernetes cluster and all in parallel. In any case, this is the goal of the stable version.

Transcript

1. BALTO, ONE TEST OUTPUT FORMAT TO UNITE BALTO, ONE TEST OUTPUT FORMAT TO UNITE THEM ALL THEM ALL Boris Feld - Fosdem 2020
2. PRESENTATION PRESENTATION
3. BORIS FELD BORIS FELD Python and testing fan. Former Mercurial reviewer. https://www.comet.ml/ @lothiraldan
4. POLL: HOW MANY WRITE UNIT-TESTS? POLL: HOW MANY WRITE UNIT-TESTS?
5. POLL: HOW MANY ONLY RUN UNIT-TESTS? POLL: HOW MANY ONLY RUN UNIT-TESTS?
6. WE ARE NOT ALONE WE ARE NOT ALONE 2017 2018 2019 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% Launching tests Writing tests => More than 70% write and run unit tests. Source: Jetbrains Developer Ecosystem Survey
7. I DO BOTH I DO BOTH
8. I WRITE TESTS I WRITE TESTS Mainly in Python but in JS too.
9. I KNOW HOW TO WRITE TESTS + RUN I KNOW HOW TO WRITE TESTS + RUN THEM THEM I know how to run all tests, A single test le, A single test, Check why my new test is not running, Know where to nd the needed data on the output, Know how to debug a failing test ef ciently.
10. PYTEST INTERFACE PYTEST INTERFACE ============================= test session starts ============================== platform linux2 -- Python 2.7.15, pytest-3.1.2, py-1.5.3, pluggy- 0.4.0 rootdir: /home/lothiraldan/Labo/presentations/balto_revolution_fr/src, inifile: collected 2 items test.py F. =================================== FAILURES =================================== ________________________ TestStringMethods.test_isupper ________________________ self = <test.TestStringMethods testMethod=test_isupper> def test_isupper(self): self.assertTrue('FOO'.isupper()) > self.assertTrue('Foo'.isupper()) E AssertionError: False is not true
11. JEST INTERFACE JEST INTERFACE FAIL src/store.test.js • We can select and deselect a test leaf expect(received).toEqual(expected) Expected value to equal: {"files": [], "suites": [], "tests": ["my_test_dir::my_test_file::my_test_1"]} Received: {"my_suite": {"files": [], "nodeids": ["my_test_dir::my_test_file::my_test_1"]}} Difference: - Expected + Received Object { + "my_suite": Object { "files": Array [], - "suites": Array [], - "tests": Array [
12. CI EXPERT CI EXPERT But I’m also the CI expert, so: I have to understand failures on test suites in other languages. I have to nd the needed data on the outputs. I have to be able to run the failing tests locally or in another server. All with an unfamiliar language, tool, and interface.
13. WE NEED SOME COMMON TOOLING! WE NEED SOME COMMON TOOLING!
14. WHAT DO WE WANT IN AN INTERFACE? WHAT DO WE WANT IN AN INTERFACE?
15. MY CHRISTMAS LIST MY CHRISTMAS LIST Color Progress bar Relaunch failed tests only Launch speci c test(s) Web interface
16. BALTO BALTO
17. BALTO KESAKO? BALTO KESAKO? Balto is a Language Independent Test Orchestrator
18. COLOR COLOR
19. PROGRESS BAR PROGRESS BAR
20. RELAUNCH FAILED TESTS ONLY RELAUNCH FAILED TESTS ONLY
21. LAUNCH SPECIFIC TEST(S) LAUNCH SPECIFIC TEST(S)
22. BALTO DEMO BALTO DEMO [Demo live]
23. BALTO 101 BALTO 101
24. THE SECRET THE SECRET The secret of Balto is that it is reading the stdout of its subprocesses.
25. THE OTHER PART OF THE SECRET THE OTHER PART OF THE SECRET But the plugin, server, and UI need to understand each other.
26. TO UNDERSTAND EACH OTHER TO UNDERSTAND EACH OTHER They need a common language.
27. TEST OUTPUT FORMATS TEST OUTPUT FORMATS
28. TEST OUTPUT FORMATS TEST OUTPUT FORMATS Junit.xml Tap Mozlog Subunit
29. JUNIT.XML JUNIT.XML Based on XML One le generated at the end of the build Format tied to Junit => Non-streamable
30. TAP TAP Famous in the Perl community Simple, hard to extend Format tied to TAP Perl => Need an independent parser in Python and JS
31. MOZLOG MOZLOG Format used internally at Mozilla 1 message at the begging and another at the end of each test => Readers need to keep some state, 1 test == 1 message is easier for the consumer
32. SUBUNIT SUBUNIT Closest to the design of LITF Is a binary streaming format with streaming lters capability An effort has been made to add the missing pieces into subunit. => No input eld
33. MY CHRISTMAS LIST MY CHRISTMAS LIST Easy to write and read. Streamable so we do not need to wait for the end of the run. Format de ned outside of an implementation.
34. A NEW FORMAT A NEW FORMAT
35. WHAT DO WE NEED? WHAT DO WE NEED? Test name, Test status (failed, passed, skipped…), Error message in case of error.
36. EXAMPLE EXAMPLE { "test_name": "test_success", "outcome": "passed" } { "test_name": "test_failed", "outcome": "failed", "error": { "humanrepr": "def test_fails():\n> assert False\nE assert False\n\ntest_func.py:9: AssertionError" } }
37. MISSING PIECES MISSING PIECES { "test_number": 6, "_type": "session_start" } { "failed": 3, "total_duration": 0.03, "_type": "session_end", "skipped": 0, "error": 0, "passed": 3 }
38. MORE DATA MORE DATA Timings Log messages Stdout and stderr Texte and image diff File, line and more
39. LAUNCH A SPECIFIC TEST? LAUNCH A SPECIFIC TEST?
40. INPUT VS OUTPUT INPUT VS OUTPUT We also need a new input format: { "collect_only": true } { "files": ["file1.py", "file2.py"], "nodes": ["file1.py::test_func"] }
41. HOW TO GET A UNIQUE IDENTIFIER? HOW TO GET A UNIQUE IDENTIFIER? Let’s add a new output eld: { "test_name": "test_success", "outcome": "passed", "_id": "test_func.py::test_success" }
42. FULL EXAMPLE FULL EXAMPLE { "test_name": "test_failed", "_id": "test_func.py::test_success", "outcome": "failed", "error": { "humanrepr": "def test_fails():\n> assert False\nE assert False\n\ntest_func.py:9: AssertionError" } }
43. LITF LITF
44. THE FORMAT 101 THE FORMAT 101 The Language Independent Test Format is de ned in its repository. https://github.com/Lothiraldan/litf
45. MESSAGE DEFINITION MESSAGE DEFINITION Each message is a JSON-Schema le which de nes mandatory and required elds.
46. HELPER SCRIPT HELPER SCRIPT The LITF repository contains a script to validate a LITF stream or le.
47. MISSING DATA MISSING DATA Version number Binary data
48. STATUS STATUS Beta, works for me™
49. MORE LANGUAGES MORE LANGUAGES JS PHP Rust Java
50. OTHER STREAM SOURCES OTHER STREAM SOURCES SSH Docker container
51. ASYNC FOR THE WIN ASYNC FOR THE WIN As Balto is using the asynchronous capabilities of Python, we could run several test suites in parallel. Or even several test suites in parallel, written in several different languages on a distance server using Docker?
52. ARCHITECTURE ARCHITECTURE Balto architecture Balto Pytest-Litf LITF over Subprocess Mercurial-Litf LITF over SSH Jest-Litf LITF over Docker Pytest Mercurial Jest
53. CONCLUSION CONCLUSION
54. A NEW PROTOCOL A NEW PROTOCOL Similar to HTTP or LSP, it’s hopefully a foundation that could be used for building tomorrow tools.
55. EXAMPLES EXAMPLES You want a curses tool to run your tests? You can! You want a detailed report on the test timings? You can!
56. INSTALL BALTO INSTALL BALTO Install pipx on your system Install Balto using pipx: pipx install balto Install the LITF plugin for you test-runner in your environment: pip install pytest-litf or npm install jest-litf Enjoy
57. HOW YOU CAN HELP HOW YOU CAN HELP Feedback about LITF, Create new LITF producer plugin, Create new LITF consumer, Use Balto or speak about it.
58. CONTRIBUTE CONTRIBUTE Balto and LITF are open-source: https://github.com/lothiraldan/litf https://github.com/lothiraldan/balto
59. QUESTIONS? QUESTIONS?