BALTO, ONE TEST OUTPUT FORMAT TO UNITE THEM ALL

Slide 1

Slide 1 text

BALTO, ONE TEST OUTPUT FORMAT TO UNITE BALTO, ONE TEST OUTPUT FORMAT TO UNITE THEM ALL THEM ALL Boris Feld - Fosdem 2020

Slide 2

Slide 2 text

PRESENTATION PRESENTATION

Slide 3

Slide 3 text

BORIS FELD BORIS FELD Python and testing fan. Former Mercurial reviewer. https://www.comet.ml/ @lothiraldan

Slide 4

Slide 4 text

POLL: HOW MANY WRITE UNIT-TESTS? POLL: HOW MANY WRITE UNIT-TESTS?

Slide 5

Slide 5 text

POLL: HOW MANY ONLY RUN UNIT-TESTS? POLL: HOW MANY ONLY RUN UNIT-TESTS?

Slide 6

Slide 6 text

WE ARE NOT ALONE WE ARE NOT ALONE 2017 2018 2019 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% Launching tests Writing tests => More than 70% write and run unit tests. Source: Jetbrains Developer Ecosystem Survey

Slide 7

Slide 7 text

I DO BOTH I DO BOTH

Slide 8

Slide 8 text

I WRITE TESTS I WRITE TESTS Mainly in Python but in JS too.

Slide 9

Slide 9 text

I KNOW HOW TO WRITE TESTS + RUN I KNOW HOW TO WRITE TESTS + RUN THEM THEM I know how to run all tests, A single test le, A single test, Check why my new test is not running, Know where to nd the needed data on the output, Know how to debug a failing test ef ciently.

Slide 10

Slide 10 text

PYTEST INTERFACE PYTEST INTERFACE ============================= test session starts ============================== platform linux2 -- Python 2.7.15, pytest-3.1.2, py-1.5.3, pluggy- 0.4.0 rootdir: /home/lothiraldan/Labo/presentations/balto_revolution_fr/src, inifile: collected 2 items test.py F. =================================== FAILURES =================================== ________________________ TestStringMethods.test_isupper ________________________ self = def test_isupper(self): self.assertTrue('FOO'.isupper()) > self.assertTrue('Foo'.isupper()) E AssertionError: False is not true

Slide 11

Slide 11 text

JEST INTERFACE JEST INTERFACE FAIL src/store.test.js ● We can select and deselect a test leaf expect(received).toEqual(expected) Expected value to equal: {"files": [], "suites": [], "tests": ["my_test_dir::my_test_file::my_test_1"]} Received: {"my_suite": {"files": [], "nodeids": ["my_test_dir::my_test_file::my_test_1"]}} Difference: - Expected + Received Object { + "my_suite": Object { "files": Array [], - "suites": Array [], - "tests": Array [

Slide 12

Slide 12 text

CI EXPERT CI EXPERT But I’m also the CI expert, so: I have to understand failures on test suites in other languages. I have to nd the needed data on the outputs. I have to be able to run the failing tests locally or in another server. All with an unfamiliar language, tool, and interface.

Slide 13

Slide 13 text

WE NEED SOME COMMON TOOLING! WE NEED SOME COMMON TOOLING!

Slide 14

Slide 14 text

WHAT DO WE WANT IN AN INTERFACE? WHAT DO WE WANT IN AN INTERFACE?

Slide 15

Slide 15 text

MY CHRISTMAS LIST MY CHRISTMAS LIST Color Progress bar Relaunch failed tests only Launch speci c test(s) Web interface

Slide 16

Slide 16 text

BALTO BALTO

Slide 17

Slide 17 text

BALTO KESAKO? BALTO KESAKO? Balto is a Language Independent Test Orchestrator

Slide 18

Slide 18 text

COLOR COLOR

Slide 19

Slide 19 text

PROGRESS BAR PROGRESS BAR

Slide 20

Slide 20 text

RELAUNCH FAILED TESTS ONLY RELAUNCH FAILED TESTS ONLY

Slide 21

Slide 21 text

LAUNCH SPECIFIC TEST(S) LAUNCH SPECIFIC TEST(S)

Slide 22

Slide 22 text

BALTO DEMO BALTO DEMO [Demo live]

Slide 23

Slide 23 text

BALTO 101 BALTO 101

Slide 24

Slide 24 text

THE SECRET THE SECRET The secret of Balto is that it is reading the stdout of its subprocesses.

Slide 25

Slide 25 text

THE OTHER PART OF THE SECRET THE OTHER PART OF THE SECRET But the plugin, server, and UI need to understand each other.

Slide 26

Slide 26 text

TO UNDERSTAND EACH OTHER TO UNDERSTAND EACH OTHER They need a common language.

Slide 27

Slide 27 text

TEST OUTPUT FORMATS TEST OUTPUT FORMATS

Slide 28

Slide 28 text

TEST OUTPUT FORMATS TEST OUTPUT FORMATS Junit.xml Tap Mozlog Subunit

Slide 29

Slide 29 text

JUNIT.XML JUNIT.XML Based on XML One le generated at the end of the build Format tied to Junit => Non-streamable

Slide 30

Slide 30 text

TAP TAP Famous in the Perl community Simple, hard to extend Format tied to TAP Perl => Need an independent parser in Python and JS

Slide 31

Slide 31 text

MOZLOG MOZLOG Format used internally at Mozilla 1 message at the begging and another at the end of each test => Readers need to keep some state, 1 test == 1 message is easier for the consumer

Slide 32

Slide 32 text

SUBUNIT SUBUNIT Closest to the design of LITF Is a binary streaming format with streaming lters capability An effort has been made to add the missing pieces into subunit. => No input eld

Slide 33

Slide 33 text

MY CHRISTMAS LIST MY CHRISTMAS LIST Easy to write and read. Streamable so we do not need to wait for the end of the run. Format de ned outside of an implementation.

Slide 34

Slide 34 text

A NEW FORMAT A NEW FORMAT

Slide 35

Slide 35 text

WHAT DO WE NEED? WHAT DO WE NEED? Test name, Test status (failed, passed, skipped…), Error message in case of error.

Slide 36

Slide 36 text

EXAMPLE EXAMPLE { "test_name": "test_success", "outcome": "passed" } { "test_name": "test_failed", "outcome": "failed", "error": { "humanrepr": "def test_fails():\n> assert False\nE assert False\n\ntest_func.py:9: AssertionError" } }

Slide 37

Slide 37 text

MISSING PIECES MISSING PIECES { "test_number": 6, "_type": "session_start" } { "failed": 3, "total_duration": 0.03, "_type": "session_end", "skipped": 0, "error": 0, "passed": 3 }

Slide 38

Slide 38 text

MORE DATA MORE DATA Timings Log messages Stdout and stderr Texte and image diff File, line and more

Slide 39

Slide 39 text

LAUNCH A SPECIFIC TEST? LAUNCH A SPECIFIC TEST?

Slide 40

Slide 40 text

INPUT VS OUTPUT INPUT VS OUTPUT We also need a new input format: { "collect_only": true } { "files": ["file1.py", "file2.py"], "nodes": ["file1.py::test_func"] }

Slide 41

Slide 41 text

HOW TO GET A UNIQUE IDENTIFIER? HOW TO GET A UNIQUE IDENTIFIER? Let’s add a new output eld: { "test_name": "test_success", "outcome": "passed", "_id": "test_func.py::test_success" }

Slide 42

Slide 42 text

FULL EXAMPLE FULL EXAMPLE { "test_name": "test_failed", "_id": "test_func.py::test_success", "outcome": "failed", "error": { "humanrepr": "def test_fails():\n> assert False\nE assert False\n\ntest_func.py:9: AssertionError" } }

Slide 43

Slide 43 text

LITF LITF

Slide 44

Slide 44 text

THE FORMAT 101 THE FORMAT 101 The Language Independent Test Format is de ned in its repository. https://github.com/Lothiraldan/litf

Slide 45

Slide 45 text

MESSAGE DEFINITION MESSAGE DEFINITION Each message is a JSON-Schema le which de nes mandatory and required elds.

Slide 46

Slide 46 text

HELPER SCRIPT HELPER SCRIPT The LITF repository contains a script to validate a LITF stream or le.

Slide 47

Slide 47 text

MISSING DATA MISSING DATA Version number Binary data

Slide 48

Slide 48 text

STATUS STATUS Beta, works for me™

Slide 49

Slide 49 text

MORE LANGUAGES MORE LANGUAGES JS PHP Rust Java

Slide 50

Slide 50 text

OTHER STREAM SOURCES OTHER STREAM SOURCES SSH Docker container

Slide 51

Slide 51 text

ASYNC FOR THE WIN ASYNC FOR THE WIN As Balto is using the asynchronous capabilities of Python, we could run several test suites in parallel. Or even several test suites in parallel, written in several different languages on a distance server using Docker?

Slide 52

Slide 52 text

ARCHITECTURE ARCHITECTURE Balto architecture Balto Pytest-Litf LITF over Subprocess Mercurial-Litf LITF over SSH Jest-Litf LITF over Docker Pytest Mercurial Jest

Slide 53

Slide 53 text

CONCLUSION CONCLUSION

Slide 54

Slide 54 text

A NEW PROTOCOL A NEW PROTOCOL Similar to HTTP or LSP, it’s hopefully a foundation that could be used for building tomorrow tools.

Slide 55

Slide 55 text

EXAMPLES EXAMPLES You want a curses tool to run your tests? You can! You want a detailed report on the test timings? You can!

Slide 56

Slide 56 text

INSTALL BALTO INSTALL BALTO Install pipx on your system Install Balto using pipx: pipx install balto Install the LITF plugin for you test-runner in your environment: pip install pytest-litf or npm install jest-litf Enjoy

Slide 57

Slide 57 text

HOW YOU CAN HELP HOW YOU CAN HELP Feedback about LITF, Create new LITF producer plugin, Create new LITF consumer, Use Balto or speak about it.

Slide 58

Slide 58 text

CONTRIBUTE CONTRIBUTE Balto and LITF are open-source: https://github.com/lothiraldan/litf https://github.com/lothiraldan/balto

Slide 59

Slide 59 text

QUESTIONS? QUESTIONS?