Slide 1

Slide 1 text

Resilient UI Test Patterns Markus Ende & Gregor Woiwode Photo by James Peacock on Unsplash

Slide 2

Slide 2 text

Software Architect & Co-founder of qupaya technologies GmbH mailto: [email protected] twitter: @_der_markusende / @qupaya Markus Ende 󰗞

Slide 3

Slide 3 text

Software Architect & CTO of co-IT.eu GmbH mailto: [email protected] twitter: @GregOnNet oss: Gregor Woiwode 👋

Slide 4

Slide 4 text

Show me the code https://github.com/ng-practice/talk-dwx-resilient-test-patterns

Slide 5

Slide 5 text

Motivation

Slide 6

Slide 6 text

Conﬁdence Product Team Customer Product build & maintain uses

Slide 7

Slide 7 text

- cannot afford to fix all tests. - Loss of confidence in tests - Begin of deleting/skipping test - Increase of regression bugs - has less confidence using . CONFIDENCE If tests break frequently...

Slide 8

Slide 8 text

CONFIDENCE new supplier new employer Once conﬁdence is lost...

Slide 9

Slide 9 text

Conﬁdence The needs of our customers change quickly. We need conﬁdence to rethink & re-design systems fast without harming the quality of our product. Therefore, we need trust in our systems, but also trust in our automated tests.

Slide 10

Slide 10 text

Conﬁdence We need tests which can tell us the truth about the state of our software product. Therefore, we have to stabilize our test continuously.

Slide 11

Slide 11 text

Resilient Factors

Slide 12

Slide 12 text

Resilience in the Context of Testing Resilience is a continuous process, making tests more stable against environment changes. The main goal is to increase maintainability of tests to allow developers to focus on feature development.

Slide 13

Slide 13 text

Resilient factors Comprehensibility Predictability Independence Maintainability

Slide 14

Slide 14 text

Comprehensibility

Slide 15

Slide 15 text

DAMP over DRY

Slide 16

Slide 16 text

DAMP DAMP stands for "Descriptive and Meaningful Phrases" and promotes the readability of the code. - Enterprise Craftsmanship

Slide 17

Slide 17 text

DAMP over DRY DAMP Beneﬁts - Reduce time reading through test-helper abstractions. - Tests are more isolated from each other. - Test contains all information needed to understand what is going on.

Slide 18

Slide 18 text

DAMP over DRY DON’T describe('Product', () => { it('can select', () => {}); describe('When a product is selected', () => { it('is marked with a green tick', () => {}); it('gets a green background color', () => {}); }); DO Test Description

Slide 19

Slide 19 text

DAMP over DRY Summary - Use shorthands - Be too general - Use technical terms (like Button, Array or Error) DON’T DO - Describe the behaviour of your algorithm. - Formulate phrases a non-technician can understand. - Provide as much context as possible to your fellow developers.

Slide 20

Slide 20 text

DAMP over DRY Prefer duplication over the wrong abstraction, and optimize for change first. - Kent C. Dodds (https://kentcdodds.com/blog/avoid-nesting-when-youre-testing) Hint

Slide 21

Slide 21 text

Cognitive Load

Slide 22

Slide 22 text

Cognitive Load - How long does it take for one of your team members to understand your test?

Slide 23

Slide 23 text

Cognitive Load Test Setup beforeEach(() => { TestBed.configureTestingModule({ /* ... */ }); fixture = TestBed.createComponent(TodosComponent); component = fixture.componentInstance; fixture.detectChanges(); }); DON’T start with one global setup for all tests.

Slide 24

Slide 24 text

Cognitive Load Test Setup it('is as concrete as possible', () => { TestBed.configureTestingModule({ /*...*/ }); fixture = TestBed.createComponent(TodosComponent); component = fixture.componentInstance; fixture.detectChanges(); }); DO

Slide 25

Slide 25 text

Cognitive Load Test Preparation

Slide 26

Slide 26 text

Cognitive Load Test Dependencies

Slide 27

Slide 27 text

Cognitive Load Description vs. Implementation

Slide 28

Slide 28 text

Cognitive Load Global Test State

Slide 29

Slide 29 text

COGNITIVE LOAD Summary - Allow too large setups - Implement logic in your tests - Share state between tests - Tolerate outdated tests - Control the amount of test dependencies - Ensure that test description and implementation still match - Keep state isolated inside one test DON’T DO

Slide 30

Slide 30 text

How to apply DAMP in your tests

Slide 31

Slide 31 text

Testing Library Simple and complete testing utilities that encourage good testing practices. testing-library.com 🛠 Write Maintainable Tests ✅ Develop with Conﬁdences 🎉 Accessible by Default

Slide 32

Slide 32 text

Testing Library TESTING LIBRARY Setup await render(TodoCheckerComponent, { componentProperties: { todo: { isDone: false, text: 'Buy milk' } } }); Vanilla Angular TestBed.configureTestingModule({ declarations: [TodoCheckerComponent] }); fixture = TestBed.createComponent( TodoCheckerComponent); fixture.componentInstance.todo = { isDone: false, text: 'Buy milk' }; fixture.detectChanges();

Slide 33

Slide 33 text

TESTING LIBRARY Template Testing Library Vanilla Angular const checkbox: = () => fixture.debugElement.query( By.css('input[type=checkbox]') ).nativeElement; checkbox().click(); checkbox().dispatchEvent( new Event('click') ); fixture.detectChanges(); expect(checkbox().checked).toBe(true); userEvent.click( screen.getByRole('checkbox') ); expect( screen.getByRole('checkbox') ).toBeChecked();

Slide 34

Slide 34 text

TESTING LIBRARY Events Testing Library Vanilla Angular fixture.debugElement.query( By.css('input[type=checkbox]') ).nativeElement.click(); fixture.detectChanges(); userEvent.click( screen.getByRole( 'Button', { name: 'Add' } ) );

Slide 35

Slide 35 text

TESTING LIBRARY Events | Built-In Testing Library

Slide 36

Slide 36 text

Template Selectors Testing Library Template expect( screen.getByTestId('todo-checkbox') ).not.toBeChecked();

Slide 37

Slide 37 text

Template Selectors Testing Library Template Role Selectors expect( screen.getByRole( 'checkbox', { name: 'todo' } ) ).not.toBeChecked();

Slide 38

Slide 38 text

Component Harness for Angular Material

Slide 39

Slide 39 text

💁 Each Material Component ships with its Test-API. Check out the Documentation to learn about the details. - material.angular.io

Slide 40

Slide 40 text

COMPONENT HARNESS ISBN has to be at least 3 characters long. ISBN is required How would you test mat-form-field?

Slide 41

Slide 41 text

COMPONENT HARNESS import { TestbedHarnessEnvironment, HarnessLoader } from '@angular/cdk/testing/testbed'; let fixture: ComponentFixture; let loader: HarnessLoader; fixture = TestBed.createComponent(SomeComponent); loader = TestbedHarnessEnvironment.loader(fixture); Setup Harness Environment

Slide 42

Slide 42 text

COMPONENT HARNESS Testing Library Integration import { TestbedHarnessEnvironment, HarnessLoader } from '@angular/cdk/testing/testbed'; // ... let fixture: ComponentFixture; let loader: HarnessLoader; // ... const { fixture} = await render(SomeComponent); loader = TestbedHarnessEnvironment.loader(fixture);

Slide 43

Slide 43 text

COMPONENT HARNESS import { MatFormFieldHarness } from '@angular/material/form-field/testing'; const formField = await loader.getHarness(MatFormFieldHarness); Get Harness for speciﬁc material component

Slide 44

Slide 44 text

COMPONENT HARNESS import { MatFormFieldHarness } from '@angular/material/form-field/testing'; const formField = await loader.getHarness( MatFormFieldHarness.with({ selector: '[data-test=isbn-field]' }) ); Specify resilient selectors

Slide 45

Slide 45 text

COMPONENT HARNESS import { MatInputHarness } from '@angular/material/input/testing'; const formField = await loader.getHarness(/* … */); const input = (await formField.getControl()) as MatInputHarness; Access child material component

Slide 46

Slide 46 text

COMPONENT HARNESS await input.setValue('12'); await input.blur(); Interact with material component

Slide 47

Slide 47 text

COMPONENT HARNESS it('test with Angular Material component', async () => { const formField = await loader.getHarness(MatFormFieldHarness); const errors = await formField.getTextErrors(); }) Component-Harness-API is asynchronous

Slide 48

Slide 48 text

COMPONENT HARNESS const errors = await formField.getTextErrors(); expect(errors).toContain( 'ISBN has to be at least 3 characters long.'); Get state information from component.

Slide 49

Slide 49 text

Predictability

Slide 50

Slide 50 text

False Positives

Slide 51

Slide 51 text

FALSE POSITIVES Negative assertions - Negative assertions check things which does not exist or are not allowed to happen. - Negative assertions can lead into False Positives.

Slide 52

Slide 52 text

consumer.service describe('Negative assertion test', () => { it('passes because assertion is not specific enough', () => { const service = mock(Service); const consumer = new Consumer(instance(service)); consumer.do(); verify(service.doWork('1', '2')).never(); }); }); FALSE POSITIVES Negative assertions class Consumer { constructor(private service: Service) {} do() { this.service.doWork('1', '2', true); } } Test is green, but the method was called.

Slide 53

Slide 53 text

FALSE POSITIVES Negative assertions - Negative assertion can also happen in UI-Tests. - We use for our demos. - BTW: Protractor ofﬁcially reaches end-of-life in 2022

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

FALSE POSITIVES Asynchronous tests it('broken: is green because done was forgotten', () => { defer(() => throwError('My error')).subscribe({ next: () => expect(true).toBe(false), // should make test red error: () => expect(true).toBe(false), // should make test red }); }); Test should be red, but...

Slide 56

Slide 56 text

FALSE POSITIVES Asynchronous tests | Done Callback it('done: evaluates error correctly', done => { defer(() => throwError('My error')).subscribe({ error: (err) => { expect(err).toBe('My error'); done(); }, }); }); Make test aware of asynchronous process.

Slide 57

Slide 57 text

FALSE POSITIVES Asynchronous tests | Promise it('promisify: evaluates error correctly', async () => { const promise = defer(() => throwError('My error')).toPromise(); await expect(promise).rejects.toEqual('My error'); }); Sometimes it is even cleaner to convert the Observable to a Promise.

Slide 58

Slide 58 text

Flakiness

Slide 59

Slide 59 text

FLAKINESS Network dependencies - Occur mostly in UI Tests - The test runner checks if the DOM matches the expectation at a certain time - When the DOM is not ready, the test fails even if the expectation matches a few milliseconds later.

Slide 60

Slide 60 text

FLAKINESS Network dependencies Test Command cy.get(...) Network Operation Test fails DOM update is too late, DOM observes

Slide 61

Slide 61 text

PREDICTABILITY Summary - Use timeouts in your tests to make them pass. - Analyse the root cause why a test is ﬂaky - Be always aware of asynchronous operations - Be as concrete as possible when using negative assertions DON’T DO

Slide 62

Slide 62 text

Independence

Slide 63

Slide 63 text

Grade of Independence How many dependencies does the subject under test have? How many dependencies does the test have to other tests? Often accidentally

Slide 64

Slide 64 text

Scope

Slide 65

Slide 65 text

Scope Dependencies of the subject under test: ● to other modules ● to external systems ● to the execution environment ● ...

Slide 66

Slide 66 text

SCOPE Module Level ● Logical: Service / Pipe / Class / Function / ... ● Visual: Component / Directive (includes DOM) Application Level ● Application Pages / Screens ● Application Use cases Subject under Test

Slide 67

Slide 67 text

Scope Broader scope logical visual Module Level App Level Screens Use Cases More dependencies

Slide 68

Slide 68 text

Scope: Module Level Broader scope App Level Screens Use Cases Jest / Karma Isolated Shallow Full Module Level More dependencies

Slide 69

Slide 69 text

Scope @Component({ selector: 'todos', template: `

Todos

`, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Component Under Test Adds one initial todo Child component for adding a todo Child component for todo checkbox

Slide 70

Slide 70 text

Scope @Component({ selector: 'todos', template: `

Todos

Slide 71

Slide 71 text

Scope it('should initialize todos', async () => { const component = new TodosComponent(); component.ngOnInit(); expect(component.todos).toEqual([{ text: 'initial', isDone: false }]); }); Isolated Unit Test Without TestBed You are responsible for Angular mechanics “Classic” unit test

Slide 72

Slide 72 text

Scope @Component({ selector: 'todos', template: `

Todos

Slide 73

Slide 73 text

Scope it('should initialize todos', async () => { await render(TodosComponent, { declarations: [TodoCheckerComponent], schemas: [NO_ERRORS_SCHEMA], }); expect( screen.getByRole('checkbox', { name: 'initial' }) ).toBeInTheDocument(); }); Shallow Unit Test Ignore all other components Include direct child components needed for this test, without its child components!

Slide 74

Slide 74 text

it('should add todo', async () => { const { fixture, detectChanges } = await render(TodosComponent, { declarations: [TodoCheckerComponent], schemas: [NO_ERRORS_SCHEMA], }); fixture.componentInstance.addTodo({ text: 'added', isDone: false }); detectChanges(); expect( screen.getByRole('checkbox', { name: 'added' }) ).toBeInTheDocument(); }); Access to componentInstance needed Scope Shallow Unit Test

Slide 75

Slide 75 text

Scope @Component({ selector: 'todos', template: `

Todos

Slide 76

Slide 76 text

Scope it('should add todo', async () => { await render(TodosComponent, { declarations: [TodoCheckerComponent, TodoQuickAddComponent], }); userEvent.type(screen.getByRole('textbox'), 'added'); userEvent.click(screen.getByRole('button', { name: 'Add' })); expect( screen.getByRole('checkbox', { name: 'initial' }) ).toBeInTheDocument(); expect( screen.getByRole('checkbox', { name: 'added' }) ).toBeInTheDocument(); }); Full Unit/Integration Test Include all child components with their dependencies

Slide 77

Slide 77 text

Which Type to choose? Isolated Shallow Full Pros: Fastest, most direct setup Fast, low maintenance Most confidence Be aware of: -No DOM -Angular mechanics manual (e.g. ngOnInit) - sometimes hard to test component interaction -Slower, especially with large modules -Breaks more often (false negatives) -Dependencies of dependencies When to use: -Testing algorithms -Testing logical units like services & pipes -Testing single part of component in isolation -Testing component integration -In combination with shallow tests

Slide 78

Slide 78 text

Test Conﬁdence

Slide 79

Slide 79 text

Scope Broader scope More dependencies Jest / Karma Isolated Shallow (NO_ERRORS_SCHEMA) Full Module Level Longer duration More confidence

Slide 80

Slide 80 text

Scope Summary - Rely on a single type of test - Mess with too many dependencies - Lots of shallow tests - Some full/integration tests for conﬁdence - Isolated tests for logical modules DON’T DO

Slide 81

Slide 81 text

What about logical dependencies? Shallow TestBed tests handle visual dependencies (components, directives). We still have dependencies to services, InjectionTokens, third-party libs, ...

Slide 82

Slide 82 text

Mocking and Stubbing

Slide 83

Slide 83 text

Stubbing via Angular’s Dependency Injection Provide fake implementations { provide: TodosService, useValue: { query: () => of([ { text: 'test TODO 1' }, { text: 'test TODO 2' }, ]), }, } We provide a fake TodosService to avoid side effects

Slide 84

Slide 84 text

Avoid accessing JS-Globals directly constructor() { const host = document.location.hostname; const token = localStorage.getItem('token'); } const LOCAL_STORAGE = new InjectionToken('LOCAL_STORAGE', { providedIn: 'root', factory: () => localStorage, }); constructor( @Inject(DOCUMENT) document: Document, @Inject(LOCAL_STORAGE) localStorage: Storage ) { const host = document.location.hostname; const token = localStorage.getItem('token'); } Not testable via DI and easy to forget → test could inﬂuence other tests This also works for 3rd-party libs that add to window Don’t Do

Slide 85

Slide 85 text

Mocking Record calls for behavior veriﬁcation style testing const queryMock = jest.fn().mockReturnValue([]); // in TestBed { provide: TodosService, useValue: { query: queryMock }, } expect(queryMock.mock.calls.length).toBe(1); expect(queryMock.mock.calls[0]).toEqual(["all"]); createSpy / createSpyObj when using Jasmine

Slide 86

Slide 86 text

What if dependency is changed? { provide: TodosService, useValue: { query: () => of([ { text: 'TODO 1' }, { text: 'TODO 2' }, ]), }, } export class TodosService { query(): Observable { // implementation... } } ⚡Renaming: query → getTodos ⚡Renaming: Todo.text → Todo.title

Slide 87

Slide 87 text

Runtime errors or no error at all! { provide: TodosService, useValue: { query: () => of([ { text: 'TODO 1' }, { text: 'TODO 2' }, ]), }, } export class TodosService { query(): Observable { // implementation... } } ⚡Renaming: query → getTodos No type safety ⚡Renaming: Todo.text → Todo.title

Slide 88

Slide 88 text

With types: Compile-time error { provide: TodosService, useValue: { oldName: () => of([ { wrong: 'test TODO 1' } as Todo, { wrong: 'test TODO 2' } as Todo ]) } as TodosService } 😍 Conversion of type '{ oldName: () => Observable; }' to type 'TodosService' may be a mistake... Conversion of type '{ wrong: string; }' to type 'Todo' may be a mistake... With type safety Use types

Slide 89

Slide 89 text

Type your stubs! The less the stubbed object diverges from the original, the more conﬁdence it gives. Changes in the original must be reﬂected! Subject under Test Original A Stubbed A Original A Stubbed A ⚡ False Positive!

Slide 90

Slide 90 text

(Some of the) Mocking Libraries that Help Framework-agnostic ts-jest jest-mock-extended Sinon.JS moq.ts ts-mockito Angular-speciﬁc Spectator (mockProvider) ng-mocks ng-mockito

Slide 91

Slide 91 text

Personal Hero: ts-mockito Setup: let mockTodosService = mock(TodosService); // set up stub when(mockTodosService.query(anyString())).thenReturn(of([])); // ...use in TestModule: providers: [{ provide: TodosService, useFactory: () => instance(mockTodosService), }], has type TodosService Type-safe + IDE autocompletion

Slide 92

Slide 92 text

Personal Hero: ts-mockito Behavior veriﬁcation: verify(mockTodosService.query('all')).called(); const [param] = capture(mockTodosService.query).first(); expect(param).toEqual('all'); Type-safe + IDE autocompletion

Slide 93

Slide 93 text

Shameless plug: ng-mockito Thin layer on top of ts-mockito providers: [ mockNg(TodosService, (mock) => when(mock.query(anyString())).thenReturn(NEVER) ), mockNg([APP_TITLE, TodosComponent], { use: 'Test Heading' }) ] Infers APP_TITLE InjectionToken type from TodosComponent’s constructor Inline stubbing mockProvider, mockPipe, mockComponent, ...

Slide 94

Slide 94 text

Slide 95

Slide 95 text

it('should add todo', async () => { const createEvent = new EventEmitter(); const { detectChanges } = await render(TodosComponent, { declarations: [ TodoCheckerComponent, mockNg(TodoQuickAddComponent, (mock) => when(mock.create).thenReturn(createEvent) ), ], schemas: [NO_ERRORS_SCHEMA], }); createEvent.next({ text: 'added', isDone: false }); detectChanges(); expect(screen.getByRole('checkbox', { name: 'added' })) .toBeInTheDocument(); }); Shallow Unit Test Revisited: Component Stub Mock @Output: (create)="addTodo($event)"

Slide 96

Slide 96 text

Mock all the things? Broader scope Jest / Karma Mocked Dependencies Real Dependencies Module Level More confidence Less Isolation

Slide 97

Slide 97 text

Mocking and Stubbing Summary - Access globals directly, it makes them hard to stub - Forget to mock globals - Use untyped stubs - Mock globals like localStorage to avoid dependencies between tests - Use Libraries for mocking, e.g. ts-mockito and ng-mockito - Make sure that changes in the dependency are reﬂected in your stubs DON’T DO

Slide 98

Slide 98 text

Cypress Integration Testing

Slide 99

Slide 99 text

Scope Broader scope Jest Isolated Shallow Full Module Level App Level Cypress Integration (Fixtures) E2E

Slide 100

Slide 100 text

Fixtures for Test Isolation Instead of Server Resets ● Stub POST/PUT/DELETE to avoid state changes on server ● Stub GET to create starting state Be aware: State transitions (e.g. different response after PUSH) must be recreated for test → real e2e tests give more conﬁdence

Slide 101

Slide 101 text

Integration Testing with Cypress Stubbing, Cypress style: cy.intercept('/api?query=all', { body: [ { text: 'test TODO 1' }, { text: 'test TODO 2' }, ], }); Automatically fakes api response

Slide 102

Slide 102 text

Typed Fixtures cy.intercept('/api?query=all', { body: [ { text: 'test TODO 1' }, { text: 'test TODO 2' }, ] as Todo[], }); Extract into shared lib and import in spec

Slide 103

Slide 103 text

Fixtures are Untyped Fixtures need same handling like other stubs. Changes in the original must be reﬂected! Cypress Test Original Response Stubbed Response Changed Response Stubbed Response False Positive!

Slide 104

Slide 104 text

Cypress Component Testing

Slide 105

Slide 105 text

Cypress Component Tests Broader scope Jest Isolated Shallow Full Module Level App Level Integration (Fixtures) E2E Cypress Component Tests

Slide 106

Slide 106 text

Storybook Storybook is a UI component explorer, mainly for ● Presentation components ● UI libraries

Slide 107

Slide 107 text

Cypress Storybook Test describe('TodoCheckerComponent', () => { it('should show todo item', () => { cy.visit('/iframe.html?id=todocheckercomponent--primary'); cy.findByRole('checkbox', { name: 'Test TODO' }).should(...); }); }); Accesses iframe

Slide 108

Slide 108 text

Cypress Storybook Test Setup Base component setup in Storybook Controls can be used to provide story parameters for Cypress. cy.visit('/iframe.html?id=todocheckercomponent--primary&args=text:Some+other+todo');

Slide 109

Slide 109 text

Cypress Storybook: Use or Not? Depends on your project. Do you also proﬁt from using Storybook as documentation? Do you develop a UI library (“widgets”)? Do you use Jest? Maybe completely replace full TestBed tests?

Slide 110

Slide 110 text

Alternative: “real” Cypress Component Tests Disclaimer: Brand new (Alpha) Angular support currently not out-of-the-box (@jscutlery/cypress-angular)

Slide 111

Slide 111 text

Cypress Component Test No need to visit, directly mount component: describe('TodoCheckerComponent', () => { it('should show text', () => { const todo: Todo = { text: 'Wow', isDone: true }; mount(TodoCheckerComponent, { inputs: { todo, }, }); cy.findByRole('checkbox', { name: 'Wow' }).should(...); }); });

Slide 112

Slide 112 text

Why Cypress for Component Testing? Setup complexity equal to full Karma/Jest Integration Tests But: ● Better Tooling (watch mode, videos, replayability, …) ● Browser Environment (especially useful when using Jest for other tests) with easy debuggability ● Easier Network handling ● Automatic waiting for DOM element changes

Slide 113

Slide 113 text

Testing Pyramid (upside down) Broader scope Jest Isolated Shallow (NO_ERRORS_SCHEMA) Full Module Level App Level Integration / E2E Cypress Component Tests Is this still valid? do less (slow + expensive) do many (fast + cheap)

Slide 114

Slide 114 text

Costs of UI tests decreased Broader scope Jest Isolated Shallow (NO_ERRORS_SCHEMA) Full Module Level App Level Integration / E2E Cypress Component Tests This is faster and cheaper now!

Slide 115

Slide 115 text

CYPRESS Summary - Use untyped ﬁxtures - Use Storybook Tests if you don’t intend to use Storybook - Lots of Integration Testing using Fixtures - Mix in real E2E tests for conﬁdence - Try Component Testing as Integration Test Alternative DON’T DO

Slide 116

Slide 116 text

Thank you for having us Markus Ende @_der_markusende & Gregor Woiwode @GregOnNet Photo by James Peacock on Unsplash