Resilient UI Test Patterns

Talk given at DWX 2021 by Markus Ende and our friend Gregor Woiwode: https://www.developer-week.de/en/program/#/talk/resilient-ui-test-patterns-2

Repository with example code: https://github.com/ng-practice/talk-dwx-resilient-test-patterns

---

This session is all about testing.

Often this quality-assuring discipline is an unloved part of software development.

In Angular, this is sometimes due to the fact that mechanisms such as the manual triggering of change detection or complex configurations with TestBeds are difficult to handle.

Things are quickly forgotten, which means that the test either doesn't work or becomes a false positive.

Resilient test procedures provide you with readable and maintainable tests.

This includes best refactoring practices, as well as various tools and frameworks.

You will learn the following techniques in the talk to secure your tests against outside changes:

- The Angular Testing Library makes it easier for you to write understandable tests.

- Typed mocks sound an alarm as soon as contracts in the frontend are significantly changed.

- Storybook allows isolated UI component tests with Cypress.

After this talk, you will be able to design and implement resilient tests.

Transcript

1. Resilient UI Test Patterns Markus Ende & Gregor Woiwode Photo by James Peacock on Unsplash
2. Software Architect & Co-founder of qupaya technologies GmbH mailto: markus@qupaya.com twitter: @_der_markusende / @qupaya Markus Ende 󰗞
3. Software Architect & CTO of co-IT.eu GmbH mailto: gregor.woiwode@co-IT.eu twitter: @GregOnNet oss: Gregor Woiwode 👋
4. Show me the code https://github.com/ng-practice/talk-dwx-resilient-test-patterns
5. Motivation
6. Confidence Product Team Customer Product build & maintain uses
7. - cannot afford to fix all tests. - Loss of confidence in tests - Begin of deleting/skipping test - Increase of regression bugs - has less confidence using . CONFIDENCE If tests break frequently...
8. CONFIDENCE new supplier new employer Once confidence is lost...
9. Confidence The needs of our customers change quickly. We need confidence to rethink & re-design systems fast without harming the quality of our product. Therefore, we need trust in our systems, but also trust in our automated tests.
10. Confidence We need tests which can tell us the truth about the state of our software product. Therefore, we have to stabilize our test continuously.
11. Resilient Factors
12. Resilience in the Context of Testing Resilience is a continuous process, making tests more stable against environment changes. The main goal is to increase maintainability of tests to allow developers to focus on feature development.
13. Resilient factors Comprehensibility Predictability Independence Maintainability
14. Comprehensibility
15. DAMP over DRY
16. DAMP DAMP stands for "Descriptive and Meaningful Phrases" and promotes the readability of the code. - Enterprise Craftsmanship
17. DAMP over DRY DAMP Benefits - Reduce time reading through test-helper abstractions. - Tests are more isolated from each other. - Test contains all information needed to understand what is going on.
18. DAMP over DRY DON’T describe('Product', () => { it('can select', () => {}); describe('When a product is selected', () => { it('is marked with a green tick', () => {}); it('gets a green background color', () => {}); }); DO Test Description
19. DAMP over DRY Summary - Use shorthands - Be too general - Use technical terms (like Button, Array or Error) DON’T DO - Describe the behaviour of your algorithm. - Formulate phrases a non-technician can understand. - Provide as much context as possible to your fellow developers.
20. DAMP over DRY Prefer duplication over the wrong abstraction, and optimize for change first. - Kent C. Dodds (https://kentcdodds.com/blog/avoid-nesting-when-youre-testing) Hint
21. Cognitive Load
22. Cognitive Load - How long does it take for one of your team members to understand your test?
23. Cognitive Load Test Setup beforeEach(() => { TestBed.configureTestingModule({ /* ... */ }); fixture = TestBed.createComponent(TodosComponent); component = fixture.componentInstance; fixture.detectChanges(); }); DON’T start with one global setup for all tests.
24. Cognitive Load Test Setup it('is as concrete as possible', () => { TestBed.configureTestingModule({ /*...*/ }); fixture = TestBed.createComponent(TodosComponent); component = fixture.componentInstance; fixture.detectChanges(); }); DO
25. Cognitive Load Test Preparation
26. Cognitive Load Test Dependencies
27. Cognitive Load Description vs. Implementation
28. Cognitive Load Global Test State
29. COGNITIVE LOAD Summary - Allow too large setups - Implement logic in your tests - Share state between tests - Tolerate outdated tests - Control the amount of test dependencies - Ensure that test description and implementation still match - Keep state isolated inside one test DON’T DO
30. How to apply DAMP in your tests
31. Testing Library Simple and complete testing utilities that encourage good testing practices. testing-library.com 🛠 Write Maintainable Tests ✅ Develop with Confidences 🎉 Accessible by Default
32. Testing Library TESTING LIBRARY Setup await render(TodoCheckerComponent, { componentProperties: { todo: { isDone: false, text: 'Buy milk' } } }); Vanilla Angular TestBed.configureTestingModule({ declarations: [TodoCheckerComponent] }); fixture = TestBed.createComponent( TodoCheckerComponent); fixture.componentInstance.todo = { isDone: false, text: 'Buy milk' }; fixture.detectChanges();
33. TESTING LIBRARY Template Testing Library Vanilla Angular const checkbox: = () => fixture.debugElement.query( By.css('input[type=checkbox]') ).nativeElement; checkbox().click(); checkbox().dispatchEvent( new Event('click') ); fixture.detectChanges(); expect(checkbox().checked).toBe(true); userEvent.click( screen.getByRole('checkbox') ); expect( screen.getByRole('checkbox') ).toBeChecked();
34. TESTING LIBRARY Events Testing Library Vanilla Angular fixture.debugElement.query( By.css('input[type=checkbox]') ).nativeElement.click(); fixture.detectChanges(); userEvent.click( screen.getByRole( 'Button', { name: 'Add' } ) );
35. TESTING LIBRARY Events | Built-In Testing Library
36. Template Selectors Testing Library Template expect( screen.getByTestId('todo-checkbox') ).not.toBeChecked(); <input type="checkbox" data-testId="todo-checkbox" [checked]="todo.isDone" (change)="emitToggle()" />
37. Template Selectors Testing Library Template Role Selectors <input type="checkbox" name="todo" [checked]="todo.isDone" (change)="emitToggle()" /> expect( screen.getByRole( 'checkbox', { name: 'todo' } ) ).not.toBeChecked();
38. Component Harness for Angular Material
39. 💁 Each Material Component ships with its Test-API. Check out the Documentation to learn about the details. - material.angular.io
40. COMPONENT HARNESS <mat-form-field data-test="isbn-field"> <input matInput formControlName="isbn" placeholder="ISBN" required /> <mat-error *ngIf="form.get('isbn')?.hasError('minlength')"> ISBN has to be at least 3 characters long. </mat-error> <mat-error *ngIf="form.get('isbn')?.hasError('required')"> ISBN is required </mat-error> </mat-form-field> How would you test mat-form-field?
41. COMPONENT HARNESS import { TestbedHarnessEnvironment, HarnessLoader } from '@angular/cdk/testing/testbed'; let fixture: ComponentFixture<SomeComponent>; let loader: HarnessLoader; fixture = TestBed.createComponent(SomeComponent); loader = TestbedHarnessEnvironment.loader(fixture); Setup Harness Environment
42. COMPONENT HARNESS Testing Library Integration import { TestbedHarnessEnvironment, HarnessLoader } from '@angular/cdk/testing/testbed'; // ... let fixture: ComponentFixture<SomeComponent>; let loader: HarnessLoader; // ... const { fixture} = await render(SomeComponent); loader = TestbedHarnessEnvironment.loader(fixture);
43. COMPONENT HARNESS import { MatFormFieldHarness } from '@angular/material/form-field/testing'; const formField = await loader.getHarness(MatFormFieldHarness); Get Harness for specific material component
44. COMPONENT HARNESS import { MatFormFieldHarness } from '@angular/material/form-field/testing'; const formField = await loader.getHarness( MatFormFieldHarness.with({ selector: '[data-test=isbn-field]' }) ); Specify resilient selectors
45. COMPONENT HARNESS import { MatInputHarness } from '@angular/material/input/testing'; const formField = await loader.getHarness(/* … */); const input = (await formField.getControl()) as MatInputHarness; Access child material component
46. COMPONENT HARNESS await input.setValue('12'); await input.blur(); Interact with material component
47. COMPONENT HARNESS it('test with Angular Material component', async () => { const formField = await loader.getHarness(MatFormFieldHarness); const errors = await formField.getTextErrors(); }) Component-Harness-API is asynchronous
48. COMPONENT HARNESS const errors = await formField.getTextErrors(); expect(errors).toContain( 'ISBN has to be at least 3 characters long.'); Get state information from component.
49. Predictability
50. False Positives
51. FALSE POSITIVES Negative assertions - Negative assertions check things which does not exist or are not allowed to happen. - Negative assertions can lead into False Positives.
52. consumer.service describe('Negative assertion test', () => { it('passes because assertion is not specific enough', () => { const service = mock(Service); const consumer = new Consumer(instance(service)); consumer.do(); verify(service.doWork('1', '2')).never(); }); }); FALSE POSITIVES Negative assertions class Consumer { constructor(private service: Service) {} do() { this.service.doWork('1', '2', true); } } Test is green, but the method was called.
53. FALSE POSITIVES Negative assertions - Negative assertion can also happen in UI-Tests. - We use for our demos. - BTW: Protractor officially reaches end-of-life in 2022
55. FALSE POSITIVES Asynchronous tests it('broken: is green because done was forgotten', () => { defer(() => throwError('My error')).subscribe({ next: () => expect(true).toBe(false), // should make test red error: () => expect(true).toBe(false), // should make test red }); }); Test should be red, but...
56. FALSE POSITIVES Asynchronous tests | Done Callback it('done: evaluates error correctly', done => { defer(() => throwError('My error')).subscribe({ error: (err) => { expect(err).toBe('My error'); done(); }, }); }); Make test aware of asynchronous process.
57. FALSE POSITIVES Asynchronous tests | Promise it('promisify: evaluates error correctly', async () => { const promise = defer(() => throwError('My error')).toPromise(); await expect(promise).rejects.toEqual('My error'); }); Sometimes it is even cleaner to convert the Observable to a Promise.
58. Flakiness
59. FLAKINESS Network dependencies - Occur mostly in UI Tests - The test runner checks if the DOM matches the expectation at a certain time - When the DOM is not ready, the test fails even if the expectation matches a few milliseconds later.
60. FLAKINESS Network dependencies Test Command cy.get(...) Network Operation Test fails DOM update is too late, DOM observes
61. PREDICTABILITY Summary - Use timeouts in your tests to make them pass. - Analyse the root cause why a test is flaky - Be always aware of asynchronous operations - Be as concrete as possible when using negative assertions DON’T DO
62. Independence
63. Grade of Independence How many dependencies does the subject under test have? How many dependencies does the test have to other tests? Often accidentally
64. Scope
65. Scope Dependencies of the subject under test: • to other modules • to external systems • to the execution environment • ...
66. SCOPE Module Level • Logical: Service / Pipe / Class / Function / ... • Visual: Component / Directive (includes DOM) Application Level • Application Pages / Screens • Application Use cases Subject under Test
67. Scope Broader scope logical visual Module Level App Level Screens Use Cases More dependencies
68. Scope: Module Level Broader scope App Level Screens Use Cases Jest / Karma Isolated Shallow Full Module Level More dependencies
69. Scope @Component({ selector: 'todos', template: ` <h1>Todos</h1> <todo-quick-add (create)="addTodo($event)"></todo-quick-add> <todo-checker [todo]="todo" *ngFor="let todo of todos"></todo-checker> `, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Component Under Test Adds one initial todo Child component for adding a todo Child component for todo checkbox
70. Scope @Component({ selector: 'todos', template: ` <h1>Todos</h1> <todo-quick-add (create)="addTodo($event)"></todo-quick-add> <todo-checker [todo]="todo" *ngFor="let todo of todos"></todo-checker> `, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Isolated Isolated Unit Test
71. Scope it('should initialize todos', async () => { const component = new TodosComponent(); component.ngOnInit(); expect(component.todos).toEqual([{ text: 'initial', isDone: false }]); }); Isolated Unit Test Without TestBed You are responsible for Angular mechanics “Classic” unit test
72. Scope @Component({ selector: 'todos', template: ` <h1>Todos</h1> <todo-quick-add (create)="addTodo($event)"></todo-quick-add> <todo-checker [todo]="todo" *ngFor="let todo of todos"></todo-checker> `, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Shallow Shallow Unit Test
73. Scope it('should initialize todos', async () => { await render(TodosComponent, { declarations: [TodoCheckerComponent], schemas: [NO_ERRORS_SCHEMA], }); expect( screen.getByRole('checkbox', { name: 'initial' }) ).toBeInTheDocument(); }); Shallow Unit Test Ignore all other components Include direct child components needed for this test, without its child components!
74. it('should add todo', async () => { const { fixture, detectChanges } = await render(TodosComponent, { declarations: [TodoCheckerComponent], schemas: [NO_ERRORS_SCHEMA], }); fixture.componentInstance.addTodo({ text: 'added', isDone: false }); detectChanges(); expect( screen.getByRole('checkbox', { name: 'added' }) ).toBeInTheDocument(); }); Access to componentInstance needed Scope Shallow Unit Test
75. Scope @Component({ selector: 'todos', template: ` <h1>Todos</h1> <todo-quick-add (create)="addTodo($event)"></todo-quick-add> <todo-checker [todo]="todo" *ngFor="let todo of todos"></todo-checker> `, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Full Full Unit/Integration Test
76. Scope it('should add todo', async () => { await render(TodosComponent, { declarations: [TodoCheckerComponent, TodoQuickAddComponent], }); userEvent.type(screen.getByRole('textbox'), 'added'); userEvent.click(screen.getByRole('button', { name: 'Add' })); expect( screen.getByRole('checkbox', { name: 'initial' }) ).toBeInTheDocument(); expect( screen.getByRole('checkbox', { name: 'added' }) ).toBeInTheDocument(); }); Full Unit/Integration Test Include all child components with their dependencies
77. Which Type to choose? Isolated Shallow Full Pros: Fastest, most direct setup Fast, low maintenance Most confidence Be aware of: -No DOM -Angular mechanics manual (e.g. ngOnInit) - sometimes hard to test component interaction -Slower, especially with large modules -Breaks more often (false negatives) -Dependencies of dependencies When to use: -Testing algorithms -Testing logical units like services & pipes -Testing single part of component in isolation -Testing component integration -In combination with shallow tests
78. Test Confidence
79. Scope Broader scope More dependencies Jest / Karma Isolated Shallow (NO_ERRORS_SCHEMA) Full Module Level Longer duration More confidence
80. Scope Summary - Rely on a single type of test - Mess with too many dependencies - Lots of shallow tests - Some full/integration tests for confidence - Isolated tests for logical modules DON’T DO
81. What about logical dependencies? Shallow TestBed tests handle visual dependencies (components, directives). We still have dependencies to services, InjectionTokens, third-party libs, ...
82. Mocking and Stubbing
83. Stubbing via Angular’s Dependency Injection Provide fake implementations { provide: TodosService, useValue: { query: () => of([ { text: 'test TODO 1' }, { text: 'test TODO 2' }, ]), }, } We provide a fake TodosService to avoid side effects
84. Avoid accessing JS-Globals directly constructor() { const host = document.location.hostname; const token = localStorage.getItem('token'); } const LOCAL_STORAGE = new InjectionToken<Storage>('LOCAL_STORAGE', { providedIn: 'root', factory: () => localStorage, }); constructor( @Inject(DOCUMENT) document: Document, @Inject(LOCAL_STORAGE) localStorage: Storage ) { const host = document.location.hostname; const token = localStorage.getItem('token'); } Not testable via DI and easy to forget → test could influence other tests This also works for 3rd-party libs that add to window Don’t Do
85. Mocking Record calls for behavior verification style testing const queryMock = jest.fn().mockReturnValue([]); // in TestBed { provide: TodosService, useValue: { query: queryMock }, } expect(queryMock.mock.calls.length).toBe(1); expect(queryMock.mock.calls[0]).toEqual(["all"]); createSpy / createSpyObj when using Jasmine
86. What if dependency is changed? { provide: TodosService, useValue: { query: () => of([ { text: 'TODO 1' }, { text: 'TODO 2' }, ]), }, } export class TodosService { query(): Observable<Todo[]> { // implementation... } } ⚡Renaming: query → getTodos ⚡Renaming: Todo.text → Todo.title
87. Runtime errors or no error at all! { provide: TodosService, useValue: { query: () => of([ { text: 'TODO 1' }, { text: 'TODO 2' }, ]), }, } export class TodosService { query(): Observable<Todo[]> { // implementation... } } ⚡Renaming: query → getTodos No type safety ⚡Renaming: Todo.text → Todo.title
88. With types: Compile-time error { provide: TodosService, useValue: { oldName: () => of([ { wrong: 'test TODO 1' } as Todo, { wrong: 'test TODO 2' } as Todo ]) } as TodosService } 😍 Conversion of type '{ oldName: () => Observable<Todo[]>; }' to type 'TodosService' may be a mistake... Conversion of type '{ wrong: string; }' to type 'Todo' may be a mistake... With type safety Use types
89. Type your stubs! The less the stubbed object diverges from the original, the more confidence it gives. Changes in the original must be reflected! Subject under Test Original A Stubbed A Original A Stubbed A ⚡ False Positive!
90. (Some of the) Mocking Libraries that Help Framework-agnostic ts-jest jest-mock-extended Sinon.JS moq.ts ts-mockito Angular-specific Spectator (mockProvider) ng-mocks ng-mockito
91. Personal Hero: ts-mockito Setup: let mockTodosService = mock(TodosService); // set up stub when(mockTodosService.query(anyString())).thenReturn(of([])); // ...use in TestModule: providers: [{ provide: TodosService, useFactory: () => instance(mockTodosService), }], has type TodosService Type-safe + IDE autocompletion
92. Personal Hero: ts-mockito Behavior verification: verify(mockTodosService.query('all')).called(); const [param] = capture(mockTodosService.query).first(); expect(param).toEqual('all'); Type-safe + IDE autocompletion
93. Shameless plug: ng-mockito Thin layer on top of ts-mockito providers: [ mockNg(TodosService, (mock) => when(mock.query(anyString())).thenReturn(NEVER) ), mockNg([APP_TITLE, TodosComponent], { use: 'Test Heading' }) ] Infers APP_TITLE InjectionToken type from TodosComponent’s constructor Inline stubbing mockProvider, mockPipe, mockComponent, ...
94. it('should add todo', async () => { const { fixture, detectChanges } = await render(TodosComponent, { declarations: [TodoCheckerComponent], schemas: [NO_ERRORS_SCHEMA], }); fixture.componentInstance.addTodo({ text: 'added', isDone: false }); detectChanges(); expect( screen.getByRole('checkbox', { name: 'added' }) ).toBeInTheDocument(); }); Access to componentInstance needed Shallow Unit Test Revisited
95. it('should add todo', async () => { const createEvent = new EventEmitter<Todo>(); const { detectChanges } = await render(TodosComponent, { declarations: [ TodoCheckerComponent, mockNg(TodoQuickAddComponent, (mock) => when(mock.create).thenReturn(createEvent) ), ], schemas: [NO_ERRORS_SCHEMA], }); createEvent.next({ text: 'added', isDone: false }); detectChanges(); expect(screen.getByRole('checkbox', { name: 'added' })) .toBeInTheDocument(); }); Shallow Unit Test Revisited: Component Stub Mock @Output: (create)="addTodo($event)"
96. Mock all the things? Broader scope Jest / Karma Mocked Dependencies Real Dependencies Module Level More confidence Less Isolation
97. Mocking and Stubbing Summary - Access globals directly, it makes them hard to stub - Forget to mock globals - Use untyped stubs - Mock globals like localStorage to avoid dependencies between tests - Use Libraries for mocking, e.g. ts-mockito and ng-mockito - Make sure that changes in the dependency are reflected in your stubs DON’T DO
98. Cypress Integration Testing
99. Scope Broader scope Jest Isolated Shallow Full Module Level App Level Cypress Integration (Fixtures) E2E
100. Fixtures for Test Isolation Instead of Server Resets • Stub POST/PUT/DELETE to avoid state changes on server • Stub GET to create starting state Be aware: State transitions (e.g. different response after PUSH) must be recreated for test → real e2e tests give more confidence
101. Integration Testing with Cypress Stubbing, Cypress style: cy.intercept('/api?query=all', { body: [ { text: 'test TODO 1' }, { text: 'test TODO 2' }, ], }); Automatically fakes api response
102. Typed Fixtures cy.intercept('/api?query=all', { body: [ { text: 'test TODO 1' }, { text: 'test TODO 2' }, ] as Todo[], }); Extract into shared lib and import in spec
103. Fixtures are Untyped Fixtures need same handling like other stubs. Changes in the original must be reflected! Cypress Test Original Response Stubbed Response Changed Response Stubbed Response False Positive!
104. Cypress Component Testing
105. Cypress Component Tests Broader scope Jest Isolated Shallow Full Module Level App Level Integration (Fixtures) E2E Cypress Component Tests
106. Storybook Storybook is a UI component explorer, mainly for • Presentation components • UI libraries
107. Cypress Storybook Test describe('TodoCheckerComponent', () => { it('should show todo item', () => { cy.visit('/iframe.html?id=todocheckercomponent--primary'); cy.findByRole('checkbox', { name: 'Test TODO' }).should(...); }); }); Accesses iframe
108. Cypress Storybook Test Setup Base component setup in Storybook Controls can be used to provide story parameters for Cypress. cy.visit('/iframe.html?id=todocheckercomponent--primary&args=text:Some+other+todo');
109. Cypress Storybook: Use or Not? Depends on your project. Do you also profit from using Storybook as documentation? Do you develop a UI library (“widgets”)? Do you use Jest? Maybe completely replace full TestBed tests?
110. Alternative: “real” Cypress Component Tests Disclaimer: Brand new (Alpha) Angular support currently not out-of-the-box (@jscutlery/cypress-angular)
111. Cypress Component Test No need to visit, directly mount component: describe('TodoCheckerComponent', () => { it('should show text', () => { const todo: Todo = { text: 'Wow', isDone: true }; mount(TodoCheckerComponent, { inputs: { todo, }, }); cy.findByRole('checkbox', { name: 'Wow' }).should(...); }); });
112. Why Cypress for Component Testing? Setup complexity equal to full Karma/Jest Integration Tests But: • Better Tooling (watch mode, videos, replayability, …) • Browser Environment (especially useful when using Jest for other tests) with easy debuggability • Easier Network handling • Automatic waiting for DOM element changes
113. Testing Pyramid (upside down) Broader scope Jest Isolated Shallow (NO_ERRORS_SCHEMA) Full Module Level App Level Integration / E2E Cypress Component Tests Is this still valid? do less (slow + expensive) do many (fast + cheap)
114. Costs of UI tests decreased Broader scope Jest Isolated Shallow (NO_ERRORS_SCHEMA) Full Module Level App Level Integration / E2E Cypress Component Tests This is faster and cheaper now!
115. CYPRESS Summary - Use untyped fixtures - Use Storybook Tests if you don’t intend to use Storybook - Lots of Integration Testing using Fixtures - Mix in real E2E tests for confidence - Try Component Testing as Integration Test Alternative DON’T DO
116. Thank you for having us Markus Ende @_der_markusende & Gregor Woiwode @GregOnNet Photo by James Peacock on Unsplash