Share
Resilient UI Test Patterns
This quality assurance discipline is often an unpopular part of software development.
In Angular, this is due to the fact that mechanisms such as manual triggering of change detection or complex configuration with TestBeds are difficult to manage.
Something is quickly forgotten, as a result of which the test either does not work or becomes a false positive.
Resilient test procedures provide you with readable and maintainable tests.
This includes best refactoring practices, as well as various tools and frameworks.
You will learn the following techniques in the lecture to secure your tests against external changes:
- The Angular Testing Library makes it easy for you to write tests that are easy to understand.
- Typed mocks sound the alarm as soon as contracts are significantly changed in the front end.
- Storybook allows isolated UI component tests with Cypress.
After this lecture, you can design and implement resilient tests.
https://www.developer-week.de/programm/#/talk/resilient-ui-test-patterns-2
Gregor Woiwode
Transcript
116 slides
-
Resilient UI Test Patterns Markus Ende & Gregor Woiwode Photo
by James Peacock on Unsplash -
Software Architect & Co-founder of qupaya technologies GmbH mailto: [email protected]
twitter: @_der_markusende / @qupaya Markus Ende -
Software Architect & CTO of co-IT.eu GmbH mailto: [email protected] twitter:
@GregOnNet oss: Gregor Woiwode 👋 -
- cannot afford to fix all tests. - Loss of
confidence in tests - Begin of deleting/skipping test - Increase of regression bugs - has less confidence using . CONFIDENCE If tests break frequently... -
Confidence The needs of our customers change quickly. We need
confidence to rethink & re-design systems fast without harming the quality of our product. Therefore, we need trust in our systems, but also trust in our automated tests. -
Confidence We need tests which can tell us the truth
about the state of our software product. Therefore, we have to stabilize our test continuously. -
Resilience in the Context of Testing Resilience is a continuous
process, making tests more stable against environment changes. The main goal is to increase maintainability of tests to allow developers to focus on feature development. -
DAMP DAMP stands for "Descriptive and Meaningful Phrases" and promotes
the readability of the code. - Enterprise Craftsmanship -
DAMP over DRY DAMP Benefits - Reduce time reading through
test-helper abstractions. - Tests are more isolated from each other. - Test contains all information needed to understand what is going on. -
DAMP over DRY DON’T describe('Product', () => { it('can select',
() => {}); describe('When a product is selected', () => { it('is marked with a green tick', () => {}); it('gets a green background color', () => {}); }); DO Test Description -
DAMP over DRY Summary - Use shorthands - Be too
general - Use technical terms (like Button, Array or Error) DON’T DO - Describe the behaviour of your algorithm. - Formulate phrases a non-technician can understand. - Provide as much context as possible to your fellow developers. -
DAMP over DRY Prefer duplication over the wrong abstraction, and
optimize for change first. - Kent C. Dodds (https://kentcdodds.com/blog/avoid-nesting-when-youre-testing) Hint -
Cognitive Load - How long does it take for one
of your team members to understand your test? -
Cognitive Load Test Setup beforeEach(() => { TestBed.configureTestingModule({ /* ...
*/ }); fixture = TestBed.createComponent(TodosComponent); component = fixture.componentInstance; fixture.detectChanges(); }); DON’T start with one global setup for all tests. -
Cognitive Load Test Setup it('is as concrete as possible', ()
=> { TestBed.configureTestingModule({ /*...*/ }); fixture = TestBed.createComponent(TodosComponent); component = fixture.componentInstance; fixture.detectChanges(); }); DO -
COGNITIVE LOAD Summary - Allow too large setups - Implement
logic in your tests - Share state between tests - Tolerate outdated tests - Control the amount of test dependencies - Ensure that test description and implementation still match - Keep state isolated inside one test DON’T DO -
Testing Library Simple and complete testing utilities that encourage good
testing practices. testing-library.com 🛠 Write Maintainable Tests ✅ Develop with Confidences 🎉 Accessible by Default -
Testing Library TESTING LIBRARY Setup await render(TodoCheckerComponent, { componentProperties: {
todo: { isDone: false, text: 'Buy milk' } } }); Vanilla Angular TestBed.configureTestingModule({ declarations: [TodoCheckerComponent] }); fixture = TestBed.createComponent( TodoCheckerComponent); fixture.componentInstance.todo = { isDone: false, text: 'Buy milk' }; fixture.detectChanges(); -
TESTING LIBRARY Template Testing Library Vanilla Angular const checkbox: =
() => fixture.debugElement.query( By.css('input[type=checkbox]') ).nativeElement; checkbox().click(); checkbox().dispatchEvent( new Event('click') ); fixture.detectChanges(); expect(checkbox().checked).toBe(true); userEvent.click( screen.getByRole('checkbox') ); expect( screen.getByRole('checkbox') ).toBeChecked(); -
TESTING LIBRARY Events Testing Library Vanilla Angular fixture.debugElement.query( By.css('input[type=checkbox]') ).nativeElement.click();
fixture.detectChanges(); userEvent.click( screen.getByRole( 'Button', { name: 'Add' } ) ); -
Template Selectors Testing Library Template expect( screen.getByTestId('todo-checkbox') ).not.toBeChecked(); <input type="checkbox"
data-testId="todo-checkbox" [checked]="todo.isDone" (change)="emitToggle()" /> -
Template Selectors Testing Library Template Role Selectors <input type="checkbox" name="todo"
[checked]="todo.isDone" (change)="emitToggle()" /> expect( screen.getByRole( 'checkbox', { name: 'todo' } ) ).not.toBeChecked(); -
💁 Each Material Component ships with its Test-API. Check out
the Documentation to learn about the details. - material.angular.io -
COMPONENT HARNESS <mat-form-field data-test="isbn-field"> <input matInput formControlName="isbn" placeholder="ISBN" required />
<mat-error *ngIf="form.get('isbn')?.hasError('minlength')"> ISBN has to be at least 3 characters long. </mat-error> <mat-error *ngIf="form.get('isbn')?.hasError('required')"> ISBN is required </mat-error> </mat-form-field> How would you test mat-form-field? -
COMPONENT HARNESS import { TestbedHarnessEnvironment, HarnessLoader } from '@angular/cdk/testing/testbed'; let
fixture: ComponentFixture<SomeComponent>; let loader: HarnessLoader; fixture = TestBed.createComponent(SomeComponent); loader = TestbedHarnessEnvironment.loader(fixture); Setup Harness Environment -
COMPONENT HARNESS Testing Library Integration import { TestbedHarnessEnvironment, HarnessLoader }
from '@angular/cdk/testing/testbed'; // ... let fixture: ComponentFixture<SomeComponent>; let loader: HarnessLoader; // ... const { fixture} = await render(SomeComponent); loader = TestbedHarnessEnvironment.loader(fixture); -
COMPONENT HARNESS import { MatFormFieldHarness } from '@angular/material/form-field/testing'; const formField
= await loader.getHarness(MatFormFieldHarness); Get Harness for specific material component -
COMPONENT HARNESS import { MatFormFieldHarness } from '@angular/material/form-field/testing'; const formField
= await loader.getHarness( MatFormFieldHarness.with({ selector: '[data-test=isbn-field]' }) ); Specify resilient selectors -
COMPONENT HARNESS import { MatInputHarness } from '@angular/material/input/testing'; const formField
= await loader.getHarness(/* … */); const input = (await formField.getControl()) as MatInputHarness; Access child material component -
COMPONENT HARNESS it('test with Angular Material component', async () =>
{ const formField = await loader.getHarness(MatFormFieldHarness); const errors = await formField.getTextErrors(); }) Component-Harness-API is asynchronous -
COMPONENT HARNESS const errors = await formField.getTextErrors(); expect(errors).toContain( 'ISBN has
to be at least 3 characters long.'); Get state information from component. -
FALSE POSITIVES Negative assertions - Negative assertions check things which
does not exist or are not allowed to happen. - Negative assertions can lead into False Positives. -
consumer.service describe('Negative assertion test', () => { it('passes because assertion
is not specific enough', () => { const service = mock(Service); const consumer = new Consumer(instance(service)); consumer.do(); verify(service.doWork('1', '2')).never(); }); }); FALSE POSITIVES Negative assertions class Consumer { constructor(private service: Service) {} do() { this.service.doWork('1', '2', true); } } Test is green, but the method was called. -
FALSE POSITIVES Negative assertions - Negative assertion can also happen
in UI-Tests. - We use for our demos. - BTW: Protractor officially reaches end-of-life in 2022 -
FALSE POSITIVES Asynchronous tests it('broken: is green because done was
forgotten', () => { defer(() => throwError('My error')).subscribe({ next: () => expect(true).toBe(false), // should make test red error: () => expect(true).toBe(false), // should make test red }); }); Test should be red, but... -
FALSE POSITIVES Asynchronous tests | Done Callback it('done: evaluates error
correctly', done => { defer(() => throwError('My error')).subscribe({ error: (err) => { expect(err).toBe('My error'); done(); }, }); }); Make test aware of asynchronous process. -
FALSE POSITIVES Asynchronous tests | Promise it('promisify: evaluates error correctly',
async () => { const promise = defer(() => throwError('My error')).toPromise(); await expect(promise).rejects.toEqual('My error'); }); Sometimes it is even cleaner to convert the Observable to a Promise. -
FLAKINESS Network dependencies - Occur mostly in UI Tests -
The test runner checks if the DOM matches the expectation at a certain time - When the DOM is not ready, the test fails even if the expectation matches a few milliseconds later. -
FLAKINESS Network dependencies Test Command cy.get(...) Network Operation Test fails
DOM update is too late, DOM observes -
PREDICTABILITY Summary - Use timeouts in your tests to make
them pass. - Analyse the root cause why a test is flaky - Be always aware of asynchronous operations - Be as concrete as possible when using negative assertions DON’T DO -
Grade of Independence How many dependencies does the subject under
test have? How many dependencies does the test have to other tests? Often accidentally -
Scope Dependencies of the subject under test: • to other
modules • to external systems • to the execution environment • ... -
SCOPE Module Level • Logical: Service / Pipe / Class
/ Function / ... • Visual: Component / Directive (includes DOM) Application Level • Application Pages / Screens • Application Use cases Subject under Test -
Scope Broader scope logical visual Module Level App Level Screens
Use Cases More dependencies -
Scope: Module Level Broader scope App Level Screens Use Cases
Jest / Karma Isolated Shallow Full Module Level More dependencies -
Scope @Component({ selector: 'todos', template: ` <h1>Todos</h1> <todo-quick-add (create)="addTodo($event)"></todo-quick-add> <todo-checker
[todo]="todo" *ngFor="let todo of todos"></todo-checker> `, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Component Under Test Adds one initial todo Child component for adding a todo Child component for todo checkbox -
Scope @Component({ selector: 'todos', template: ` <h1>Todos</h1> <todo-quick-add (create)="addTodo($event)"></todo-quick-add> <todo-checker
[todo]="todo" *ngFor="let todo of todos"></todo-checker> `, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Isolated Isolated Unit Test -
Scope it('should initialize todos', async () => { const component
= new TodosComponent(); component.ngOnInit(); expect(component.todos).toEqual([{ text: 'initial', isDone: false }]); }); Isolated Unit Test Without TestBed You are responsible for Angular mechanics “Classic” unit test -
Scope @Component({ selector: 'todos', template: ` <h1>Todos</h1> <todo-quick-add (create)="addTodo($event)"></todo-quick-add> <todo-checker
[todo]="todo" *ngFor="let todo of todos"></todo-checker> `, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Shallow Shallow Unit Test -
Scope it('should initialize todos', async () => { await render(TodosComponent,
{ declarations: [TodoCheckerComponent], schemas: [NO_ERRORS_SCHEMA], }); expect( screen.getByRole('checkbox', { name: 'initial' }) ).toBeInTheDocument(); }); Shallow Unit Test Ignore all other components Include direct child components needed for this test, without its child components! -
it('should add todo', async () => { const { fixture,
detectChanges } = await render(TodosComponent, { declarations: [TodoCheckerComponent], schemas: [NO_ERRORS_SCHEMA], }); fixture.componentInstance.addTodo({ text: 'added', isDone: false }); detectChanges(); expect( screen.getByRole('checkbox', { name: 'added' }) ).toBeInTheDocument(); }); Access to componentInstance needed Scope Shallow Unit Test -
Scope @Component({ selector: 'todos', template: ` <h1>Todos</h1> <todo-quick-add (create)="addTodo($event)"></todo-quick-add> <todo-checker
[todo]="todo" *ngFor="let todo of todos"></todo-checker> `, }) export class TodosComponent implements OnInit { todos: Todo[] = []; ngOnInit(): void { this.todos = [{ text: 'initial', isDone: false }]; } addTodo(todo: Todo) { this.todos = [...this.todos, todo]; } } Full Full Unit/Integration Test -
Scope it('should add todo', async () => { await render(TodosComponent,
{ declarations: [TodoCheckerComponent, TodoQuickAddComponent], }); userEvent.type(screen.getByRole('textbox'), 'added'); userEvent.click(screen.getByRole('button', { name: 'Add' })); expect( screen.getByRole('checkbox', { name: 'initial' }) ).toBeInTheDocument(); expect( screen.getByRole('checkbox', { name: 'added' }) ).toBeInTheDocument(); }); Full Unit/Integration Test Include all child components with their dependencies -
Which Type to choose? Isolated Shallow Full Pros: Fastest, most
direct setup Fast, low maintenance Most confidence Be aware of: -No DOM -Angular mechanics manual (e.g. ngOnInit) - sometimes hard to test component interaction -Slower, especially with large modules -Breaks more often (false negatives) -Dependencies of dependencies When to use: -Testing algorithms -Testing logical units like services & pipes -Testing single part of component in isolation -Testing component integration -In combination with shallow tests -
Scope Broader scope More dependencies Jest / Karma Isolated Shallow
(NO_ERRORS_SCHEMA) Full Module Level Longer duration More confidence -
Scope Summary - Rely on a single type of test
- Mess with too many dependencies - Lots of shallow tests - Some full/integration tests for confidence - Isolated tests for logical modules DON’T DO -
What about logical dependencies? Shallow TestBed tests handle visual dependencies
(components, directives). We still have dependencies to services, InjectionTokens, third-party libs, ... -
Stubbing via Angular’s Dependency Injection Provide fake implementations { provide:
TodosService, useValue: { query: () => of([ { text: 'test TODO 1' }, { text: 'test TODO 2' }, ]), }, } We provide a fake TodosService to avoid side effects -
Avoid accessing JS-Globals directly constructor() { const host = document.location.hostname;
const token = localStorage.getItem('token'); } const LOCAL_STORAGE = new InjectionToken<Storage>('LOCAL_STORAGE', { providedIn: 'root', factory: () => localStorage, }); constructor( @Inject(DOCUMENT) document: Document, @Inject(LOCAL_STORAGE) localStorage: Storage ) { const host = document.location.hostname; const token = localStorage.getItem('token'); } Not testable via DI and easy to forget → test could influence other tests This also works for 3rd-party libs that add to window Don’t Do -
Mocking Record calls for behavior verification style testing const queryMock
= jest.fn().mockReturnValue([]); // in TestBed { provide: TodosService, useValue: { query: queryMock }, } expect(queryMock.mock.calls.length).toBe(1); expect(queryMock.mock.calls[0]).toEqual(["all"]); createSpy / createSpyObj when using Jasmine -
What if dependency is changed? { provide: TodosService, useValue: {
query: () => of([ { text: 'TODO 1' }, { text: 'TODO 2' }, ]), }, } export class TodosService { query(): Observable<Todo[]> { // implementation... } } ⚡Renaming: query → getTodos ⚡Renaming: Todo.text → Todo.title -
Runtime errors or no error at all! { provide: TodosService,
useValue: { query: () => of([ { text: 'TODO 1' }, { text: 'TODO 2' }, ]), }, } export class TodosService { query(): Observable<Todo[]> { // implementation... } } ⚡Renaming: query → getTodos No type safety ⚡Renaming: Todo.text → Todo.title -
With types: Compile-time error { provide: TodosService, useValue: { oldName:
() => of([ { wrong: 'test TODO 1' } as Todo, { wrong: 'test TODO 2' } as Todo ]) } as TodosService } 😍 Conversion of type '{ oldName: () => Observable<Todo[]>; }' to type 'TodosService' may be a mistake... Conversion of type '{ wrong: string; }' to type 'Todo' may be a mistake... With type safety Use types -
Type your stubs! The less the stubbed object diverges from
the original, the more confidence it gives. Changes in the original must be reflected! Subject under Test Original A Stubbed A Original A Stubbed A ⚡ False Positive! -
(Some of the) Mocking Libraries that Help Framework-agnostic ts-jest jest-mock-extended
Sinon.JS moq.ts ts-mockito Angular-specific Spectator (mockProvider) ng-mocks ng-mockito -
Personal Hero: ts-mockito Setup: let mockTodosService = mock(TodosService); // set
up stub when(mockTodosService.query(anyString())).thenReturn(of([])); // ...use in TestModule: providers: [{ provide: TodosService, useFactory: () => instance(mockTodosService), }], has type TodosService Type-safe + IDE autocompletion -
Personal Hero: ts-mockito Behavior verification: verify(mockTodosService.query('all')).called(); const [param] = capture(mockTodosService.query).first();
expect(param).toEqual('all'); Type-safe + IDE autocompletion -
Shameless plug: ng-mockito Thin layer on top of ts-mockito providers:
[ mockNg(TodosService, (mock) => when(mock.query(anyString())).thenReturn(NEVER) ), mockNg([APP_TITLE, TodosComponent], { use: 'Test Heading' }) ] Infers APP_TITLE InjectionToken type from TodosComponent’s constructor Inline stubbing mockProvider, mockPipe, mockComponent, ... -
it('should add todo', async () => { const { fixture,
detectChanges } = await render(TodosComponent, { declarations: [TodoCheckerComponent], schemas: [NO_ERRORS_SCHEMA], }); fixture.componentInstance.addTodo({ text: 'added', isDone: false }); detectChanges(); expect( screen.getByRole('checkbox', { name: 'added' }) ).toBeInTheDocument(); }); Access to componentInstance needed Shallow Unit Test Revisited -
it('should add todo', async () => { const createEvent =
new EventEmitter<Todo>(); const { detectChanges } = await render(TodosComponent, { declarations: [ TodoCheckerComponent, mockNg(TodoQuickAddComponent, (mock) => when(mock.create).thenReturn(createEvent) ), ], schemas: [NO_ERRORS_SCHEMA], }); createEvent.next({ text: 'added', isDone: false }); detectChanges(); expect(screen.getByRole('checkbox', { name: 'added' })) .toBeInTheDocument(); }); Shallow Unit Test Revisited: Component Stub Mock @Output: (create)="addTodo($event)" -
Mock all the things? Broader scope Jest / Karma Mocked
Dependencies Real Dependencies Module Level More confidence Less Isolation -
Mocking and Stubbing Summary - Access globals directly, it makes
them hard to stub - Forget to mock globals - Use untyped stubs - Mock globals like localStorage to avoid dependencies between tests - Use Libraries for mocking, e.g. ts-mockito and ng-mockito - Make sure that changes in the dependency are reflected in your stubs DON’T DO -
Scope Broader scope Jest Isolated Shallow Full Module Level App
Level Cypress Integration (Fixtures) E2E -
Fixtures for Test Isolation Instead of Server Resets • Stub
POST/PUT/DELETE to avoid state changes on server • Stub GET to create starting state Be aware: State transitions (e.g. different response after PUSH) must be recreated for test → real e2e tests give more confidence -
Integration Testing with Cypress Stubbing, Cypress style: cy.intercept('/api?query=all', { body:
[ { text: 'test TODO 1' }, { text: 'test TODO 2' }, ], }); Automatically fakes api response -
Typed Fixtures cy.intercept('/api?query=all', { body: [ { text: 'test TODO
1' }, { text: 'test TODO 2' }, ] as Todo[], }); Extract into shared lib and import in spec -
Fixtures are Untyped Fixtures need same handling like other stubs.
Changes in the original must be reflected! Cypress Test Original Response Stubbed Response Changed Response Stubbed Response False Positive! -
Cypress Component Tests Broader scope Jest Isolated Shallow Full Module
Level App Level Integration (Fixtures) E2E Cypress Component Tests -
Storybook Storybook is a UI component explorer, mainly for •
Presentation components • UI libraries -
Cypress Storybook Test describe('TodoCheckerComponent', () => { it('should show todo
item', () => { cy.visit('/iframe.html?id=todocheckercomponent--primary'); cy.findByRole('checkbox', { name: 'Test TODO' }).should(...); }); }); Accesses iframe -
Cypress Storybook Test Setup Base component setup in Storybook Controls
can be used to provide story parameters for Cypress. cy.visit('/iframe.html?id=todocheckercomponent--primary&args=text:Some+other+todo'); -
Cypress Storybook: Use or Not? Depends on your project. Do
you also profit from using Storybook as documentation? Do you develop a UI library (“widgets”)? Do you use Jest? Maybe completely replace full TestBed tests? -
Alternative: “real” Cypress Component Tests Disclaimer: Brand new (Alpha) Angular
support currently not out-of-the-box (@jscutlery/cypress-angular) -
Cypress Component Test No need to visit, directly mount component:
describe('TodoCheckerComponent', () => { it('should show text', () => { const todo: Todo = { text: 'Wow', isDone: true }; mount(TodoCheckerComponent, { inputs: { todo, }, }); cy.findByRole('checkbox', { name: 'Wow' }).should(...); }); }); -
Why Cypress for Component Testing? Setup complexity equal to full
Karma/Jest Integration Tests But: • Better Tooling (watch mode, videos, replayability, …) • Browser Environment (especially useful when using Jest for other tests) with easy debuggability • Easier Network handling • Automatic waiting for DOM element changes -
Testing Pyramid (upside down) Broader scope Jest Isolated Shallow (NO_ERRORS_SCHEMA)
Full Module Level App Level Integration / E2E Cypress Component Tests Is this still valid? do less (slow + expensive) do many (fast + cheap) -
Costs of UI tests decreased Broader scope Jest Isolated Shallow
(NO_ERRORS_SCHEMA) Full Module Level App Level Integration / E2E Cypress Component Tests This is faster and cheaper now! -
CYPRESS Summary - Use untyped fixtures - Use Storybook Tests
if you don’t intend to use Storybook - Lots of Integration Testing using Fixtures - Mix in real E2E tests for confidence - Try Component Testing as Integration Test Alternative DON’T DO -
Thank you for having us Markus Ende @_der_markusende & Gregor
Woiwode @GregOnNet Photo by James Peacock on Unsplash