Test Doubles Terminology, Definitions and Illustrations with Examples Part 1 Test Doubles @philip_schwarz slides by https://fpilluminated.org/ Examples 

This deck is about a subset of the test automation patterns in Gerard Meszaros’ great book, xUnit Test Patterns – Refactoring Test Code. The subset in question consists of the patterns relating to the concept of Test Doubles. The deck is inspired by the patterns, and heavily reliant on extracts from the book. The motivation for the deck is my belief that it is quite beneficial, when using and discussing Test Doubles, to rely on standardised terminology and patterns. Later on we’ll see how Gerard Meszaros describes the benefits of such standardisation. @gerardmes Gerard Meszaros @philip_schwarz This deck was designed using resources from Flaticon.com 

While this deck may serve as an introduction to test doubles, or help reinforce the topic, it is aimed at spelling out in sufficient detail the context and terminology necessary for a firm understanding of the different types (variations) of Test Double, and their motivation. Here is a diagram showing the different types. It is based on a diagram found in xUnit Test Patterns, but I have annotated it with icons and added two types (variations) of Test Stub. 

One of the small challenges faced when presenting the material covered in this slide deck is that some terms are better defined only after introducing certain topics which: • provide essential context for understanding the terms’ definitions… • …and yet contain a few forward references to those as yet undefined terms! In upcoming slides for example, you’ll come across two or three forward references to both of the following terms: In practice however, it is quite feasible to make notes (mental or otherwise) of such forward references, so that when the definitions of the terms are reached, the sections containing the forward references can then be revisited in order to fully understand them. Test Stub Mock Object 

In xUnit Test Patterns, each of the test doubles highlighted in green is a test pattern, while the ones highlighted in orange are variations of the Test Stub pattern. Before we look at the patterns, we are going to familiarise ourselves with the following: 1. some essential terminology 2. two other test patterns - State Verification and Behaviour Verification. 

The next four slides consist of the the definition, from the glossary of xUnit Test Patterns, of the following terms : • system under test • depended-on component • interaction points: • control point • observation point • input: • direct • indirect • output: • direct • indirect 

system under test (SUT) Whatever thing we are testing. The SUT is always defined from the perspective of the test. When we are writing unit tests, the SUT is whatever class (also known as CUT), object (also known as OUT), or method (also known as MUT) we are testing; when we are writing customer tests, the SUT is probably the entire application (also known as AUT) or at least a major subsystem of it. The parts of the application that we are not verifying in this particular test may still be involved as a depended-on component (DOC). depended-on component (DOC) An individual class or a large-grained component on which the system under test (SUT) depends. The dependency is usually one of delegation via method calls. In test automation, the DOC is primarily of interest in that we need to be able to observe and control its interactions with the SUT to get complete test coverage. SUT DOC 

interaction point A point at which a test interacts with the system under test (SUT). An interaction point can be either a control point or an observation point. control point How the test asks the system under test (SUT) to do something for it. A control point could be created for the purpose of setting up or tearing down the fixture or it could be used during the exercise SUT phase of the test. It is a kind of interaction point. Some control points are provided strictly for testing purposes; they should not be used by the production code because they bypass input validation or short- circuit the normal life cycle of the SUT or some object on which it depends. observation point The means by which the test observes the behavior of the system under test (SUT). This kind of interaction point can be used to inspect the post-exercise state of the SUT or to monitor interactions between the SUT and its depended-on components. Some observation points are provided strictly for the tests; they should not be used by the production code because they may expose private implementation details of the SUT that cannot be depended on not to change. 

direct input A test may interact with the system under test (SUT) directly via its "front door" or public application programming interface (API) or indirectly via its "back door." The stimuli injected by the test into the SUT via its front door are direct inputs of the SUT. Direct inputs may consist of method or function calls to another component or messages sent on a message channel (e.g., MQ or JMS) and the arguments or contents thereof. indirect input When the behavior of the system under test (SUT) is affected by the values returned by another component whose services it uses, we call those values the indirect inputs of the SUT. Indirect inputs may consist of actual return values of functions, updated (out) parameters of procedures or subroutines, and any errors or exceptions raised by the depended-on component (DOC). Testing of the SUT behavior with indirect inputs requires the appropriate control point on the "back side" of the SUT. We often use a Test Stub to inject the indirect inputs into the SUT. SUT Test DOC SUT front door back door 

indirect output When the behavior of the system under test (SUT) includes actions that cannot be observed through the public application programming interface (API) of the SUT but that are seen or experienced by other systems or application components, we call those actions the indirect outputs of the SUT. Indirect outputs may consist of method or function calls to another component, messages sent on a message channel (e.g., MQ or JMS), and records inserted into a database or written to a file. Verification of the indirect output behaviors of the SUT requires the use of appropriate observation points on the "back side" of the SUT. Mock Objects are often used to implement the observation point by intercepting the indirect outputs of the SUT and comparing them to the expected values. direct output A test may interact with the system under test (SUT) directly via its "front door" or public application programming interface (API) or indirectly via its "back door." The responses received by the test from the SUT via its front door are direct outputs of the SUT. Direct outputs may consist of the return values of method or function calls, updated arguments passed by reference, exceptions raised by the SUT, or messages received on a message channel (e.g., MQ or JMS) from the SUT. SUT Test DOC SUT front door back door 

The next three slides contain diagrams aimed at reinforcing our understanding of the terms whose definitions we have just seen. 

output DOC SUT Test input output input output input direct input direct output indirect output indirect input Setup Exercise Verify Teardown Test input output 

output DOC SUT Test input output input output input direct input direct output indirect output indirect input Setup Exercise Verify Teardown Test input output A test may interact with the system under test (SUT) directly via its "front door" or public application programming interface (API) or indirectly via its "back door." front door back door 

output DOC SUT Test input output input output input direct input direct output indirect output indirect input potential observation points Setup Exercise Verify Teardown Test input output Verification of the indirect output behaviors of the SUT requires the use of appropriate observation points on the "back side" of the SUT. 

output DOC SUT Test input output input output input direct input direct output indirect output indirect input potential observation points Setup Exercise Verify Teardown Test input output Verification of the indirect output behaviors of the SUT requires the use of appropriate observation points on the "back side" of the SUT. Testing of the SUT behavior with indirect inputs requires the appropriate control point on the "back side" of the SUT. potential control points 

Having familiarised ourselves with that essential terminology, it is time to look at the State Verification pattern. Let’s see some excerpts from xUnit Test Patterns. 

State Verification How do we make tests self-checking when there is state to be verified? We inspect the state of the SUT after it has been exercised and compare it to the expected state. A Self-Checking Test (see page 26) must verify that the expected outcome has occurred without manual intervention by whoever is running the test. But what do we mean by "expected outcome"? The SUT may or may not be "stateful"; if it is stateful, it may or may not have a different state after it has been exercised. As test automaters, it is our job to determine whether our expected outcome is a change of final state or whether we need to be more specific about what occurs while the SUT is being exercised. State Verification involves inspecting the state of the SUT after it has been exercised. Also known as: State-Based Testing Fixture DOC Setup Exercise Verify Teardown SUT Get State Exercise B A C Behaviour (Indirect Outputs) Note that because we don’t intend to verify indirect output behaviours of the SUT, we are not setting up observation points on the "back side" of the SUT. State Verification no observation points observation points 

How It Works We exercise the SUT by invoking the methods of interest. Then, as a separate step, we interact with the SUT to retrieve its post-exercise state and compare it with the expected end state by calling Assertion Methods (page 362). Normally, we can access the state of the SUT simply by calling methods or functions that return its state. This is especially true when we are doing test-driven development because the tests will have ensured that the state is easily accessible. When we are retrofitting tests, however, we may find it more challenging to access the relevant state information. In these cases, we may need to use a Test-Specific Subclass (page 579) or some other technique to expose the state without introducing Test Logic in Production (page 217). A related question is "Where is the state of the SUT stored?" Sometimes, the state is stored within the actual SUT; in other cases, the state may be stored in another component such as a database. In the latter case, State Verification may involve accessing the state within the other component (essentially a layer-crossing test). By contrast, Behavior Verification (page 468) would involve verifying the interactions between the SUT and the other component. Fixture DOC Setup Exercise Verify Teardown SUT Get State Exercise B A C Behaviour (Indirect Outputs) When to Use It We should use State Verification when we care about only the end state of the SUT—not how the SUT got there. Taking such a limited view helps us maintain encapsulation of the implementation of the SUT. State Verification comes naturally when we are building the software inside out. That is, we build the innermost objects first and then build the next layer of objects on top of them. Of course, we may need to use Test Stubs (page 529) to control the indirect inputs of the SUT to avoid Production Bugs (page 268) caused by untested code paths. Even then, we are choosing not to verify the indirect outputs of the SUT. When we do care about the side effects of exercising the SUT that are not visible in its end state (its indirect outputs), we can use Behavior Verification to observe the behavior directly. We must be careful, however, not to create Fragile Tests (page 239) by overspecifying the software. see next slide State Verification no observation points observation points 

extract from smell Production Bugs Production Bugs We find too many bugs during formal tests or in production. Symptoms We have put a lot of effort into writing automated tests, yet the number of bugs showing up in formal (i.e., system) testing or production remains too high. Impact … Causes … Cause: Infrequently Run Tests … Cause: Lost Test … Cause: Missing Unit Test … Cause: Untested Code Cause: Untested Requirement … Cause: Neverfail Test … Cause: Untested Code Symptoms We may just "know" that some piece of code in the SUT is not being exercised by any tests. Perhaps we have never seen that code execute, or perhaps we used code coverage tools to prove this fact beyond a doubt. In the following example, how can we test that when timeProvider throws an exception, this exception is handled correctly? … Root Cause The most common cause of Untested Code is that the SUT includes code paths that react to particular ways that a depended-on component (DOC) behaves and we haven't found a way to exercise those paths. Typically, the DOC is being called synchronously and either returns certain values or throws exceptions. During normal testing, only a subset of the possible equivalence classes of indirect inputs are actually encountered. Another common cause of Untested Code is incompleteness of the test suite caused by incomplete characterization of the functionality exposed via the SUT's interface. Possible Solution If the Untested Code is caused by an inability to control the indirect inputs of the SUT, the most common solution is to use a Test Stub (page 529) to feed the various kinds of indirect inputs into the SUT to cover all the code paths. Otherwise, it may be sufficient to configure the DOC to cause it to return the various indirect inputs required to fully test the SUT. see previous slide 

Now that we have seen the State Verification pattern, let’s turn to the Behaviour Verification pattern. Again, let’s see some excerpts from xUnit Test Patterns. 

Behaviour Verification Behaviour Verification How do we make tests self-checking when there is no state to verify? We capture the indirect outputs of the SUT as they occur and compare them to the expected behavior. A Self-Checking Test (see page 26) must verify that the expected outcome has occurred without manual intervention by whoever is running the test. But what do we mean by "expected outcome"? The SUT may or may not be "stateful"; if it is stateful, it may or may not be expected to end up in a different state after it has been exercised. The SUT may also be expected to invoke methods on other objects or components. Behavior Verification involves verifying the indirect outputs of the SUT as it is being exercised. Also known as: Interaction Testing Note that because we intend to verify indirect output behaviours of the SUT, we are setting up observation points on the "back side" of the SUT. Setup Exercise Verify Teardown Fixture DOC Setup Exercise Verify Teardown SUT Exercise B A C Behaviour (Indirect Outputs) Verify observation points 

When to Use It Behavior Verification is primarily a technique for unit tests and component tests. We can use Behavior Verification whenever the SUT calls methods on other objects or components. We must use Behavior Verification whenever the expected outputs of the SUT are transient and cannot be determined simply by looking at the post-exercise state of the SUT or the DOC. This forces us to monitor these indirect outputs as they occur. A common application of Behavior Verification is when we are writing our code in an "outside-in" manner. This approach, which is often called need-driven development, involves writing the client code before we write the DOC. It is a good way to find out exactly what the interface provided by the DOC needs to be based on real, concrete examples rather than on speculation. The main objection to this approach is that we need to use a lot of Test Doubles (page 522) to write these tests. That could result in Fragile Tests (page 239) because each test knows so much about how the SUT is implemented. Because the tests specify the behavior of the SUT in terms of its interactions with the DOC, a change in the implementation of the SUT could break a lot of tests. This kind of Overspecified Software (see Fragile Test) could lead to High Test Maintenance Cost (page 265). The jury is still out on whether Behavior Verification is a better approach than State Verification. In most cases, State Verification is clearly necessary; in some cases, Behavior Verification is clearly necessary. What has yet to be determined is whether Behavior Verification should be used in all cases or whether we should use State Verification most of the time and resort to Behavior Verification only when State Verification falls short of full test coverage. How It Works Each test specifies not only how the client of the SUT interacts with it during the exercise SUT phase of the test, but also how the SUT interacts with the components on which it should depend. This ensures that the SUT really is behaving as specified rather than just ending up in the correct post-exercise state. Behavior Verification almost always involves interacting with or replacing a depended-on component (DOC) with which the SUT interacts at runtime. The line between Behavior Verification and State Verification (page 462) can get a bit blurry when the SUT stores its state in the DOC because both forms of verification involve layer-crossing tests. We can distinguish between the two cases based on whether we are verifying the post-test state in the DOC (State Verification) or whether we are verifying the method calls made by the SUT on the DOC (Behavior Verification). Setup Exercise Verify Teardown Setup Exercise Verify Teardown Fixture DOC Setup Exercise Verify Teardown SUT Exercise B A C Behaviour (Indirect Outputs) Verify Behaviour Verification See next slide for an example of when to use Behaviour Verification observation points 

extract from smell Production Bugs Production Bugs We find too many bugs during formal tests or in production. Symptoms We have put a lot of effort into writing automated tests, yet the number of bugs showing up in formal (i.e., system) testing or production remains too high. Impact … Causes … Cause: Infrequently Run Tests … Cause: Lost Test … Cause: Missing Unit Test … Cause: Untested Code Cause: Untested Requirement Cause: Neverfail Test … Cause: Untested Requirement Symptoms We may just "know" that some piece of functionality is not being tested. Alternatively, we may be trying to test a piece of software but cannot see any visible functionality that can be tested via the public interface of the software. All the tests we have written pass, however. When doing test-driven development, we know we need to add some code to handle a requirement. However, we cannot find a way to express the need for code to … … Root Cause The most common cause of Untested Requirements is that the SUT includes behavior that is not visible through its public interface. It may have expected "side effects" that cannot be observed directly by the test (such as writing out a file or record or calling a method on another object or component)—in other words, it may have indirect outputs. When the SUT is an entire application, the Untested Requirement may be a result of not having a full suite of customer tests that verify all aspects of the visible behavior of the SUT. Possible Solution If the problem is missing customer tests, we need to write at least enough customer tests to ensure that all components are integrated properly. This may require improving the design-for-testability of the application by separating the presentation layer from the business logic layer. When we have indirect outputs that we need to verify, we can do Behavior Verification (page 468) through the use of Mock Objects (page 544). Testing of indirect outputs is covered in Chapter 11, Using Test Doubles. 

I find it is useful to compare the diagrams for State Verification and Behaviour Verification, which is what the next slide does. 

Fixture DOC Setup Exercise Verify Teardown SUT Get State Exercise B A C Behaviour (Indirect Outputs) Fixture DOC Setup Exercise Verify Teardown SUT Exercise B A C Behaviour (Indirect Outputs) Verify State Verification observation points no observation points Behaviour Verification observation points 

Let’s finally turn to the test double patterns. Let’s start with Test Double. 

Fixture DOC Setup Exercise Verify Teardown SUT Test Double Test Double How can we verify logic independently when code it depends on is unusable? How can we avoid Slow Tests? We replace a component on which the SUT depends with a "test-specific equivalent." Sometimes it is just plain hard to test the SUT because it depends on other components that cannot be used in the test environment. Such a situation may arise because those components aren't available, because they will not return the results needed for the test, or because executing them would have undesirable side effects. In other cases, our test strategy requires us to have more control over or visibility of the internal behavior of the SUT. When we are writing a test in which we cannot (or choose not to) use a real depended-on component (DOC), we can replace it with a Test Double. The Test Double doesn't have to behave exactly like the real DOC; it merely has to provide the same API as the real DOC so that the SUT thinks it is the real one! Also known as: Imposter indirect output indirect input Test Double 

How It Works When the producers of a movie want to film something that is potentially risky or dangerous for the leading actor to carry out, they hire a "stunt double" to take the place of the actor in the scene. The stunt double is a highly trained individual who is capable of meeting the specific requirements of the scene. The stunt double may not be able to act, but he or she knows how to fall from great heights, crash a car, or do whatever the scene calls for. How closely the stunt double needs to resemble the actor depends on the nature of the scene. Usually, things can be arranged such that someone who vaguely resembles the actor in stature can take the actor's place. For testing purposes, we can replace the real DOC (not the SUT!) with our equivalent of the "stunt double": the Test Double. During the fixture setup phase of our Four-Phase Test (page 358), we replace the real DOC with our Test Double. Depending on the kind of test we are executing, we may hard-code the behavior of the Test Double or we may configure it during the setup phase. When the SUT interacts with the Test Double, it won't be aware that it isn't talking to the real McCoy, but we will have achieved our goal of making impossible tests possible. Regardless of which variation of Test Double we choose to use, we must keep in mind that we don't need to implement the entire interface of the DOC. Instead, we provide only the functionality needed for our particular test. We can even build different Test Doubles for different tests that involve the same DOC. Fixture DOC Setup Exercise Verify Teardown SUT Test Double Test Double 

As an aside, the icon chosen for Test Double is not great, but that’s not going to be a big issue because we won’t be using the icon as much as the ones for the various Test Double variations. For what it is worth, here is how I arrived at it. Stunt Double Test Double “For testing purposes, we can replace the real DOC (not the SUT!) with our equivalent of the "stunt double": the Test Double.“ 

When to Use it We might want to use some sort of Test Double during our tests in the following circumstances: • If we have an Untested Requirement (see Production Bugs on page 268) because neither the SUT nor its DOCs provide an observation point for the SUT's indirect output that we need to verify using Behavior Verification (page 468) • If we have Untested Code (see Production Bugs) and a DOC does not provide the control point to allow us to exercise the SUT with the necessary indirect inputs • If we have Slow Tests (page 253) and we want to be able to run our tests more quickly and hence more often Each of these scenarios can be addressed in some way by using a Test Double. Of course, we have to be careful when using Test Doubles because we are testing the SUT in a different configuration from the one that will be used in production. For this reason, we really should have at least one test that verifies the SUT works without a Test Double. We need to be careful that we don't replace the parts of the SUT that we are trying to verify because that practice can result in tests that test the wrong software! Also, excessive use of Test Doubles can result in Fragile Tests (page 239) as a result of Overspecified Software. Test Doubles come in several major flavors, as summarized in Figure 23.1. The implementation variations of these patterns are described in more detail in the corresponding pattern write-ups. Figure 23.1. Types of Test Doubles. Dummy Objects are really an alternative to the value patterns. Test Stubs are used to verify indirect inputs; Test Spies and Mock Objects are used to verify indirect outputs. Fake objects provide an alternative implementation. Test Double These variations are classified based on how/why we use the Test Double. We will deal with variations around how we build the Test Doubles in the "Implementation" section. 

Our coverage of the Test Double pattern continues on the next slide with an introductory look at Test Double usage pattern types (variations). 

We use a Test Stub (page 529) to replace a real component on which the SUT depends so that the test has a control point for the indirect inputs of the SUT. Its inclusion allows the test to force the SUT down paths it might not otherwise execute. We can further classify Test Stubs by the kind of indirect inputs they are used to inject into the SUT. A Responder (see Test Stub) injects valid values, while a Saboteur (see Test Stub) injects errors or exceptions. Some people use the term "test stub" to mean a temporary implementation that is used only until the real object or procedure becomes available. I prefer to call this usage a Temporary Test Stub (see Test Stub) to avoid confusion. We can use a more capable version of a Test Stub, the Test Spy (page 538), as an observation point for the indirect outputs of the SUT. Like a Test Stub, a Test Spy may need to provide values to the SUT in response to method calls. The Test Spy, however, also captures the indirect outputs of the SUT as it is exercised and saves them for later verification by the test. Thus, in many ways, the Test Spy is "just a" Test Stub with some recording capability. While a Test Spy is used for the same fundamental purpose as a Mock Object (page 544), the style of test we write using a Test Spy looks much more like a test written with a Test Stub. We can use a Mock Object as an observation point to verify the indirect outputs of the SUT as it is exercised. Typically, the Mock Object also includes the functionality of a Test Stub in that it must return values to the SUT if it hasn't already failed the tests but the emphasis1 is on the verification of the indirect outputs. Therefore, a Mock Object is a lot more than just a Test Stub plus assertions: It is used in a fundamentally different way. We use a Fake Object (page 551) to replace the functionality of a real DOC in a test for reasons other than verification of indirect inputs and outputs of the SUT. Typically, a Fake Object implements the same functionality as the real DOC but in a much simpler way. While a Fake Object is typically built specifically for testing, the test does not use it as either a control point or an observation point. The most common reason for using a Fake Object is that the real DOC is not available yet, is too slow, or cannot be used in the test environment because of deleterious side effects. The sidebar "Faster Tests Without Shared Fixtures" (page 319) describes how we encapsulated all database access behind a persistence layer interface and then replaced the database with in-memory hash tables and made our tests run 50 times faster. Chapter 6, Test Automation Strategy, and Chapter 11, Using Test Doubles, provide an overview of the various techniques available for making our SUT easier to test. Some method signatures of the SUT may require objects as parameters. If neither the test nor the SUT cares about these objects, we may choose to pass in a Dummy Object (page 728), which may be as simple as a null object reference, an instance of the Object class, or an instance of a Pseudo-Object (see Hard-Coded Test Double on page 568). In this sense, a Dummy Object isn't really a Test Double per se but rather an alternative to the value patterns Literal Value (page 714), Derived Value (page 718), and Generated Value (page 723). Test Stub Test Spy Mock Object Fake Object Dummmy Object Responder Saboteur Test Double Usage Pattern Variations 

After that introductory look at Test Double usage pattern types (variations), it is time to see how Gerard Meszaros describes the benefits of standardising terminology. 

Why Standardize Testing Patterns? …I think it is important for us to standardize the names of the test automation patterns, especially those related to Test Stubs (page 529) and Mock Objects (page 544). The key issue here relates to succinctness of communication. When someone tells you, "Put a mock in it" (pun intended!), what advice is that person giving you? Depending on what the person means by a "mock," he or she could be suggesting that you control the indirect inputs of your SUT using a Test Stub or that you replace your database with a Fake Database (see Fake Object on page 551) that will reduce test interactions and speed up your tests by a factor of 50. … Or perhaps the person is suggesting that you verify that your SUT calls the correct methods by installing an Eager Mock Object (see Mock Object) preconfigured with the Expected Behavior (see Behavior Verification on page 468). If everyone used "mock" to mean a Mock Object—no more or less—then the advice would be pretty clear. As I write this, the advice is very murky because we have taken to calling just about any Test Double (page 522) a "mock object" (despite the objections of the authors of the original paper on Mock Objects [ET]). 

The next slide introduces two Test Double implementation (construction) pattern variations. 

Configurable Hard-Coded 

Next, here is Pseudo-Object, a variation of the Hard-Coded Test Double pattern. Variation: Pseudo-Object One challenge facing writers of Hard-Coded Test Doubles is that we must implement all the methods in the interface that the SUT might call. In statically typed languages such as Java and C#, we must at least implement all methods declared in the interface implied by the class or type associated with however we access the DOC. This often "forces" us to subclass from the real DOC to avoid providing dummy implementations for these methods. One way of reducing the programming effort is to provide a default class that implements all the interface methods and throws a unique error. We can then implement a Hard-Coded Test Double by subclassing this concrete class and overriding just the one method we expect the SUT to call while we are exercising it. If the SUT calls any other methods, the Pseudo-Object throws an error, thereby failing the test. PSEUDO-OBJECT 

We are now going to go through each Test Double usage pattern type (variation) in turn, beginning with Test Stub. As we work our way through the different types, we shall also look at code examples of their usage. 

In many circumstances, the environment or context in which the SUT operates very much influences the behavior of the SUT. To get adequate control over the indirect inputs of the SUT, we may have to replace some of the context with something we can control —namely, a Test Stub. How It Works First, we define a test-specific implementation of an interface on which the SUT depends. This implementation is configured to respond to calls from the SUT with the values (or exceptions) that will exercise the Untested Code (see Production Bugs on page 268) within the SUT. Before exercising the SUT, we install the Test Stub so that the SUT uses it instead of the real implementation. When called by the SUT during test execution, the Test Stub returns the previously defined values. The test can then verify the expected outcome in the normal way. Fixture DOC Setup Exercise Verify Teardown SUT Test Stub Test Stub How can we verify logic independently when it depends on indirect inputs from other software components? We replace a real object with a test-specific object that feeds the desired indirect inputs into the SUT. Also known as: Stub indirect input indirect output indirect input Test Stub Return Values Installation Creation front door back door observation point control point 

When to Use it A key indication for using a Test Stub is having Untested Code caused by our inability to control the indirect inputs of the SUT. We can use a Test Stub as a control point that allows us to control the behavior of the SUT with various indirect inputs and we have no need to verify the indirect outputs. We can also use a Test Stub to inject values that allow us to get past a particular point in the software where the SUT calls software that is unavailable in our test environment. If we do need an observation point that allows us to verify the indirect outputs of the SUT, we should consider using a Mock Object (page 544) or a Test Spy (page 538). Of course, we must have a way of installing a Test Double (page 522) into the SUT to be able to use any form of Test Double. Test Stub A Test Stub that is used to inject invalid indirect inputs into the SUT is often called a Saboteur because its purpose is to derail whatever the SUT is trying to do so that we can see how the SUT copes under these circumstances. The "derailment" might be caused by returning unexpected values or objects, or it might result from raising an exception or causing a runtime error. Each test may be either a Simple Success Test or an Expected Exception Test (see Test Method), depending on how the SUT is expected to behave in response to the indirect input. Fixture DOC Setup Exercise Verify Teardown SUT Test Stub indirect input Return Values Installation Creation front door back door observation point control point A Test Stub that is used to inject valid indirect inputs into the SUT so that it can go about its business is called a Responder. Responders are commonly used in "happy path" testing when the real component is uncontrollable, is not yet available, or is unusable in the development environment. The tests will invariably be Simple Success Tests (see Test Method on page 348). Variation: Responder Variation: Saboteur 

Now that we have seen the Test Stub pattern, we are going to look at examples of its usage. Among the Configurable Test Double variations in xUnit Test Patterns there are the Hand-Built Test Double, and the Dynamically Generated Test Double. The first one can be seen below. The second one will be covered in part two of this deck. The Test Stub examples that we are about to see are all either hand-built or hard-coded. We will look at dynamically generated Test Stub examples in part 2. Variation: Hand-Built Test Double A Hand-Built Test Double is one that was defined by the test automater for one or more specific tests. A Hard-Coded Test Double is inherently a Hand-Built Test Double, while a Configurable Test Double can be either hand-built or generated. This book uses Hand-Built Test Doubles in a lot of the examples because it is easier to see what is going on when we have actual, simple, concrete code to look at. This is the main advantage of using a Hand-Built Test Double; indeed, some people consider this benefit to be so important that they use Hand-Built Test Doubles exclusively. We may also use a Hand-Built Test Double when no third-party toolkits are available or if we are prevented from using those tools by project or corporate policy. Hand Built 

Configurable Hard-Coded Test Stub Responder Saboteur Test Stub Responder Configurable Configurable Typelevel Rite of Passage Test Stub Responder To conclude part 1, we are going to pick, from each of the following books, one example of using the Test Stub pattern. Hard-Coded Test Stub Responder Hard-Coded Test Stub Responder PSEUDO-OBJECT ! 1 2 ! 1 2 

The first example of using the Test Stub pattern is from the book Scala Programming Projects. It is somewhat similar to the pattern’s motivating example in xUnit Test Patterns (see next slide). That’s because in both examples, the system clock is a DOC that gets replaced by a Test Stub. Hard-Coded Test Stub Responder PSEUDO-OBJECT ! 1 2 ! 1 2 https://github.com/PacktPublishing/Scala-Programming-Projects/blob/master/Chapter10-11/bitcoin-analyser 

Test Stub … Motivating Example The following test verifies the basic functionality of a component that formats an HTML string containing the current time. Unfortunately, it depends on the real system clock so it rarely ever passes! … Refactoring Notes We can achieve proper verification of the indirect inputs by getting control of the time. To do so, we use the Replace Dependency with Test Double (page 522) refactoring to replace the real system clock (represented here by TimeProvider) with a Virtual Clock [VCTP]. We then implement it as a Test Stub that is configured by the test with the time we want to use as the indirect input to the SUT. [VCTP] The Virtual Clock Test Pattern http://www.nusco.org/docs/virtual_clock.pdf By: Paolo Perrotta This paper describes a common example of a Responder called Virtual Clock [VCTP]. The author uses the Virtual Clock Test Pattern as a Decorator [GOF] for the real system clock, which allows the time to be "frozen" or resumed. One could use a Hard-Coded Test Stub or a Configurable Test Stub just as easily for most tests. Paolo Perrotta summarizes the thrust of his article: We can have a hard time unit-testing code that depends on the system clock. This paper describes both the problem and a common, reusable solution. 

Example #1 DOC SparkSession Timer[IO] BatchProducer SUT Timer[F] DOC DOC SparkSession FakeTimer BatchProducer SUT Timer[F] BatchProducerIT Test Stub Production Integration Test Responder PSEUDO-OBJECT ! 1 2 ! 1 2 

class AppContext(val transactionStorePath: URI) (implicit val spark: SparkSession, implicit val timer: Timer[IO]) object BatchProducer { val WaitTime: FiniteDuration = 59.minute /** Number of seconds required by the API to make a transaction visible */ val ApiLag: FiniteDuration = 5.seconds … def processOneBatch(fetchNextTransactions: IO[Dataset[Transaction]], transactions: Dataset[Transaction], saveStart: Instant, saveEnd: Instant)(implicit appCtx: AppContext) : IO[(Dataset[Transaction], Instant, Instant)] = { import appCtx._ val transactionsToSave = filterTxs(transactions, saveStart, saveEnd) for { _ <- BatchProducer.save(transactionsToSave, appCtx.transactionStorePath) _ <- IO.sleep(WaitTime) beforeRead <- currentInstant // We are sure that lastTransactions contain all transactions until end end = beforeRead.minusSeconds(ApiLag.toSeconds) nextTransactions <- fetchNextTransactions } yield (nextTransactions, saveEnd, end) } … def currentInstant(implicit timer: Timer[IO]): IO[Instant] = timer.clockRealTime(TimeUnit.SECONDS) map Instant.ofEpochSecond DOC SparkSession FakeTimer BatchProducer SUT Timer[F] BatchProducerIT Test Stub Responder PSEUDO-OBJECT ! 1 2 ! 1 2 def sleep(duration: FiniteDuration)(implicit timer: Timer[IO]): IO[Unit] = timer.sleep(duration) cats.effect.IO Example #1 Test Stub • FakeTimer Control Point • clockRealTime function of FakeTimer Indirect Input • time returned by clockRealTime function 

implicit object FakeTimer extends Timer[IO] { private var clockRealTimeInMillis: Long = Instant.parse("2018-08-02T01:00:00Z").toEpochMilli def clockRealTime(unit: TimeUnit): IO[Long] = IO(unit.convert(clockRealTimeInMillis, TimeUnit.MILLISECONDS)) def sleep(duration: FiniteDuration): IO[Unit] = IO { clockRealTimeInMillis = clockRealTimeInMillis + duration.toMillis } def shift: IO[Unit] = ??? def clockMonotonic(unit: TimeUnit): IO[Long] = ??? } DOC SparkSession FakeTimer BatchProducer SUT Timer[F] BatchProducerIT Test Stub Responder PSEUDO-OBJECT Hard-Coded ! 1 2 ! 1 2 Example #1 In the usual way of implementing the Pseudo-Object pattern, we would have (1) a base class providing undefined implementations (??? expressions) of all Timer’s functions, and (2) a concrete subclass that overrides only the functions we expect the SUT to call. Instead, here we have a concrete Timer that provides (1) real implementations of the functions we expect the SUT to call, and (2) undefined implementations of functions that the SUT is NOT expected to call. By the way, note that while the term `fake` is included in the name of the timer implementation, it is not being used in a way that is consistent with the terminology described in this deck series. Test Stub • FakeTimer Control Point • clockRealTime function of FakeTimer Indirect Input • time returned by clockRealTime function 

class BatchProducerIT extends WordSpec with Matchers with SharedSparkSession { … "BatchProducer.processOneBatch" should { "filter and save a batch of transaction, wait 59 mn, fetch the next batch" in withTempDir { tmpDir => implicit object FakeTimer extends Timer[IO] { … } implicit val appContext: AppContext = new AppContext(transactionStorePath = tmpDir.toURI) implicit def toTimestamp(str: String): Timestamp = Timestamp.from(Instant.parse(str)) val tx1 = Transaction("2018-08-01T23:00:00Z", 1, 7657.58, true, 0.021762) val tx2 = Transaction("2018-08-02T01:00:00Z", 2, 7663.85, false, 0.01385517) val tx3 = Transaction("2018-08-02T01:58:30Z", 3, 7663.85, false, 0.03782426) val tx4 = Transaction("2018-08-02T01:58:59Z", 4, 7663.86, false, 0.15750809) val tx5 = Transaction("2018-08-02T02:30:00Z", 5, 7661.49, true, 0.1) // Start at 01:00, tx 2 ignored (too soon) val txs0 = Seq(tx1) // Fetch at 01:59, get nb 2 and 3, but will miss nb 4 because of Api lag val txs1 = Seq(tx2, tx3) // Fetch at 02:58, get nb 3, 4, 5 val txs2 = Seq(tx3, tx4, tx5) // Fetch at 03:57, get nothing val txs3 = Seq.empty[Transaction] val start0 = Instant.parse("2018-08-02T00:00:00Z") val end0 = Instant.parse("2018-08-02T00:59:55Z") val threeBatchesIO = … val (ds1, start1, end1, ds2, start2, end2) = threeBatchesIO.unsafeRunSync() ds1.collect() should contain theSameElementsAs txs1 start1 should ===(end0) // initialClock + 1mn - 15s - 5s end1 should ===(Instant.parse("2018-08-02T01:58:55Z")) ds2.collect() should contain theSameElementsAs txs2 start2 should ===(end1) // initialClock + 1mn -15s + 1mn -15s -5s = end1 + 45s end2 should ===(Instant.parse("2018-08-02T02:57:55Z")) val lastClock = Instant.ofEpochMilli(FakeTimer.clockRealTime(TimeUnit.MILLISECONDS).unsafeRunSync()) lastClock should === (Instant.parse("2018-08-02T03:57:00Z")) … DOC SparkSession FakeTimer BatchProducer SUT Timer[F] BatchProducerIT Test Stub val threeBatchesIO = for { tuple1 <- BatchProducer.processOneBatch(IO(txs1.toDS()), txs0.toDS(), start0, end0) // end - Api lag (ds1, start1, end1) = tuple1 tuple2 <- BatchProducer.processOneBatch(IO(txs2.toDS()), ds1, start1, end1) (ds2, start2, end2) = tuple2 _ <- BatchProducer.processOneBatch(IO(txs3.toDS()), ds2, start2, end2) } yield (ds1, start1, end1, ds2, start2, end2) Example #1 Test Stub • FakeTimer Control Point • clockRealTime function of FakeTimer Indirect Input • time returned by clockRealTime function 

The second example is from the book Pure functional HTTP APIs in Scala. Configurable Test Stub Responder Hand Built https://github.com/jan0sch/pfhais/blob/main/pure/src/test/scala/com/wegtam/books/pfhais/pure/api/ Example #2 

TestRepository ProductRoutes ProductRoutesTest SUT DoobieRepository ProductRoutes SUT DOC Production Unit Test Repository Repository Test Stub Responder Example #2 DOC 

final class ProductRoutes[F[_]: Sync](repo: Repository[F]) extends Http4sDsl[F] { implicit def decodeProduct: EntityDecoder[F, Product] = jsonOf implicit def encodeProduct[A[_]: Applicative]: EntityEncoder[A, Product] = jsonEncoderOf val routes: HttpRoutes[F] = HttpRoutes.of[F] { case GET -> Root / "product" / UUIDVar(id) => for { rows <- repo.loadProduct(id) resp <- Product.fromDatabase(rows).fold(NotFound())(p => Ok(p)) } yield resp case req @ PUT -> Root / "product" / UUIDVar(id) => … } } class TestRepository[F[_]: Effect](data: Seq[Product]) extends Repository[F] { override def loadProduct(id: ProductId): F[Seq[(ProductId, LanguageCode, ProductName)]] = data.find(_.id === id) match { case None => Seq.empty.pure[F] case Some(p) => val ns = p.names.toNonEmptyList.toList.to[Seq] ns.map(n => (p.id, n.lang, n.name)).pure[F] } … } trait Repository[F[_]] { /** * Load a product from the database repository. * * @param id The unique ID of the product. * @return A list of database rows for a single product which you'll need to combine. */ def loadProduct(id: ProductId): F[Seq[(ProductId, LanguageCode, ProductName)]] … } TestRepository ProductRoutes Repository ProductRoutesTest SUT Test Stub • TestRepository Control Point • loadProduct function of TestRepository Indirect Input • sequence of tuples returned by loadProduct function Test Stub Responder Configurable 

final class ProductRoutesTest extends BaseSpec { implicit def decodeProduct: EntityDecoder[IO, Product] = jsonOf implicit def encodeProduct[A[_]: Applicative]: EntityEncoder[A, Product] = jsonEncoderOf private val emptyRepository: Repository[IO] = new TestRepository[IO](Seq.empty) "ProductRoutes" when { "GET /product/ID" when { "product does not exist" must { val expectedStatusCode = Status.NotFound s"return $expectedStatusCode" in { forAll("id") { id: ProductId => Uri.fromString("/product/" + id.toString) match { case Left(_) => fail("Could not generate valid URI!") case Right(u) => def service: HttpRoutes[IO] = Router("/" -> new ProductRoutes(emptyRepository).routes) val response: IO[Response[IO]] = service.orNotFound.run( Request(method = Method.GET, uri = u) ) val result = response.unsafeRunSync result.status must be(expectedStatusCode) result.body.compile.toVector.unsafeRunSync must be(empty) }}}} "product exists" must { val expectedStatusCode = Status.Ok s"return $expectedStatusCode and the product" in { forAll("product") { p: Product => Uri.fromString("/product/" + p.id.toString) match { case Left(_) => fail("Could not generate valid URI!") case Right(u) => val repo: Repository[IO] = new TestRepository[IO](Seq(p)) def service: HttpRoutes[IO] = Router("/" -> new ProductRoutes(repo).routes) val response: IO[Response[IO]] = service.orNotFound.run( Request(method = Method.GET, uri = u) ) val result = response.unsafeRunSync result.status must be(expectedStatusCode) result.as[Product].unsafeRunSync must be(p)}}}} } TestRepository ProductRoutes Repository ProductRoutesTest SUT Test Stub Responder Test Stub • TestRepository Control Point • loadProduct function of TestRepository Indirect Input • sequence of tuples returned by loadProduct function 

The third example is from the book Practical FP in Scala. Saboteur Test Stub Responder Configurable Configurable Hand Built https://github.com/gvolpe/pfps-shopping-cart/tree/second-edition/modules/tests/src/test/scala/shop 

Items ItemRoutes SUT DOC Production Unit Test Items TestItems ItemRoutes Items ItemRoutesSuite SUT dataItems failingItems Test Stub Responder Saboteur 

def dataItems(items: List[Item]) = new TestItems { override def findAll: IO[List[Item]] = IO.pure(items) override def findBy(brand: BrandName): IO[List[Item]] = IO.pure(items.find(_.brand.name === brand).toList) } def failingItems(items: List[Item]) = new TestItems { override def findAll: IO[List[Item]] = IO.raiseError(DummyError) *> IO.pure(items) override def findBy(brand: BrandName): IO[List[Item]] = findAll } protected class TestItems extends Items[IO] { def findAll: IO[List[Item]] = IO.pure(List.empty) def findBy(brand: BrandName): IO[List[Item]] = IO.pure(List.empty) def findById(itemId: ItemId): IO[Option[Item]] = IO.pure(none[Item]) def create(item: CreateItem): IO[ItemId] = ID.make[IO, ItemId] def update(item: UpdateItem): IO[Unit] = IO.unit } final case class ItemRoutes[F[_]: Monad]( items: Items[F] ) extends Http4sDsl[F] { private[routes] val prefixPath = "/items" object BrandQueryParam extends OptionalQueryParamDecoderMatcher[BrandParam]("brand") private val httpRoutes: HttpRoutes[F] = HttpRoutes.of[F] { case GET -> Root :? BrandQueryParam(brand) => Ok(brand.fold(items.findAll)(b => items.findBy(b.toDomain))) } val routes: HttpRoutes[F] = Router(prefixPath -> httpRoutes) } trait Items[F[_]] { def findAll: F[List[Item]] def findBy(brand: BrandName): F[List[Item]] def findById(itemId: ItemId): F[Option[Item]] def create(item: CreateItem): F[ItemId] def update(item: UpdateItem): F[Unit] } TestItems ItemRoutes Items ItemRoutesSuite SUT dataItems failingItems Test Stub Responder Saboteur Test Stubs • dataItems, failingItems Control Point • findAll function of dataItems and failingItems Indirect Input • list of items Configurable Configurable 

object ItemRoutesSuite extends HttpSuite { … test("GET items succeeds") { forall(Gen.listOf(itemGen)) { it => val req = GET(uri"/items") val routes = ItemRoutes[IO](dataItems(it)).routes expectHttpBodyAndStatus(routes, req)(it, Status.Ok) } } … test("GET items fails") { forall(Gen.listOf(itemGen)) { it => val req = GET(uri"/items") val routes = ItemRoutes[IO](failingItems(it)).routes expectHttpFailure(routes, req) } … } Test Stubs • dataItems, failingItems Control Point • findAll function of dataItems and failingItems Indirect Input • list of items TestItems ItemRoutes Items ItemRoutesSuite SUT dataItems failingItems Test Stub Responder Saboteur 

The fourth and final example is from the course Typelevel Rite of Passage. Typelevel Rite of Passage Hard-Coded Test Stub Responder Hard-Coded Test Stub Responder https://rockthejvm.com/courses/typelevel-rite-of-passage https://github.com/rockthejvm/typelevel-rite-of-passage 

AuthRoutes AuthRoutesSpec mockedAuthenticator mockedAuth AuthRoutes SUT Auth DOC Users Tokens Emails JWTAuthenticator LiveAuth AuthRoutes SUT Auth Backing Store Identity Store DOC DOC DOC Test Stub Responder Identity Store idStore Test Stub Responder Production Unit Test Typelevel Rite of Passage 

Note that while the term `mocked` is included in the names of the highlighted items on the right, it is not being used in a way that is consistent with the terminology described in this deck series. AuthRoutes AuthRoutesSpec mockedAuthenticator mockedAuth SUT Auth DOC DOC Test Stub Responder Identity Store idStore Test Stub Responder Unit Test Typelevel Rite of Passage 

type Authenticator[F[_]] = JWTAuthenticator[F, String, User, Crypto] class AuthRoutes[F[_]: Concurrent: Logger: SecuredHandler] private ( auth: Auth[F], authenticator: Authenticator[F] ) extends HttpValidationDsl[F] { // POST /auth/login { LoginInfo } => 200 Ok with Authorization Bearer {jwt} private val loginRoute: HttpRoutes[F] = HttpRoutes.of[F] { case req@POST -> Root / "login" => req.validate[LoginInfo] { loginInfo => val maybeJwtToken = for maybeUser <- auth.login(loginInfo.email, loginInfo.password) _ <- Logger[F].info(s"User logging in: ${loginInfo.email} ") maybeToken <- maybeUser.traverse(user => authenticator.create(user.email)) yield maybeToken maybeJwtToken.map { case Some(token) => authenticator.embed(Response(Status.Ok), token) case None => Response(Status.Unauthorized) } } } … } AuthRoutes AuthRoutesSpec mockedAuthenticator mockedAuth AuthRoutes SUT Auth DOC DOC Test Stub Responder idStore Test Stub Responder Typelevel Rite of Passage 

val mockedAuth: Auth[IO] = probedAuth(None) def probedAuth(maybeRefEmailToTokenMap: Option[Ref[IO, Map[String, String]]]): Auth[IO] = new Auth[IO]: override def login(email: String, password: String): IO[Option[User]] = if email == danielEmail && password == danielPassword then IO(Some(Daniel)) else IO.pure(None) override def signup(newPasswordInfo: NewUserInfo): IO[Option[User]] = … override def changePassword(email: String, newPasswordInfo: NewPasswordInfo): IO[Either[String, Option[User]]] = … override def delete(email: String): IO[Boolean] = IO.pure(true) override def sendPasswordRecoveryToken(email: String): IO[Unit] = … override def recoverPasswordFromToken(email: String, token: String, newPassword: String): IO[Boolean] = … type Authenticator[F[_]] = JWTAuthenticator[F, String, User, Crypto] trait SecuredRouteFixture extends UserFixture: val mockedAuthenticator: Authenticator[IO] = { val key = HMACSHA256.unsafeGenerateKey val idStore: IdentityStore[IO, String, User] = (email: String) => if email == danielEmail then OptionT.pure(Daniel) else if email == riccardoEmail then OptionT.pure(Riccardo) else OptionT.none[IO, User] JWTAuthenticator.unbacked.inBearerToken( expiryDuration = 1.day, maxIdle = None, identityStore = idStore, signingKey = key ) } Test Stub Responder Test Stub Responder Hard-Coded Typelevel Rite of Passage Hard-Coded 

class AuthRoutesSpec extends AsyncFreeSpec with AsyncIOSpec with Matchers with Http4sDsl[IO] with SecuredRouteFixture { … val authRoutes: HttpRoutes[IO] = AuthRoutes[IO](mockedAuth, mockedAuthenticator).routes given Logger[IO] = Slf4jLogger.getLogger[IO] "AuthRoutes" - { "should return a 401 Unauthorised if login fails" in { for response <- authRoutes.orNotFound.run( Request(method = Method.POST, uri"/auth/login") .withEntity( LoginInfo(email = danielEmail, password = "wrongpassword"))) yield response.status shouldBe Status.Unauthorized } "should return a 200 Ok + a JWT if login is successful" in { for response <- authRoutes.orNotFound.run( Request(method = Method.POST, uri"/auth/login") .withEntity( LoginInfo(email = danielEmail, password = danielPassword))) yield { response.status shouldBe Status.Ok response.headers.get(ci"Authorization") shouldBe defined } } … } } AuthRoutes mockedAuthenticator mockedAuth AuthRoutes SUT Auth Test Stub Responder idStore Test Stub Responder Typelevel Rite of Passage AuthRoutesSpec 

That’s all for part 1. I hope you found it useful. See you in part 2.