Upgrade to Pro — share decks privately, control downloads, hide ads and more …

True End-to-End Testing in Scala

Orr Sella
September 22, 2014

True End-to-End Testing in Scala

Video: https://parleys.com/play/5460c793e4b0dab81f955667

Exploring what end-to-end tests are, why they are important for Scala apps, and how to write them. A talk given at Scalapeno 2014. Talk is accompanied by a code example: https://github.com/orrsella/scala-e2e-testing

Abstract:

It seems that by 2014 we have all come to the realization that testing is good, and today no reputable library/framework will be released without comprehensive tests. Whether you're developing using TDD or not, you're probably writing tests in some capacity. We have a plethora of testing libraries and tools that make writing tests in Scala extremely easy.

In this talk we will show how we can take our unit/integration tests a step further, and see how to test our applications end-to-end. Scala is a prime candidate for end-to-end testing, having all the right pieces in place: an easily extendable build tool (SBT), fluent DSL testing libraries (Specs2/ScalaTest), and the power/tooling of the JVM behind it.

We will see how to approach testing our application from the outside, and touch on concepts such as: the Test Harness, abstracting our SUT (System Under Test) by using test Drivers, using Simplicators for testing against external dependencies, and more. We will put everything together by using tools such as: SBT, Ansible and Vagrant.

Orr Sella

September 22, 2014
Tweet

Other Decks in Programming

Transcript

  1. AGENDA WHAT ARE THEY WHY WE NEED THEM HOW TO

    WRITE THEM END-TO-END TESTS TEST HARNESS DRIVERS FAKES …
  2. AGENDA WHAT ARE THEY WHY WE NEED THEM HOW TO

    WRITE THEM END-TO-END TESTS TEST HARNESS DRIVERS FAKES … CONCRETE EXAMPLE github.com/orrsella/scala-e2e-testing
  3. Why SCALA & END-TO- END TESTS 1 THE APPS Typical

    Scala applications are ripe for end-to- end tests ! THE LANGUAGE & ECOSYSTEM Build tools, testing libraries and language features are a great fit 2
  4. LEVELS OF THE TESTINGKNOWN TO MAN 1 UNIT Do our

    objects do the right thing? ! INTEGRATION Does our code work against code we can’t change? ! END-TO-END Does the whole system work? 2 3 (not really)
  5. INVERSE THE ICE CREAMCONE (or pyramid) END- TO-END INTEGRATION UNIT

    MANUAL TESTING (if you must) NUMBER OF TESTS
  6. END-TO-END TESTS Interact with the system from the outside Black

    Box MOST IMPORTANTLY: Exercising both the system and the process by which it’s built and deployed Include interaction with external environment
  7. AUTOMATIC AN IDEAL BUILDWOULD 1 Compile and unit-test the code

    Integrate and package the system Perform a production-like deployment to a realistic environment Exercise the system through its external access points 2 3 4
  8. AUTOMATIC AN IDEAL BUILDWOULD 1 Compile and unit-test the code

    Integrate and package the system Perform a production-like deployment to a realistic environment Exercise the system through its external access points 2 3 4
  9. FLUSH OUT THE UNKNOWNS 1 Requires asking (and answering) many

    awkward questions Expose uncertainty early, including technical and organizational risks Forces to understand how a system fits into the world Identify all the integration (read: potential failure) points
  10. FLUSH OUT THE UNKNOWNS 1 Requires asking (and answering) many

    awkward questions Expose uncertainty early, including technical and organizational risks Forces to understand how a system fits into the world Identify all the integration (read: potential failure) points EXAMPLE IN 3 SLIDES
  11. FEEDBACK & CONFIDENCE 2 Feedback that would otherwise only show

    in staging/production Safety net of system-wide regression Assurance that our chosen technology stack (“plumbing”) works as expected Confidence to make “risky” system-wide changes (e.g: swap datastore, application server)
  12. FORCED AUTOMATION 3 Everything must be automated so it can

    be tested Crucial for deployment which is error-prone Automatic Build-Deploy-Test cycle means we can ship frequently
  13. LOAD BALANCER HAPROXY REVERSE PROXY NGINX APP SERVER FINATRA DATASTORE

    ELASTICSEARCH TRANSLATION SERVICE YANDEX SCALA!..
  14. LOAD BALANCER HAPROXY REVERSE PROXY NGINX APP SERVER FINATRA DATASTORE

    ELASTICSEARCH TRANSLATION SERVICE YANDEX SCALA!.. PRETTY STRAIGHTFORWARD
  15. WHAT CAN POSSIBLY GO WRONG & BE MISCONFIGURED? Ok. So.

    i.e. integration (failure) points
  16. REVERSE PROXY NGINX APP SERVER FINATRA PROXY PORT 7770 /ETC/NGINX/

    NGINX.CONF INIT.D SCRIPT /ETC/NGINX/ SITES-AVAILABLE JAVA APPLICATION.CONF LOGBACK.XML DEPLOY NEW VERSION
  17. SO A LOT CAN GO WRONG Ok, (read: will) WE

    WANT OUR END-TO-END TESTS TO EXERCISE ALL OF THIS!
  18. HOW DO WE TEST ALL OF THIS END-TO-END? here is

    one approach… THERE ARE OTHERS
  19. HOW DO WE TEST ALL OF THIS END-TO-END? here is

    one approach… THERE ARE OTHERS BUT THIS IS THE BEST, OBVIOUSLY
  20. REQUIRED FOR STEPS END-TO-ENDTESTS 1 BUILD & PACKAGE Package the

    app for production deployment 2 3 0 VIRTUAL MACHINE Fire-up a local VM instance with production-like env CONFIGURATION & DEPLOYMENT Run production config and deployment code, having our entire system on one box RUN TESTS Execute end-to-end tests against the VM
  21. THE TEST HARNESS Enter: EXTENSION OF THE BUILD Runs all

    the necessary setup/teardown actions and the tests themselves ! STANDARD/SIMPLE/SCALA BUILD TOOL Build and package our app (sbt-native-packager) Fire-up the virtual machine (Vagrant) Run configuration and deployment scripts (Ansible) (aka sbt)
  22. VAGRANT Meet PORTABLE DEV ENVIRONMENTS Lightweight, reproducible and programmable !

    VIRTUALIZATION Wrapper around VirtualBox, VMWare, more ! SIMPLE & QUICK Vagrantfile => `$ vagrant up`
  23. 2 CONFIGURATION & DEPLOYMENT Run production config and deployment code,

    having our entire system on one box STEPS REQUIRED FOR THE END-TO-ENDTESTS BUILD & PACKAGE Package the app for production-like deployment 3 0 RUN TESTS Execute end-to-end tests against the VM 1 VIRTUAL MACHINE Fire-up a local VM instance with production-like env
  24. Test Harness: MANAGING VAGRANT SBT // project/Vagrant.scala ! object Vagrant

    { ! private lazy val vagrant = settingKey[Vagrant]("vagrant") ! lazy val settings = Seq( test in EndToEndTest <<= (test in EndToEndTest).dependsOn(publishLocal), testOptions in EndToEndTest += Tests.Setup(() => vagrant.value.setup()), testOptions in EndToEndTest += Tests.Cleanup(() => vagrant.value.cleanup()) ) }
  25. Test Harness: MANAGING VAGRANT SBT // project/Vagrant.scala ! object Vagrant

    { ! private lazy val vagrant = settingKey[Vagrant]("vagrant") ! lazy val settings = Seq( test in EndToEndTest <<= (test in EndToEndTest).dependsOn(publishLocal), testOptions in EndToEndTest += Tests.Setup(() => vagrant.value.setup()), testOptions in EndToEndTest += Tests.Cleanup(() => vagrant.value.cleanup()) ) } TEARDOWN HOOK SETUP HOOK
  26. Test Harness: MANAGING VAGRANT SBT class Vagrant(vagrantFile: File) { !

    // cli method wrappers private def up() = Process("vagrant" :: "up" :: Nil, dir)! private def provision() = Process("vagrant" :: "provision" :: Nil, dir)! ! def setup(): Unit = { prevStatus = status() prevStatus match { case Running => provision() case Saved => up(); provision() case NotCreated => up() case Unknown => up() } } ! def cleanup(): Unit = if (prevStatus != Running) suspend() }
  27. Test Harness: MANAGING VAGRANT SBT class Vagrant(vagrantFile: File) { !

    // cli method wrappers private def up() = Process("vagrant" :: "up" :: Nil, dir)! private def provision() = Process("vagrant" :: "provision" :: Nil, dir)! ! def setup(): Unit = { prevStatus = status() prevStatus match { case Running => provision() case Saved => up(); provision() case NotCreated => up() case Unknown => up() } } ! def cleanup(): Unit = if (prevStatus != Running) suspend() }
  28. Test Harness: MANAGING VAGRANT SBT class Vagrant(vagrantFile: File) { !

    // cli method wrappers private def up() = Process("vagrant" :: "up" :: Nil, dir)! private def provision() = Process("vagrant" :: "provision" :: Nil, dir)! ! def setup(): Unit = { prevStatus = status() prevStatus match { case Running => provision() case Saved => up(); provision() case NotCreated => up() case Unknown => up() } } ! def cleanup(): Unit = if (prevStatus != Running) suspend() } Tip: VAGRANT STATE “JUGGLING”, SAVES A LOT OF TIME!
  29. STEPS REQUIRED FOR THE END-TO-ENDTESTS 1 BUILD & PACKAGE Package

    the app for production-like deployment 3 0 VIRTUAL MACHINE Fire-up a local VM instance with production-like env RUN TESTS Execute end-to-end tests against the VM 2 CONFIGURATION & DEPLOYMENT Run production config and deployment code, having our entire system on one box
  30. DEPLOYMENT THE SCRIPTS CONFIGURATION MANAGEMENT Ensure that our system is

    configured properly (OS, settings, packages, file system, etc.) ! DEPLOY THE APP Upgrade to the latest version just compiled and packaged, as would be done in production ! APPLICATION CONFIG Configure the application itself the same way it would in production (but with test values) (Provision in Vagrant parlance)
  31. Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| ! config.vm.box = "ubuntu/trusty64" config.vm.network "forwarded_port", guest:

    7770, host: 7769 ... ! config.vm.provision "ansible" do |ansible| ansible.playbook = "ansible/site.yml" ansible.inventory_path = "ansible/inventories/vagrant" end ! end Provision: VAGRANTFILE
  32. Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| ! config.vm.box = "ubuntu/trusty64" config.vm.network "forwarded_port", guest:

    7770, host: 7769 ... ! config.vm.provision "ansible" do |ansible| ansible.playbook = "ansible/site.yml" ansible.inventory_path = "ansible/inventories/vagrant" end ! end Provision: VAGRANTFILE
  33. Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| ! config.vm.box = "ubuntu/trusty64" config.vm.network "forwarded_port", guest:

    7770, host: 7769 ... ! config.vm.provision "ansible" do |ansible| ansible.playbook = "ansible/site.yml" ansible.inventory_path = "ansible/inventories/vagrant" end ! end Provision: VAGRANTFILE VAGRANT INVOKES THE PROVISIONER
  34. CONFIGURATION & DEPLOYMENT Run production config and deployment code, having

    our entire system on one box BUILD & PACKAGE Package the app for production-like deployment VIRTUAL MACHINE Fire-up a local VM instance with production-like env STEPS REQUIRED FOR THE END-TO-ENDTESTS 3 RUN TESTS Execute end-to-end tests against the VM 2 1 0
  35. CONFIGURATION & DEPLOYMENT Run production config and deployment code, having

    our entire system on one box BUILD & PACKAGE Package the app for production-like deployment VIRTUAL MACHINE Fire-up a local VM instance with production-like env STEPS REQUIRED FOR THE END-TO-ENDTESTS 3 RUN TESTS Execute end-to-end tests against the VM 2 1 0 CHECK OUT CODE EXAMPLE
  36. 2 CONFIGURATION & DEPLOYMENT Run production config and deployment code,

    having our entire system on one box STEPS REQUIRED FOR THE END-TO-ENDTESTS 1 BUILD & PACKAGE Package the app for production-like deployment 0 VIRTUAL MACHINE Fire-up a local VM instance with production-like env 3 RUN TESTS Execute end-to-end tests against the VM
  37. GUIDELINES SOME E2E TESTS FOR effective TEST SPARINGLY End-to-End tests

    are slow; test few “sunny-day” scenarios ! ABSTRACTION Tests validate features and business logic, not implementation details ! AVOID COUPLING Tests should exercise the system from the outside without reaching for its guts
  38. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } }
  39. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } } USING SPECS2
  40. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } }
  41. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } } TESTING THE SYSTEM BY USING ITSELF
  42. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } } TEST IS DECOUPLED FROM THE CONTROLLER API ITSELF BY USING DRIVERS
  43. Interact with the SUT directly instead of the test Coupled

    to the API/protocol Run same test with different drivers Drivers: ABSTRACTING THE SUT
  44. Drivers: ABSTRACTING THE SUT trait NotesControllerDriver { ! def anAddNoteRequest

    = AddNoteRequest() ! case class AddNoteRequest(text: String = "Lorem ipsum") extends Request[AddNoteResponse] { ! val method = HttpMethod.POST val path = "/notes" val params: Map[String, String] = Map() val headers: Map[String, String] = Map() val body: Option[String] = Some("{ \"text\": \"" + text + "\" }") ! def withText(text: String) = copy(text = text) } ... }
  45. Drivers: ABSTRACTING THE SUT trait NotesControllerDriver { ! def anAddNoteRequest

    = AddNoteRequest() ! case class AddNoteRequest(text: String = "Lorem ipsum") extends Request[AddNoteResponse] { ! val method = HttpMethod.POST val path = "/notes" val params: Map[String, String] = Map() val headers: Map[String, String] = Map() val body: Option[String] = Some("{ \"text\": \"" + text + "\" }") ! def withText(text: String) = copy(text = text) } ... } ONLY THE DRIVER INTIMATELY KNOWS THE CONTROLLER
  46. Drivers: ABSTRACTING THE SUT trait NotesControllerDriver { ! def anAddNoteRequest

    = AddNoteRequest() ! case class AddNoteRequest(text: String = "Lorem ipsum") extends Request[AddNoteResponse] { ! val method = HttpMethod.POST val path = "/notes" val params: Map[String, String] = Map() val headers: Map[String, String] = Map() val body: Option[String] = Some("{ \"text\": \"" + text + "\" }") ! def withText(text: String) = copy(text = text) } ... } ONLY THE DRIVER INTIMATELY KNOWS THE CONTROLLER
  47. Drivers: ABSTRACTING THE SUT trait NotesControllerDriver { ! def anAddNoteRequest

    = AddNoteRequest() ! case class AddNoteRequest(text: String = "Lorem ipsum") extends Request[AddNoteResponse] { ! val method = HttpMethod.POST val path = "/notes" val params: Map[String, String] = Map() val headers: Map[String, String] = Map() val body: Option[String] = Some("{ \"text\": \"" + text + "\" }") ! def withText(text: String) = copy(text = text) } ... } ONLY THE DRIVER INTIMATELY KNOWS THE CONTROLLER
  48. Drivers: ABSTRACTING THE SUT trait NotesControllerDriver { ! def anAddNoteRequest

    = AddNoteRequest() ! case class AddNoteRequest(text: String = "Lorem ipsum") extends Request[AddNoteResponse] { ! val method = HttpMethod.POST val path = "/notes" val params: Map[String, String] = Map() val headers: Map[String, String] = Map() val body: Option[String] = Some("{ \"text\": \"" + text + "\" }") ! def withText(text: String) = copy(text = text) } ... } ONLY THE DRIVER INTIMATELY KNOWS THE CONTROLLER class AddNoteResponse(response: Response) extends BaseResponse(response) with JsonResponse { ! lazy val noteId = (json \ "noteId").extract[String] }
  49. Drivers: ABSTRACTING THE SUT trait NotesControllerDriver { ! def anAddNoteRequest

    = AddNoteRequest() ! case class AddNoteRequest(text: String = "Lorem ipsum") extends Request[AddNoteResponse] { ! val method = HttpMethod.POST val path = "/notes" val params: Map[String, String] = Map() val headers: Map[String, String] = Map() val body: Option[String] = Some("{ \"text\": \"" + text + "\" }") ! def withText(text: String) = copy(text = text) } ... } ONLY THE DRIVER INTIMATELY KNOWS THE CONTROLLER class AddNoteResponse(response: Response) extends BaseResponse(response) with JsonResponse { ! lazy val noteId = (json \ "noteId").extract[String] } ONLY THE DRIVER KNOWS THE RESPONSE IS JSON
  50. Drivers: ABSTRACTING THE SUT trait NotesControllerDriver { ! def anAddNoteRequest

    = AddNoteRequest() ! case class AddNoteRequest(text: String = "Lorem ipsum") extends Request[AddNoteResponse] { ! val method = HttpMethod.POST val path = "/notes" val params: Map[String, String] = Map() val headers: Map[String, String] = Map() val body: Option[String] = Some("{ \"text\": \"" + text + "\" }") ! def withText(text: String) = copy(text = text) } ... } ONLY THE DRIVER INTIMATELY KNOWS THE CONTROLLER class AddNoteResponse(response: Response) extends BaseResponse(response) with JsonResponse { ! lazy val noteId = (json \ "noteId").extract[String] } ONLY THE DRIVER KNOWS THE RESPONSE IS JSON OR THE FIELD NAME
  51. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } }
  52. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } }
  53. Matchers: ABSTRACTING THE SUT The other side of the Driver,

    i.e: the response Help decoupling from the protocol, make the test be about features Make test more readable
  54. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } }
  55. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } } trait ResponseMatchers extends Matchers { private implicit def intToIntMatcher(t: Int): Matcher[Int] = beEqualTo(t) ! def beOk = haveStatus(200) def beBadRequest = haveStatus(400) def beNotFound = haveStatus(404) ! def haveStatus(status: Matcher[Int]): Matcher[Response] = ((_: Response).status) ^^ status }
  56. Memento: NOTES CONTROLLERTEST class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with

    ResponseMatchers { ! "Notes controller" should { "add a note and then successfully get it" in { ! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty ! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } } }
  57. FAKES FOR Avoid making external network calls in tests Create

    Fakes based on available documentation Fakes should have minimal implementation to only support testing Test the Fake against the real service in a contract test that’s manually run USE EXTERNAL SERVICES NOT EASILY AVAILABLE(Sometimes called Simplicators)
  58. class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with ResponseMatchers { !

    private val port = 9921 private val server = new FakeYandexTranslateServer(port) ! step { server.start() } ! "Notes controller" should { "add a note and then get it translated" in { ... val translated = aTranslateRequest.withId(noteId).withLang("es").execute() translated must beOk translated.text must_== "Buenos días" } } ! step { server.stop() } }
  59. class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with ResponseMatchers { !

    private val port = 9921 private val server = new FakeYandexTranslateServer(port) ! step { server.start() } ! "Notes controller" should { "add a note and then get it translated" in { ... val translated = aTranslateRequest.withId(noteId).withLang("es").execute() translated must beOk translated.text must_== "Buenos días" } } ! step { server.stop() } } class FakeYandexTranslateServer(port: Int) extends SimpleHttpServer(port) { ! private val json = """{ |"code": 200, |"lang": "en-es", |"text": ["Buenos días"] |} """.stripMargin ! override protected def onSimpleRequest(request: HttpRequest): String = json }
  60. CONTINUE FROM HERE How to CHECK OUT THE EXAMPLE github.com/orrsella/scala-e2e-testing

    ! END-TO-END TEST NEW PROJECTS It is easier to get started with e2e tests on a new project, integrating into an existing one is harder ! READ Growing Object-Oriented Software Guided by Tests
  61. CONTINUE FROM HERE How to CHECK OUT THE EXAMPLE github.com/orrsella/scala-e2e-testing

    ! END-TO-END TEST NEW PROJECTS It is easier to get started with e2e tests on a new project, integrating into an existing one is harder ! READ Growing Object-Oriented Software Guided by Tests