Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microservices and Testing, talking from the experience

129b5ff44f5d43b6e7cf6703211abe7e?s=47 Tuenti
November 30, 2015

Microservices and Testing, talking from the experience

At Tuenti we have been using microservices for a while and we have enjoyed the advantages but also suffered the disadvantages. In this talk we want talk about our experiencia with microservices and testing.

129b5ff44f5d43b6e7cf6703211abe7e?s=128

Tuenti

November 30, 2015
Tweet

Transcript

  1. Jairo González Pérez Tuenti MADRID · NOV 27-28 · 2015

    Microservices and testing Talking from the experience
  2. - How can I test the interactions with all my

    microservices? - Very carefully and with lots of love. Next question?
  3. Part 1: The test who knew too much...

  4. The test who knew too much... Storage FriendsApi UsersApi MomentsApi

    MVC Memcache Feature 1 Feature 2 Feature N Clients Tuenti social network architecture
  5. MVC Clients Storage FriendsApi UsersApi MomentsApi Memcache Feature 1 Feature

    2 Feature N Tuenti social network architecture The test who knew too much...
  6. MVC Clients Storage SubscriptionsApi UsersApi MomentsApi Memcache Feature 2 Service

    Gateway Scripts Tuenti Movil Feature N Tuenti social network architecture during the “transition” The test who knew too much...
  7. MVC SubscriptionsApi UsersApi MomentsApi Feature 2 Service Gateway Scripts Tuenti

    Movil Feature N Storage (tests DB) Fixtures Test A IntegrationTestA Tests during the “transition” The test who knew too much...
  8. MVC SubscriptionsApi UsersApi MomentsApi Feature 2 Service Gateway Scripts Tuenti

    Movil Feature N Storage (tests DB) Fixtures Test A BrowserTestA Selenium/ WebDriver Common Fixtures Tests during the “transition” The test who knew too much...
  9. MVC SubscriptionsApi UsersApi MomentsApi Feature 2 Tuenti Movil Feature N

    BrowserTestA Selenium/ WebDriver Service1 Service1 Service1 IntegrationTestA Tests during the “transition” The test who knew too much...
  10. MVC SubscriptionsApi UsersApi MomentsApi Feature 2 Tuenti Movil Feature N

    BrowserTestA Selenium/ WebDriver Service1 Service1 Service1 IntegrationTestA The test who knew too much... Tests during the “transition”
  11. The test who knew too much... Was it worth the

    trouble? - For Tuenti, yes. “The Big Snowball” Divide & Conquer Everything under control More complex Limited by PHP Different languages fit better for different purposes Huge releases, blocked released Deployment of small pieces is faster Monsters of general purpose Dedicated machines with a specific purpose
  12. Part 2: The dog ate my fixtures

  13. When the persistency layer stops being accessible from the testing

    environment because it is in a service... a. Have an instance of the service running on testing environment. b. Offer a “Stub-Service” with each service to use in testing environment. c. Mock/Stub the code that calls the service. d. Hybrid solution The dog ate my fixtures
  14. a. Have an instance of the service running on testing

    environment. ◦ Pros ▪ All my business flows tests could be still valid! ▪ My tests are going to detect the service contract changes. ◦ Cons ▪ Persistency of the data. ◦ There should exist a setUp routine for each test (fixtures?). ▪ For a certain use case could be too complicated. ▪ Still, nothing ensures that the testing instance behaves like the production instance. ▪ The testing environment becomes more complicated (logs/traces coming from services) The dog ate my fixtures
  15. b. Offer a “Stub-Service” with each service to use in

    testing environment. ◦ Pros ▪ My tests are going to detect the service contract changes. ◦ Cons ▪ Who configures the Stub-Service responses? • How? ▪ To keep working my my business flows tests, the stub needs to be too smart ▪ The Stub-Service gains complexity as the service grows. The dog ate my fixtures
  16. c. Mock/Stub the code that calls the service. ◦ Pros

    ▪ Don’t worry anymore about how it works. Just worry about the input and output. ▪ Traces are simpler ▪ Helps to detect code smells ◦ Cons ▪ All the business flows tests are not valid anymore. • Well, they can work with some twisted logic, but that is not the purpose. ▪ Question: Do the tests still satisfy the contract? • Contract testing The dog ate my fixtures
  17. d. Hybrid solution: ◦ There is no “silver bullet”. ▪

    Do all my tests need to cover the whole stack? ▪ Am I going to be confident using just mocked data? ▪ Where is the limit? ◦ A good combination of test types leads us to a high coverage. The dog ate my fixtures
  18. d. Hybrid solution (cont): • Thumb rule: Low coupling ◦

    Minimize the contact between the components of my application. ◦ And we know how… we’ve been taught! ▪ Remember your OO teacher yelling “high cohesion, low coupling!” ▪ SOLID ▪ Clean Architecture ▪ ... The dog ate my fixtures
  19. Part 3: Low coupling

  20. 1. Isolate the services. 2. Assume asynchronous interactions. 3. More

    than testing: logs & monitoring. Low coupling
  21. 1. Isolate the services • Behind a semantic and meaningful

    layer of apis for my application. ◦ How this API interacts with the rest of the world should not be of my interest. If i use it, I trust on it. ◦ Keep my business logic services agnostic • The test of my features should test my features, not the code in a service! ◦ “The nest of spiders” paradigm ◦ The easier-to-use and more meaningful the APIs and their methods are, the easier it gets the stubbing. Low coupling
  22. 1. Isolate the services My application (My happy features with

    their happy tests) Semantic API layer (Castle black, it just works) Something that the tests of my happy features should not interact with. (The nest of spiders) Low coupling
  23. 1. Isolate the service behind a semantic and meaningful layer

    of apis for my application. My application (My happy features with their happy tests) Semantic API layer (Castle black, it just works) Something that the tests of my happy features should not interact with. (The nest of spiders) Low coupling
  24. 1. Isolate the service behind a semantic and meaningful layer

    of apis for my application. My application (My happy features with their happy tests) Semantic API layer (Castle black, it just works) Something that the tests of my happy features should not interact with. (The spiders nest) Low coupling
  25. 1. Isolate the service behind a semantic and meaningful layer

    of apis for my application. My application (My happy features with their happy tests) Semantic API layer (Castle black, it just works) Something that the tests of my happy features should not interact with. (The spiders nest) Low coupling
  26. 2. Assume asynchronous interactions. Deal with it. • Why? ◦

    Not all the environments have the same requirements ▪ Scripts (background mode). I can wait forever. • retries/monitoring/controlled timeouts => Synchronous. ▪ With the user waiting. • Beware of the timeouts! => Asynchronous. • The user does not like to wait. Early return and process in background. Low coupling
  27. 2. Assume asynchronous interactions. Deal with it. • How? ◦

    Polling, promises, events, notifications, queued jobs… ◦ Ideally I should be able to instantiate the service in synchronous or asynchronous mode for execution. ◦ If the service is slow, I should wrap the call to a service in a background task. ◦ In both cases (asynchronous service response or background task) my API should state explicitly that this method is asynchronous. Low coupling
  28. 3. More than testing: logs & monitoring - Broken you

    said? Impossible! the tests are passing!! - Is testing environment, for God’s sake! Nobody thinks about production? • Even with a high coverage and specially when we have distributed architecture we need monitoring and logs more than ever. Low coupling
  29. My application, my features My application Semantic API layer stub/mock

    Low coupling: How to test? The nest of spiders Unit Integration Acceptance
  30. The semantic API layer Semantic API layer Gateway Low coupling:

    How to test? Unit Integration? The service
  31. The most difficult part Gateway The service Low coupling: How

    to test? Contract testing Service Stub Testing service instance Unit But at least, we have isolated the problem Another application, same strategy.
  32. function onPaymentOk($purchaseId) { $purchase = PurchaseApi::getInstance() ->getPurchaseByPurchaseId($purchaseId); list($invoiceNumber, $invoicePdf) =

    InvoicePdfGenerator::getInstance() ->generateInvoice($purchase); EmailSenderApi::getInstance()->send( $purchase->getEmail(), "SuperStore Invoice num. $invoiceNumber", $pdf ); } Low coupling. Example:
  33. function onPaymentOk($millieuros, $purchaseId) { $purchase = PurchaseApi::getInstance()->getPurchase($purchaseId); InvoiceApi::getInstance()->generateAndSendInvoice($purchase); } function

    generateAndSendInvoice(PurchaseData $purchaseData) { $name = $purchaseData->getName(); InvoiceService::get()->generateAndSendInvoice($name, ...); } // Can we trust an early return and asynchronous execution or should I look into the Invoice service? Low coupling. Example:
  34. // If the invoice service works in asynchronous mode… /*

    @return AsynchronousInvoice */ function generateAndSendInvoice(PurchaseData $purchaseData) { $status = InvoiceService::get(InvoiceServiceMode::ASYNCHRONOUS) ->generateAndSendInvoice(...); if ($status === InvoiceServiceResponseStatus::UNAVAILABLE) $response = AsynchronousInvoice::createUnavailable(); } else { $response = AsynchronousInvoice::createAvailable($data) } return $response; } Low coupling. Example:
  35. // If the invoice service DOES NOT work in asynchronous

    mode… /* @return AsynchronousInvoice */ function generateAndSendInvoice(PurchaseData $purchaseData) { InvoiceGenerationQueuedJob::getInstance() ->generateAndSendInvoice($purchaseData); return AsynchronousInvoice::createUnavailable(); } Low coupling. Example:
  36. … And if something goes wrong, I have monitoring and

    logs! Low coupling. Example:
  37. The End Questions?

  38. Microservices and testing: Talking from the experience Thank you! 2015

    Jairo González Pérez <gpjairo@gmail.com>