José San Román Álvarez de Lara_Bank on the outside, start up on the inside, developing software @ scale_Codemotion Madrid 2019

Building decoupled software is essential when you want to maintain velocity while developing a product in a global, distributed way. However, following the wrong parameters will handicap your quality assurance. How to be sure your product works? I’m a staunch advocate of using unit tests. They are faster and less flaky than integrated tests and allow you to scale up like a startup. Don’t be fooled! Integrated tests will kill your product. Among other things I will introduce the concept of contract testing and show you examples of PACT, the most suitable Java library to implement this concept.

About:
José San Román Álvarez de Lara, CIO, ING Bank España

José is the Chief Information Officer (CIO) of ING Bank in Spain, responsible for developing the local IT strategy and aligning it with ING’s global ambition to build the digital bank of the future. Before joining ING as a software architect in 2005, he helped various tech companies to construct complex applications and systems using advanced software engineering methodologies, tools and practices. In particular he’s an avid fan of Agile, XP, devops and continuous delivery. Passionate about innovation and technology, he also loves MTB downhill and occasionally runs.

Transcript

1. Bank from the outside, startup in the inside Developing software @ scale José San Román Álvarez de Lara (CIO ING Spain) 24-25 September, 2019
2. Developing software @ scale
6. +50 teams having the speed and the customer focus of a start-up
7. As software ages and grows it appears to atrophy
8. Codebase grows Not easy to assess impact of changes Minimum impact work style Complexity Lower speed
9. How to scale?
11. Each piece should be a complete system itself
12. Decoupled software
13. Shares state via a clear contract Built, test & deployed independently Does not require big knowledge to consume Decoupled software
14. Decoupled teams
15. Each team is a complete organization itself
16. How the team implements software should not be important outside the team
17. Rely on a common platform to ensure proper alignment
19. Platform /ˈplatfɔːm/ “A raised level surface on which people or things can stand” (*) Source: https://www.lexico.com/en/definition/platform
20. Organization
21. Conway’s law “… organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.” (*) Source: http://www.melconway.com/Home/Conways_Law.html
22. Inverting the Conway’s law
23. Build an organization that promotes your desired architecture
25. Mortgages Loans Debit card Risk engine Account Management System Simulation & data collection Origination Servicing Case manager
26. Handicaps
27. Decoupled organization handicaps More complexity in operations Dependencies may slow you down The system is not easy to test
28. Complex operations
30. Business functionality Configuration Discovery Security Monitoring Logging Networking …
31. Rely on a platform
32. Dependencies
34. Economy of scale does not apply to long-term software
35. Reuse only due to strong requirements
36. Avoid inter-domain dependencies
37. Allow dependencies only on vertical slices
38. Mortgages Loans Debit card Risk engine Account Management System Dependencies
39. Apply the dependency inversion principle @ organization level
40. Mortgages Loans Debit card Risk engine Account Management System Dependencies
41. Not easy to test
43. Integrated tests “any test whose result (success of fail) depends on the correctness of the implementation of more than one piece of non-trivial behaviour” (*) Source: https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam
44. Integrated tests are very very flaky
45. Flaky tests by Google (*) Source: Google’s testing blog https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html
46. Flaky tests will kill your speed
48. Team starts writing unit tests We find bugs Team writes some integrations tests We still find bugs Team writes some UI tests Integration and/or UI tests becomes slow, unstable and not easy to maintain Team starts disabling tests
50. Isolated tests “tests that does not have any side effects”
51. Unit tests are the better isolated tests
52. ~77% #bugs found in production can be detected using unit tests (*) Simple Testing Can Prevent Most Critical Failures: https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
53. Can we use unit tests to check the integrations points?
54. Use stubs (canned response) to test behaviour
55. Use mocks (expectations) to test collaboration
56. “Don’t mock what you don’t own”
57. How to check your mocking assumptions?
59. A contract includes the syntax & the semantics
60. Interface & behaviour
61. Consumer driven
62. Dependency inversion
63. Contract Provider contract test Provider verifier Provider Request Response Consumer contract test Consumer Provider Mock Request Response
64. Same confidence Lower costs Easy to maintain Very stable
65. Microservices
66. Services Domain Repositories Third parties Resources Database Provider’s contracts
67. Domain behaves as expected Domain collaborates as expected Consumed third parties comply with contracts
68. Domain behaves as expected
69. Services Domain Stubbed repositories Stubbed third parties Resources Database Provider’s contracts Behaviour unit tests
70. Domain collaborates as expected
71. Services Domain Stubbed repositories Mocked third parties Resources Database Provider’s contracts Collaboration unit tests
72. Third parties comply the contracts
73. Services Domain Repositories Third parties Resources Database Stubbed provider’s contracts Consumer contract unit tests
74. Stubbed services Domain Repositories Third parties Resources Database Provider’s contracts Provider contract unit tests
75. How to?
77. Move money Risk engine
78. <artifactId>move-money</artifactId> <parent> <artifactId>codemotion-2019</artifactId> <groupId>com.ing.jsr</groupId> <version>1.0-SNAPSHOT</version> </parent> <dependencies> […] <dependency> <groupId>au.com.dius</groupId> <artifactId> pact-jvm-consumer-java8_2.12 </artifactId> <version>${pact.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>au.com.dius</groupId> <artifactId> pact-jvm-consumer-junit_2.12</artifactId> <version>${pact.version}</version> <scope>test</scope> </dependency> </dependencies> <artifactId>risk-engine</artifactId> <parent> <artifactId>codemotion-2019</artifactId> <groupId>com.ing.jsr</groupId> <version>1.0-SNAPSHOT</version> </parent> <dependencies> […] <dependency> <groupId>au.com.dius</groupId> <artifactId>pact-jvm-provider-junit_2.12</artifactId> <version>${pact.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>au.com.dius</groupId> <artifactId>pact-jvm-provider-spring_2.12</artifactId> <version>${pact.version}</version> <scope>test</scope> </dependency> </dependencies>
79. public class MoveMoneyTest { @Rule public PactProviderRuleMk2 riskEngine = new PactProviderRuleMk2("risk-engine", PactSpecVersion.V3, this); @Pact(provider = "risk-engine", consumer = "move-money") public RequestResponsePact pactForHighRiskOperation(PactDslWithProvider builder) { // See source code in next slides } @PactVerification(fragment = "pactForHighRiskOperation") @Test public void testHighRiskOperation() { // See source code in next slides } }
80. @Pact(provider = "risk-engine", consumer = "move-money") public RequestResponsePact pactForHighRiskOperation(PactDslWithProvider builder) { Map<String, String> headers = new HashMap<>(); headers.put("Content-Type", MediaType.APPLICATION_JSON_UTF8_VALUE); return builder .given("A high risk operation") .uponReceiving("A request to run a risk validation") .path("/api/validate").method(RequestMethod.POST.name()) .body(newJsonBody((o) -> { o.stringValue("channel", Channel.WEB.toString()); o.stringValue("source", RiskEngineInfoBuilder.SOURCE_ACCOUNT); o.stringValue("dest", RiskEngineInfoBuilder.DEST_ACCOUNT); o.decimalType("quantity", RiskEngineInfoBuilder.HIGH_QUANTITY); }).build()) .willRespondWith().headers(headers).status(200) .body(newJsonBody((o) -> { o.stringValue("riskLevel", RiskLevel.HIGH.toString()); }).build()) .toPact(); } }
81. @Test @PactVerification(fragment = "pactForHighRiskOperation") public void testHighRiskOperation() { RiskEngine engine = new RiskEngineRestImpl(riskEngine .getUrl()); RiskEngineInfo info = RiskEngineInfoBuilder.create() .forWebChannel() .withAHighAmmount() .build(); RiskEngineResult result = engine.validate(info); assertEquals(result.getRiskLevel(), RiskLevel.HIGH.toString()); } }
82. @RestController public class RiskEngineController { @Autowired private RiskEngine engine; @PostMapping(path = "/api/validate") public RiskEngineResult validate(@RequestBody RiskEngineInfo riskInfo) { return engine.validate(riskInfo); } } @Service public class RiskEngine { @Autowired private BadAccountsRepository badAccounts; @Autowired private KnownAccountsRepository knownAccounts; public RiskEngineResult validate(RiskEngineInfo info) {[…]} private void calculateRiskDueToDestionation (RiskEngineInfo info, RiskEngineResult result){[…]} private void calculateRiskDueToAmmount (RiskEngineInfo info, RiskEngineResult result) {[…]} }
83. @RunWith(SpringRestPactRunner.class) @Provider("risk-engine") @PactFolder("pacts") @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) public class RiskEngineContractTests { @MockBean private RiskEngine engine; @TestTarget public final Target target = new SpringBootHttpTarget(); @State(”A high risk operation") public void highRiskOperation() { when(engine.validate(any())) .thenReturn(new RiskEngineResult().setRiskLevel(RiskLevel.HIGH.toString())); } }
84. @RunWith(SpringRunner.class) @SpringBootTest public class RiskEngineTests { @MockBean private BadAccountsRepository badAccounts; @MockBean private KnownAccountsRepository knownAccounts; @Autowired private RiskEngine riskEngine; @Test public void testForbiddenOperationDueToBadAccount() { when(badAccounts.find(any())).thenReturn(true); when(knownAccounts.find(any())).thenReturn(false); RiskEngineInfo info = RiskEngineInfoBuilder.create().forWebChannel().withALowAmmount().build(); RiskEngineResult risk = riskEngine.validate(info); assertEquals(RiskLevel.FORBBIDEN.toString(), riskEngine.validate(info).getRiskLevel()); } [….] } }
85. Takeaways
86. Build decoupled software to ensure you are able to maintain your speed
87. Organize teams in a way that promotes your desired architecture
88. Ensure you simplify your operations relying on a platform
89. Limit & manage your dependencies
90. Write isolated tests Avoid integrated tests
91. Use contract tests to check integration points and validate mocking assumptions
92. Invert dependencies applying consumer driven
94. Thks! ;-)
95. Questions?