jPrime 2024: The evolution of integration testing within Spring and Quarkus

In this session we will have a look at how the landscape of integration tests, especially in a setup that requires external services, has changed over the past 5 years. Back at Spring Boot 2.0 times testing your application agains external services such as databases, message queues and alike was still kind of a mess: Several approaches like Docker Maven plugin existed, but they have been a lot of caveats and downsides. With the rise of Testcontainers things changed a lot. We will have a quick look what it means creating a Testcontainers module - using Neo4j as an example - and then going from the original Spring approach with dynamic properties, the breath of fresh air that Quarkus introduced with their Devservices into modern testing with Spring Boot 3.1+. I am gonna also demonstrate how valuable a programmatic approach is for me as a developer of services must be compatible with a broad range of database versions and what role a cloud offering for Testcontainers plays in that…

Transcript

1. jPrime 2024 The evolution of integration testing within Spring and Quarkus
2. “We improved our app” 😱 • More, not less features • More and more frequent versions • Less time for everything, especially testing and quality Michael Simons at jPrime 2024 // © Neo4j Inc. 2
3. Agenda 1. What’s my stake in this? 2. Mocking? 3. A short trip into the dark ages and how Docker and declarative Spring Boot config brought some light to scene in this decade 4. To embed or Not to embed? 5. Enter Testcontainers 6. Wow, 5 years already that Quarkus brought fresh ideas! 7. How did Spring Boot evolve? Michael Simons Java & Testcontainers Champion, Senior Staff Engineer at Neo4j Michael Simons at jPrime 2024 // © Neo4j Inc. 3
4. No AI 🤖 today, sorry. Neo4j Inc. All rights reserved 2024 4 Neo4j has a couple of things in store, though: • The vector store and index itself • Graph Data Science library ◦ Embeddings, Link Predications, Centrality, Community detection and more • Implementing Advanced Retrieval RAG Strategies With Neo4j https://neo4j.com/developer-blog/advanced-rag-strategies-neo4j/ • Works great with Spring AI, and Ollama models in… Testcontainers: https://java.testcontainers.org/modules/ollama/ Principles explained down the road apply, too!
5. What’s my stake in this? Michael Simons at jPrime 2024 // © Neo4j Inc. 5
6. Query languages are not an offence. Michael Simons at jPrime 2024 // © Neo4j Inc. 6 • My whole career has been somewhat related to databases ◦ Client/Server Oracle Forms 6i used to be simple ◦ Things got wild with Java, Spring and ORMs ▪ Do I want to test their queries? Maybe just for performance? ▪ And then you write custom queries • Got hired by Neo4j to work on an 🥁 ORGM ◦ With Gerrit Meier core maintainer of Spring Data Neo4j ◦ Now generating all the queries ◦ And faced with a multidimensional test-matrix: ▪ Multiple versions of a database ▪ Multiple versions of a driver for that database Neo4jContainer<> ftw!
7. What about Mocking? Michael Simons at jPrime 2024 // © Neo4j Inc. 7
8. This is about integration testing Michael Simons at jPrime 2024 // © Neo4j Inc. 8 • Mocking aims at testing the components above the data access layer • Happy to apply mocking in my work for the mapping approach • I would rather not mock away the state given in a bigger database • Same goes for mocking behaviour of products (database, drivers) ◦ Think Spring Data Neo4j (SDN) as a suite of involuntary regression tests for our database
9. Memory lane Michael Simons at jPrime 2024 // © Neo4j Inc. 9 Shared credentials for test instances and “oh, did your test just nuke my data?” fun…
10. A really, long time ago… Michael Simons at jPrime 2024 // © Neo4j Inc. 10 One of many persistence.xml
11. Things got better, eventually… Michael Simons at jPrime 2024 // © Neo4j Inc. 11
12. Michael Simons at jPrime 2024 // © Neo4j Inc. 12
13. How did this look like? Michael Simons at jPrime 2024 // © Neo4j Inc. 13 Lifecycle and profile management, essentially…
14. Still in use today • Any testing on the module path https://github.com/neo4j/neo4j-jdbc/blob/main/neo4j-jdbc-it/neo4j-jdbc-it-mp/p om.xml#L71 • Pointing to an external service on purpose, circumventing dev-services in Quarkus extension testing https://github.com/quarkiverse/quarkus-neo4j/blob/main/integration-tests/pom. xml#L81 Michael Simons at jPrime 2024 // © Neo4j Inc. 14
15. To embed or Not to embed? Michael Simons at jPrime 2024 // © Neo4j Inc. 15 Or: Close, but no cigar.
16. The embedded component approach Michael Simons at jPrime 2024 // © Neo4j Inc. 16 • The component runs on the JVM ◦ Databases like Neo4j ◦ Event-stores and stream processors like Kafka ◦ Message brokers like ActiveMQ • There are wrappers to run it on the JVM ◦ Flapdoodle (i.e. MongoDB) • There’s something “close enough” (Only if at least a driver abstraction such as JDBC is used) ◦ Embedded or in-process databases for relational ◦ Different message broker, abstracted away behind JMS
17. To embed, because… • No (or little) infrastructure required (It’s all the same VM or at least machine) • Locality: Your tests will never interfere with other peoples tests running at the same time (You can still mess up other tests with bad test-data of course) • Performance: Often times embedded is fast to start • You actually want to test something inside the product, like a stored procedure Michael Simons at jPrime 2024 // © Neo4j Inc. 17
18. Not to embed, because… • Coupling ◦ Your tests now requires a JDK version that the embedded thing needs ◦ Your tests may bring up Javax v. Jakarta dependencies, issues in both directions ◦ Generally speaking: Dependency hell is real 🔥 • Leaves out any networking issues to test ◦ Latency, Traffic-size, retries etc. • Just because abstraction X (like JDBC), does not mean the query language is fully supported or semantically equivalent Michael Simons at jPrime 2024 // © Neo4j Inc. 18
19. Demo: The case for not to embed. Michael Simons at jPrime 2024 // © Neo4j Inc. 19
20. Testcontainers Michael Simons at jPrime 2024 // © Neo4j Inc. 20 The elephant whale 🐳 in the room…
21. What is Testcontainers Michael Simons at jPrime 2024 // © Neo4j Inc. 21 • Testing library, not a framework, for lifecycle management of real services wrapper in containers • API for straight access to exposed ports on random host ports • Same for generated credentials • Couple the lifecycle of the container to the lifecycle of your typical integration tests ◦ Before test: Start all required services, reconfigure your framework to use them ◦ Tests: As usual ◦ After test: Safely clean up resources, regardless whether your test crashes the whole JVM
22. Ever seen that guy? Ryuk, a Shinigami, which is a god of death in Japanese. Making sure people whose names have been put into a death note 📙, die. Here, it’s just a container brought up by Testcontainers, that kills remaining containers. Michael Simons at jPrime 2024 // © Neo4j Inc. 22
23. Not only databases… There’s a lot more, such as: • LocalStackModule (Local AWS Stack) https://java.testcontainers.org/modules/localstack • GCloud (Same, but for GCP, currently incubating) https://java.testcontainers.org/modules/gcloud/ • Ollama https://java.testcontainers.org/modules/ollama/ Neo4j Inc. All rights reserved 2024 23
24. Demo: The Testcontainers way. Michael Simons at jPrime 2024 // © Neo4j Inc. 24
25. How does that help a library developer? Michael Simons at jPrime 2024 // © Neo4j Inc. 25 Safely set to true, can be disabled on ci via Testcontainers settings if needed Configure via env properties on CI
26. How does that help a library developer? Michael Simons at jPrime 2024 // © Neo4j Inc. 26 Programmatically generate a matrix
27. Frameworks Michael Simons at jPrime 2024 // © Neo4j Inc. 27 The new challenger: Quarkus
28. Change and challenge is good Michael Simons at jPrime 2024 // © Neo4j Inc. 28 • Quarkus initially released in March 2019 • Focus on ◦ fast startup time ◦ developer joy • We (Neo4j) want to be part of that platform, too ◦ Learn a lot about built-time optimization ◦ Made our driver fly with GraalVM => Benefited all our integrations • Maintainers of https://github.com/quarkiverse/quarkus-neo4j • Dev-Services support!
29. Developer joy keypoints • Live coding • + unified config • + dev services • + continues testing Michael Simons at jPrime 2024 // © Neo4j Inc. 29
30. Demo: Managed Testcontainers in Quarkus Michael Simons at jPrime 2024 // © Neo4j Inc. 30
31. Full fledged resource management Quarkus provides QuarkusTestResourceLifecycleManager: • Can enhance the environment similar to Spring Boots DynamicPropertyRegistry • Dedicated API managing container based services, including shared networking ◦ In Quarkus mostly helpful when testing the already containerized application (putting things on the same network) Michael Simons at jPrime 2024 // © Neo4j Inc. 31
32. Application managed containers Michael Simons at jPrime 2024 // © Neo4j Inc. 32
33. Back to Spring Boot Michael Simons at jPrime 2024 // © Neo4j Inc. 33 Learning new tricks, too
34. What’s new since 3.1? Michael Simons at jPrime 2024 // © Neo4j Inc. 34 • Managed Testcontainers dependencies ◦ Including the basic junit-jupiter Testcontainers module • ConnectionDetails, a marker interface for a generic contract, indicating access to remote resources ◦ Generic JDBC, dedicated relational databases, Neo4j, Kafka, Cassandra, you name it… • @ServiceConnection, bringing together resources in containers, the connection details into managed beans • Augmented main classes, think dev-services, but explicit!
35. Michael Simons at jPrime 2024 // © Neo4j Inc. 35
36. Demo: Managed Testcontainers in Spring Boot Michael Simons at jPrime 2024 // © Neo4j Inc. 36
37. Recap Michael Simons at jPrime 2024 // © Neo4j Inc. 37 • Containers made it easy to test the “real thing” • Testcontainers bring lifecycle management • Reusable Testcontainers mitigate startup penalty • Frameworks picked this up • Mutual respect and inspiration forsters the whole Java ecosystem
38. Thank you! Michael Simons at jPrime 2024 // © Neo4j Inc. 38 Contact michael.simons@neo4j.com Social media @rotnroll666(@mastodon.social) Profile https://github.com/michael-simons Repository https://github.com/neo4j-examples/neo4jX testcontainers I write books: