How fixing a broken window cut down our build time by 50%

The broken windows (not the operating system - 😁) theory not only applies to criminological cases but also codebases. It only takes one lazy developer to break the first window. Chances are high that others will follow a similar path and take workarounds or re-apply a not so optimal pattern thinking, "It passed the code review, I must not be that bad".

For our Spring Boot integration test setup, we had such a broken window that was causing ever-growing build times. While most books about clean architecture emphasize composition over inheritance, we drifted off from this rule and coupled our tests to an AbstractIntegrationTest class that we then annotated with @DirtiesContext...

This resulted in ever-growing build times as each new integration test was launching its own Spring TestContext, adding valuable seconds to the overall build time. Our feedback cycles, as well as the overall satisfaction of the team, started to suffer from this.

This talk describes our journey of getting rid of @DirtiesContext on top of an abstract test class (aka. the "broken window") that made reusing the Spring TestContext impossible and resulted in ever-growing build times. Furthermore, we'll discuss recipes, best practices, and antipatterns for resuing the Spring TestContext and testing Spring Boot applications in general.

After fixing this "broken window", our overall build time went down by 50% from 25 to 12 minutes.

Transcript

2. @rieckpil How Fixing a Broken Window Cut Down Our Build Time by 50%
3. @rieckpil How It Started 3
4. The Broken Windows Theory* “The broken windows theory is a criminological theory that states that visible signs of crime, anti-social behavior, and civil disorder create an urban environment that encourages further crime and disorder, including serious crimes. The theory suggests that policing methods that target minor crimes such as vandalism, loitering, public drinking, jaywalking, and fare evasion help to create an atmosphere of order and lawfulness.” https://en.wikipedia.org/wiki/Broken_windows_theor y *no relationship to the operating system 4
5. @rieckpil It Does Not Always Have to Be a Window 5
6. @rieckpil It Does Not Always Have to Be a Window 6
7. @rieckpil Impact of Fixing the Broken Window Before fixing the broken window After fixing the broken window 7
8. @rieckpil Agenda for This Talk 1. Introduction ✅ 2. About Me 3. Theory: Spring Test Context Caching & @DirtiesContext 4. Practice: Our Journey of Fixing the Broken Window 5. Conclusion and Q&A 8
9. @rieckpil Goals for This Talk Emphasize the importance of well- written tests Learn more about the Spring Test context caching mechanism Don’t be the person that breaks the first window! Understand the impact a small workaround can have on the health of your project 9 🏆
10. @rieckpil About Me - Freelance IT consultant from Berlin, Germany - Enjoys writing blog posts on rainy days in cozy cafes (#TeamWordPress) - Blogging & content creation for more than five years. Since two years with a focus on testing Java and specifically Spring Boot applications - Co-authored Stratospheric - Form Zero to Production With Spring Boot and AWS together with Björn Wilmsmann (@bwilmsmann) and Tom Hombergs (@TomHombergs) - Blog: https://rieckpil.de - GitHub: https://github.com/rieckpil/ - YouTube: https://www.youtube.com/c/rieckpil 10
11. Spring TestContext Framework: Context Caching Theory - Part of Spring Test (automatically part of every Spring Boot project via spring-boot-starter- test) - Spring TestContext Framework: caches an already started Spring context for later reuse - Speed up the test execution time by reusing an already started context 11
12. @rieckpil Spring TestContext Framework Context Caching - Configurable cache size (default is 32) with LRU (least recently used) strategy - This goes into the cache key (MergedContextConfiguration): - activeProfiles (@ActiveProfiles) - contextInitializersClasses (@ContextConfiguration) - propertySourceLocations (@TestPropertySource) - propertySourceProperties (@TestPropertySource) - contextCustomizer (@MockBean, @DynamicPropertySource, …) - etc. 12
13. @rieckpil Can Spring Reuse the Context? 13
14. @rieckpil Can Spring Reuse the Context? 14
15. @rieckpil Context Caching and Reusability Red Flags 15
16. @rieckpil Context Caching Indicators 16
17. @rieckpil Context Caching Indicators 17
18. @rieckpil @DritiesContext aka. Our Broken Window From the @DirtiesContext Javadoc > Test annotation which indicates that the ApplicationContext associated with a test is dirty and should therefore be closed and removed from the context cache. Use this annotation if a test has modified the context — for example, by modifying the state of a singleton bean, modifying the state of an embedded database, etc. Subsequent tests that request the same context will be supplied a new context. 18
19. @rieckpil Our Status Quo - Almost all our integration tests (*IT) extended the AbstractIntegrationTest class - Every new IT was implicitly inheriting this “bad behavior” and increased our build time by 20-30 seconds - No effort was made to work on a fix - We had multiple attempts of “just remove @DirtiesContext” and see what breaks - The increasing build times resulted in a decreasing motivation to add further valuable and important integration test - Our overall team motivation decreased as the feedback cycles went slower and slower 19
20. @rieckpil Share the new knowledge Get rid of @DirtiesContext Identify hot spots and rabbit holes Plan this tech debt removal accordingly Our Strategy 20 🚀
21. @rieckpil Pitfall #1: Message Thefts 21
22. @rieckpil Pitfall #1: Message Thefts 22
23. @rieckpil Pitfall #1: Message Thefts 23
24. @rieckpil Pitfall #1: Message Thefts 24
25. @rieckpil Pitfall #1: Message Thefts 25
26. @rieckpil Pitfall #2: Data Overflow 26
27. @rieckpil Pitfall #2: Data Overflow - Avoid any test data leftovers between tests - Concepts to understand: - EntityManager flushing - Second-level vs. first-level cache - Transaction handling during tests - Ensure every test starts with a clean state - Provision the database via Testcontainers - Use schema migration tools like Flyway or Liquibase - LocalStack for S3 Buckets, etc. 27 💽
28. @rieckpil Pitfall #3: Remote Connections 28
29. @rieckpil Pitfall #3: Remote Connections 29 - Aim for “airplane” mode - Avoid any real HTTP communication to external services during the build - Predictable and reproducible test outcomes - WireMock to the rescue!
30. @rieckpil Pitfall #3: Remote Connections 30
31. @rieckpil Pitfall #3: Remote Connections 31
32. @rieckpil Pitfall #3: Remote Connections 32 …
33. @rieckpil Pitfall #4: @SpringBootTest Obsession 33
34. @rieckpil Pitfall #4: @SpringBootTest Obsession 34 🙈
35. @rieckpil Pitfall #4: @SpringBootTest Obsession 35
36. @rieckpil Pitfall #4: @SpringBootTest Obsession 36
37. @rieckpil Pitfall #5: Setup Variety 37
38. @rieckpil Pitfall #5: Setup Variety - Group the config to the least amount of profiles needed, for example: integration-test and web-test - Avoid “unique snowflake” context setups - Don’t overuse @MockBean & @SpyBean - Avoid manual property overrides where possible - Understand what goes into the context cache key - (Custom) JUnit Jupiter extensions > extending an abstract test base class 38 🥗
39. @rieckpil The Result - Three-day full-time investment - Reducing the build time from 26 minutes to 12 minutes - New integration tests are only adding a negligible amount of additional build time - Final context cache size: 3 (before: 20) - Internal knowledge sharing session about the identified pitfalls and applied techniques & patterns - Increased awareness for this topic when writing tests for Spring Boot applications 39 🥳
40. @rieckpil Testing Spring Boot Applications Masterclass - Testing real-world Spring Boot applications - 130+ course lessons with 10h+ hands-on video content - Tech-Stack: Spring Boot 2.7, PostgreSQL, React 18, Java 17, Keycloak (OAuth 2 login), communication with a remote HTTP system, async workflows with a messaging queue (Amazon SQS) - Techniques, pitfalls, and patterns for unit, integration, and end-to-end testing - Best of breed testing tools: JUnit 5, Mockito, LocalStack, Testcontainers, Awaitility, WireMock, MockWebServer, Selenide, etc. - PS: Spring Test context caching is also covered in-depth - https://rieckpil.de/testing-spring-boot-applications-masterclass/ 40
41. @rieckpil Stratospheric - From Zero to Production with Spring Boot and AWS - 450+ pages long hands-on ebook about getting your Spring Boot app running on AWS: Available on Leanpub - Java 17, Spring Boot 2.7 with Spring Cloud AWS - Fully automated CI/CD pipeline with GitHub Actions - AWS Services: ECS, ECR, SQS, CloudWatch, RDS, Cognito, MQ, etc. - Infrastructure as Code (written in Java) with the AWS CDK v2 - Complementary online course is currently in progress - More information (discount, mailing list, etc.): https://stratospheric.dev/ 41
42. @rieckpil Conclusion - Think twice before implementing a workaround (aka. breaking the first window) - Treat testing code as production code and keep it maintainable and clean - Don’t religiously try to minimize the number of contexts: measure & adjust - Try to avoid using @DirtiesContext at a central place - Show some love for your test suite (your future self and coworkers will thank you) - Avoid having unique snowflakes aka. fruit salads when it comes to your test ApplicationContext setup - Composition over inheritance - Short feedback cycles ftw! 42 🏆
43. @rieckpil Q&A Questions? Slides will be uploaded ✅ The best way to get in contact is via Twitter @rieckpil (DMs are open) ✅ Visit my blog and join the mailing list for more testing-related Spring Boot content ✅ Enjoy the rest of the conference ✅ 43