Tweet
Tweet

Slide 1

Slide 1 text

ACCELERATING MAVEN BUILDS FROM SNAIL'S PACE TO ROCKET SPEED #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 2

Slide 2 text

| @bmarwell blog.bmarwell.de #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 3

Slide 3 text

, licensed under CC BY-NC 2.5, adopted to match the dark background and not hurt your eyes. "Compiling" by Randall Munroe #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 4

Slide 4 text

(Hans Dockter, Gradle Inc.) β€œ We haven’t seen a company yet where investing into more ef cient builds was not leading to a signi cant return on investment. The most successful software teams on the planet are the ones with an ef cient build infrastructure. Quantifying the Costs of Builds #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 5

Slide 5 text

HOW TO ACCELERATE? ⏱ Measure ̎ Analyse a Improve #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 6

Slide 6 text

MEASURE YOUR BUILD MEASURE YOUR BUILD MEASURE YOUR BUILD MEASURE YOUR BUILD MEASURE YOUR BUILD #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 7

Slide 7 text

THE OPENTELEMETRY MAVEN EXTENSION Time execution recorder for Maven Log time taken by each Mojo in the build lifecycle Comes as a "Plugin on steroids": replaces or hooks into Maven core features Maven extension #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 8

Slide 8 text

INSTALL THE EXTENSION 1. Create .mvn/extensions.xml in your project 2. Register the extension 3. Start an OpenTelemetry collector (such as Jaeger "all-in-one") β†’ io.opentelemetry.contrib groupId> opentelemetry maven extension artifactId> 1.36.0-alpha version> extension> extensions> Simpler alternatives #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 9

Slide 9 text

ACTIVATE THE EXTENSION Run Maven as you did before, but add -Dotel.traces.exporter=otlp When done, open OpenTelemetry #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 10

Slide 10 text

DEMO | βŒ› OpenTelemetry Screenshot #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 11

Slide 11 text

PARALLEL TEST EXECUTION PARALLEL TEST EXECUTION PARALLEL TEST EXECUTION PARALLEL TEST EXECUTION PARALLEL TEST EXECUTION #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 12

Slide 12 text

WHAT IS IT? Parallel execution of tests inside a module. #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 13

Slide 13 text

WHEN TO APPLY If you have idle cores or tests waiting for I/O. 1. Executes large number of tests in parallel 2. Executes long-running tests in parallel #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 14

Slide 14 text

⚠ CAVEATS 1. Tests must be parallelizable Non-thread safe tests will make the build will fail miserably. a Fix the tests a Add @Execute(SAME_THREAD) (JUnit speci c) 2. Can bottleneck the build Use resources that would've built another Maven module. a Disable parallel test execution for speci c modules Applies to parallel builds - more on that later. #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 15

Slide 15 text

HOW TO EXECUTE TESTS IN PARALLEL? 1. Through Sure re 2. Through JUnit #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 16

Slide 16 text

1. USING MAVEN'S SUREFIRE PLUGIN Specify what to run in parallel using the parallel parameter: 1. none 2. methods 3. classes 4. classesAndMethods 5. all #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 17

Slide 17 text

1. USING MAVEN'S SUREFIRE PLUGIN (CTD.) Command line: mvn -Dparallel=classesAndMethods verify . In pom.xml : org.apache.maven.plugins groupId> maven suref re plugin artifactId> 3.2.5 version> classesAndMethods parallel> conf guration> plugin> #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 18

Slide 18 text

PROS AND CONS Advantages: Supports both JUnit 4 and TestNG. Maintained by the Maven core team Disadvantages: All tests in a module need to be thread-safe. Con guration can be repetitive / verbose. Does not apply to JUnit Jupiter. #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 19

Slide 19 text

2. USING JUNIT JUPITER 1. Using src/test/resources/junit-platform.properties 2. Using system properties through argLine parameter 3. Using Java annotations #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 20

Slide 20 text

Using src/test/resources/junit-platform.properties # activation of parallel test execution junit.jupiter.execution.parallel.enabled = true # tests per class (but classes in sequence) junit.jupiter.execution.parallel.mode.default = concurrent # classes in parallel junit.jupiter.execution.parallel.mode.classes.default = concurrent # both options can be combined! #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 21

Slide 21 text

Using system properties through argLine parameter org.apache.maven.plugins groupId> maven suref re plugin artifactId> 3.2.5 version> -Djunit.jupiter.execution.parallel.enabled=true -Djunit.jupiter.execution.parallel.mode.default=concurrent argLine> conf guration> plugin> #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 22

Slide 22 text

Using Java annotations @Execution(CONCURRENT) class MyServiceTest { test methods are now being executed in parallel } #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 23

Slide 23 text

PROS AND CONS Advantages: Finer granularity of test executions Run single tests selectively in sequence Maintained by the JUnit Jupiter team Great IDE support w/ property les and annotations Disadvantages: Speci c to JUnit Jupiter, not in TestNG, Spock and others Harder to control from the command line #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 24

Slide 24 text

DEMO #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 25

Slide 25 text

WITH SEQUENTIAL TESTS #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 26

Slide 26 text

WITH PARALLEL TESTS #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 27

Slide 27 text

MAVEN DAEMON MAVEN DAEMON MAVEN DAEMON MAVEN DAEMON MAVEN DAEMON #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 28

Slide 28 text

WHAT IS IT? Provides faster Maven builds using a daemon process. #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 29

Slide 29 text

HOW DOES IT HELP? 1. Keeps the JVM and plugins "warm". 2. Runs multi-threaded by default without cluttering output. 3. Improved parallelism compared to "vanilla" Maven. Uses to build in topological order. Takari Smart builder #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 30

Slide 30 text

WHEN TO APPLY? 1. Large project. 2. Many plugins. 3. "Heavy" plugins. #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 31

Slide 31 text

⚠ CAVEATS 1. Multi-threaded by default Non-threadsafe plugins may make build fail miserably. a Circumvent by specifying -T 1 a Address the root cause: x the plugin(s) 2. Re-implements some parts of Maven Parsing .mvn/maven.config has some bugs: , . apache/maven-mvnd#913 apache/maven-mvnd#917 #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 32

Slide 32 text

HOW? 1. Install Maven Daemon 2. Retrain nger gymnastics: mvn β†’ mvnd #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 33

Slide 33 text

HOW? (CTD.) A few new tricks: Use mvnd --status to inspect details of the daemon process. Add -Dmvnd.buildTime=true for simple timing report. Add -Dsmartbuilder.profiling=true for advanced timing report. #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 34

Slide 34 text

DEMO Screenshots: | Maven Maven Daemon #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 35

Slide 35 text

PARALLELIZING YOUR BUILD PARALLELIZING YOUR BUILD PARALLELIZING YOUR BUILD PARALLELIZING YOUR BUILD PARALLELIZING YOUR BUILD ! MAVEN MODULES EVERYWHERE ! #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 36

Slide 36 text

CASE STUDY: HEXAGONAL ARCHITECTURE #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 37

Slide 37 text

INITIAL MAVEN MODULES #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 38

Slide 38 text

REFACTORED MAVEN MODULES #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 39

Slide 39 text

EVALUATION PROS Faster builds when there are enough modules with few, distinct dependencies (Almost) no need for ArchUnit or alike Easier to exclude transient dependencies via Maven CONS Could hinder navigation between classes and modules Needs awareness of runtime vs compile dependencies May not be suitable for libraries #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 40

Slide 40 text

TO DO OR NOT TO DO? As always: it depends! Pro ts vs. added complexity For libraries: impact for consumers? For applications: does it impact architecture? #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 41

Slide 41 text

DEMO This is what happens when you do not have any long-running tests: SHIRO W/ 62 MODULES (2:01 MIN) SHIRO W/ 58 MODULES (1:59 MIN) #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 42

Slide 42 text

MAVEN BUILD CACHE EXTENSION MAVEN BUILD CACHE EXTENSION MAVEN BUILD CACHE EXTENSION MAVEN BUILD CACHE EXTENSION MAVEN BUILD CACHE EXTENSION #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 43

Slide 43 text

HOW DOES THE BUILD CACHE WORK? Reduces build time by skipping parts of the build which don't need to be executed. 1. Only execute parts of the build which changed. 2. Build subtrees of a multi-module project in isolation. 3. Normalize the version to avoid rebuilding across forks. 4. Project state restoration (e.g. artifacts) #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 44

Slide 44 text

1. WHICH PARTS HAVE CHANGED? A. MODIFICATIONS TO THE POM Changing dependencies, modifying plugins or executions, will invalidate the cache. B. MODIFICATION OF SOURCE CODE Use content ngerprinting to invalidate cache when source code changes. #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 45

Slide 45 text

WHEN TO USE THE CACHE EXTENSION? 1. You use Maven 3.9 or Maven 4. 2. Your project does not rely on 'quirks'. 3. You want to speed up 1. local developer build, and/or 2. CI builds (PRs), and/or 3. want to share cache with co-workers. 4. You are willing to invest time to make the cache portable. #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 46

Slide 46 text

INSTALL THE EXTENSION 1. Create .mvn/extensions.xml in your project. 2. Register the extension by adding the following content: 3. (Optional) extend the default cache con guration org.apache.maven.extensions groupId> maven build cache extension artifactId> 1.2.0 version> extension> extensions> #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 47

Slide 47 text

DEMO DEMO DEMO DEMO DEMO #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 48

Slide 48 text

⚠ CAVEATS 1. Con gure relevant input les 2. Add plugin parameters for reconciliation 3. Filter non-essential les (IDE, etc) 4. Expect your tests not to run 5. Remote cache not used (OS, pro les...) 6. Cache eviction #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 49

Slide 49 text

HONOURABLE MENTIONS HONOURABLE MENTIONS HONOURABLE MENTIONS HONOURABLE MENTIONS HONOURABLE MENTIONS We did not cover: Reduce logging (Logback, Log4J, mvn -q ) β€” try it! Christian Stein's "junit5-looming" (showcase): JUnit 5 test engine with virtual thread support Virus and threat protection User home directory on network drive #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 50

Slide 50 text

WHAT DID WE LEARN? WHAT DID WE LEARN? WHAT DID WE LEARN? WHAT DID WE LEARN? WHAT DID WE LEARN? 1. Measure your builds 2. Parallel test execution per module 3. Using the Maven Daemon 4. Feed mvnd with architecture improvements 5. Maven Build Cache Extension #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)

Slide 51

Slide 51 text

REFERENCES bit.ly/snail-pace-rocket-speed-demos blog.bmarwell.de maarten.mulders.it @bmarwell @mthmulders #Maven #SnailToRocket Benjamin Marwell (@bmarwell) & Maarten Mulders (@mthmulders)