Rebooting the OpenOffice.org Build System

OpenOffice conference 2010, Budapest Hungary

Transcript

2. <Insert Picture Here> Björn Michaelsen Software Developer Rebooting the OpenOffice.org Build System
4. <Insert Picture Here> 4 Program Agenda • Build Environment Effort at Oracle - goals - general approach - wind of change - other topics • Status Quo of the Build System - architecture, problems and scalability - a way to do it better? • GNU Make Build System
5. <Insert Picture Here> 5 Build Environment Effort at Oracle Build Environment Effort at Oracle
6. <Insert Picture Here> 6 Albert Einstein “Everything should be made as simple as possible, but not simpler.” It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience. – "On the Method of Theoretical Physics" The Herbert Spencer Lecture, delivered at Oxford (10 June 1933)
7. 7 Build Environment Effort: Goals • Create a compelling, user friendly build experience by making building OOo less complex, stable and faster • Concrete goals: – Improve the build performance – Make the build reliable even for non-experts – Better support for execution of tests in the build – Create runnable office in the build
8. 8 Build Environment Effort: General Approach Build environment and code influence each other. If something shall be fixed, consider all options: • Fix the build environment • Fix the code • Optimize cooperation by fixing both
9. 9 • Abandon def and map files on Windows • Fix DLL_IMPLEMENTATION usage • Fix tools requiring backslashes in cmd line • Reduce number of library naming schemas • Separate internal and delivered headers • Rethink registration process for UNO libraries • Get rid of delivered static libraries/object files • Get rid of stlport • ... Build Environment Effort: Wind of Change
10. 10 • Documentation • Common Trees • Resources – Images – HIDs Build Environment Effort: Other Topics See also: • “The Taming Of the Shrew” • “Go Test! (life is faithful there)”
11. <Insert Picture Here> 11 Status Quo of the Build System Status Quo of the Build System
12. 12 • We are currently using two tools to build OpenOffice.org: – build.pl / deliver – dmake • dmake works more or less like any other make tool • build.pl iterates over “modules” and directories and starts dmake processes • build.pl learns about dependencies by “build.lst” files • deliver copies module output into the “solver” Current Build System: Architecture
13. 13 Current Build System: Scalability
14. 14 • Parallelization • Full dependencies – module borders – “compatible builds” • Cargo cult programming in makefiles • Source tree is not read-only • Lots and lots of process instantiations • Too many tools for the same job • Maintenance of “homegrown” tools Current Build System: Problems
15. 15 A Way to Do It Better? • Problems in the current system – manual dependencies – incompatible builds – scalability – TIMTOWTDI • Consequences – broken dependencies – horizon effects – build segmentation – bad quality of makefiles
16. 16 A Way to Do It Better?
17. 17 A Way to Do It Better?
18. 18 Requirements for a new Build System • Scalability/Parallelization • Lean dependencies • No custom tooling • Full dependencies • Easy to use • Code quality • Multiple repository support
19. <Insert Picture Here> 19 GNU Make Build System GNU Make Build System
20. 20 GNU Make Build System: Scalability • Number of targets: not limit found • Number of rules: problems start at ~10000 rules Solution: Use pattern rules • Command line limits not hit until now – Cygwin ~ 1MB However, link.exe is quite picky about long command lines. solution: responsefiles – Linux ~ 100 KB (depending on stack limit) • GNU Make internal limits: only limited by amount of RAM
21. 21 GNU Make Build System: Approach • One make process per build • One makefile per deliverable • Single, complete dependency tree • Using eval/call for declarative description of targets • Target local variables • GObject-like OOP in GNU Make
22. 22 GNU Make Build System: Current Syntax $(eval $(call gb_Library_set_include,sw, … )) call namespace class function instance parameters
23. 23 • All paths are relative to three locations: – SRCDIR – WORKDIR – OUTDIR • One all including makefile • One makefile per module for “compatible builds” • C/C++ is organized around the “link targets” GNU Make Build System: Architecture
24. 24 Header targets depend on everything needed to: • build against the link target • build its objects • generate dependencies GNU Make Build System: Architecture Objects depend on the header targets of the targets: • they are part of • they link against Link targets are the final result (i.e. a library)
25. 25 GNU Make Build System: Building a “Module” • source LinuxX86-64Env.Set.sh • cd $SRC_ROOT/sw • make -srj9 – or build -- -P9 && deliver • make -srj9 clean – or rm -rf unxlngx6.pro && undeliver
26. 26 GNU Make Build System: Files • In module root directory: – Makefile – Module_foo.mk • Library_foo.mk • Executable_foo.mk • Package_foo.mk • … • In the prj directory in the module: – makefile.mk
27. 27 GNU Make Build System: Building All • source LinuxX86-64Env.Set.sh • cd $SRC_ROOT • make -srj9 – Compares to: cd instsetoo_native && build --all -- -P9 && deliver • make -srj9 clean – Compares to: find . -maxdepth 2 -type d -name unxlngx6.pro|xargs rm -rf && rm -rf solver
28. 28 GNU Make Build System: Files • In repository root directory: – GNUMakefile – Module_ooo.mk – SourcePath.mk – Targetnames.mk • In the solenv/gbuild directory: – gbuild.mk for bootstrapping the build system – one file for each type of target (for example: library) – the platform subdirectory with platform dependent content • In the solenv/doc directory: – Files to generate gbuild docs with doxygen
29. 29 sec 10.9 205.9 GNU Make Build System: NoOp Performance Modules: framework, sfx2, svl, svtools, sw, toolkit, tools, xmloff build.pl GNU Make
30. 30 GNU Make Build System: Migration First replace dmake: • Migrate modulewise • Keep build.pl at top level • Dmake just calls GNU Make Then replace build.pl: • Use GNU Make instead • No more two stage build • Single build process • All dependencies in one process
31. 31 Solver! What is it good for? … absolutely nothin'! ;-) (well, we need it to keep compatible with our current packaging process – but that can be changed)
32. 32 GNU Make Build System: Features • Scalability/Parallelization • Lean dependencies • No custom tooling • Full dependencies • Easy to use • Code quality • Multiple repository support
33. <Insert Picture Here> 33 Donald E. Knuth “Beware of bugs in the above code; I have only proved it correct, not tried it.” picture above is copyright CC-BY-SA-2.5 by Jacob Appelbaum
34. 34 GNU Make Build System: Open Tasks • Testing on as much platforms as possible – especially all those linux flavors, MinGW etc. • Options: – optimizing for ccache (separate dep generation) – “hard build order” – dependency generation on linking – mixed debug/non-debug builds • Dependency generation with MSVC • Build Configuration • distcc/ccache • Migration Tooling: makefile generator etc.
35. 35 GNU Make Build System: Join In!
36. 36 Contributors and Thanks • Build Environment Effort Team, especially Ause Lankenau, Heiner Richtien of Hamburg RelEng and Mathias Bauer • Kay Ramme, Thorsten Behrens and Matthias Huetsch for architectural discussions • Jussi Pakkanen and Martin Hollmichel for work on CMake prototyping • the dev@tools.openoffice.org mailing list
39. <Insert Picture Here> 39 Appendix
40. 40 GNU make prototype: example $(eval $(call gb_Library_Library,sw)) $(eval $(call gb_Library_set_include,sw,\ -I$(SRCDIR)/sw/source/core/inc \ -I$(SRCDIR)/sw/source/filter/inc \ -I$(SRCDIR)/sw/source/ui/inc \ -I$(SRCDIR)/sw/inc/pch \ -I$(SRCDIR)/sw/inc \ -I$(WORKDIR)/inc/sw/sdi \ -I$(WORKDIR)/inc/sw/ \ $$(INCLUDE) \ -I$(OUTDIR)/inc/offuh \ -I$(OUTDIR)/inc \ )) $(eval $(call gb_Library_set_defs,sw,\ $$(DEFS) \ -DACCESSIBLE_LAYOUT \ -DSW_DLLIMPLEMENTATION \ ))
41. 41 GNU make prototype: example $(eval $(call gb_Library_add_exception_objects,sw,\ sw/source/core/SwNumberTree/SwNodeNum \ … \ ))
42. 42 GNU make prototype: example