Managing Dependencies at Build Time

Libraries, service definitions, and sub-projects – oh my! Most projects of any size depend on things from third party or internally built projects. How do we keep track of what's in our builds? When should we rebuild our project? When is source control and Maven/NuGet not enough?

Transcript

1. Build Time Dependencies
2. Build Dependencies Part of a wider dependency challenge Develop  /  Build   Run/me   Infrastructure   source   3rd  Party  libs   Internal  Libs   Config   templates   VM  Images   Environment   Templates   Deployment   Manifests   Deployable   Builds   Environment   Manifests   Sub-­‐Projects   Middleware   “Code”   Build  Tools  
3. Today’s Focus §  Libraries §  Sub-Projects §  Interface Definitions Reusing compiled code
4. The plan for today §  The challenge §  Basics of a dependency solution §  Best practices for CI at scale §  Managing 3rd party dependencies 4
5. The plan for today §  The challenge §  Basics of a dependency solution §  Best practices for CI at scale §  Managing 3rd party dependencies 5
6. Why not source level reuse? 6 Release / Reuse Equivalency
7. Why not source level reuse? §  Changing Project A breaks Project B 7 Release / Reuse Equivalency
8. Why not source level reuse? 8 Release / Reuse Equivalency
9. Why not source level reuse? 9 Release / Reuse Equivalency §  A bug fix in Project A never gets to Project B §  New features have to be done repeatedly
10. Why not source level reuse? 10 Release / Reuse Equivalency
11. Why not source level reuse? §  Use versioned copies of C -  Bugs found in C can be tracked and fixed 11 Release / Reuse Equivalency hHp://www.urbancode.com/html/resources/arLcles/reuse-­‐maturity-­‐model.html   “The  granule  of  reuse  is  the  granule   of  release.  Only  components  that   are  released  through  a  tracking   system  can  be  effec:vely  reused.”     -­‐  Robert  C.  MarLn’s  C++  Report   1997    
12. Dependency Graph Represent relationship visually A B C
13. Dependency graphs 13 Complex, rarely clean, and important
14. The Challenge §  Successfully build the app §  Easily provide new versions to dependents §  Rebuild when my dependency changes §  Get a BOM §  Manage 3rd party libs 14
15. The plan for today 15 §  The challenge §  Basics of a dependency solution §  Best practices for CI at scale §  Managing 3rd party dependencies
16. Elements of a solution 1.  A description of our dependencies (rules) 2.  A repository of versioned components 3.  Automated retrieval at build time 4.  A manifest of what was retrieved 5.  Intelligent build triggering for CI 16
17. Elements of a solution 17 #1 - A description of our dependencies (rules) Project?     Version?     What  if  something  goes   wrong?     Subset  of  files?    
18. Elements of a solution §  Authoritative Source §  Tamper Resistant §  Access Control §  Release Meta-Data §  Retention Policies 18 #2 - A repository of versioned components
19. Elements of a solution §  Authoritative Source §  Tamper Resistant §  Access Control §  Release Meta-Data §  Retention Policies 19 #2 - A repository of versioned components Examples:   •     Maven  Repos:   •   (ArLfactory,  Nexus…)   •     CodeStaLon   •     NuGet   •     Ivy    
20. Repository Failure Patterns Binary dependencies are versioned with the source code 20 It’s in the “lib” directory
21. Repository Failure Patterns Using a public repository Don’t mind me, I’m just waiting for the whole internet to download…
22. Elements of a solution §  Pulling the files down §  Support various build types -  On the developer’s desktop -  In the authoritative build environment 7/30/13 UrbanCode  Inc.  Proprietary  and  ConfidenLal  ©2012   22 #3 - Automated retrieval at build time
23. Elements of a solution What dependencies at what version 23 #4 - A manifest of what was retrieved hHp://mvnrepository.com/arLfact/com.sun.jersey/jersey-­‐bundle/1.16  
24. Elements of a solution …Harder than it looks 24 #5 - Intelligent build triggering for CI
25. 5 Elements of a solution 1.  A description of our dependencies (rules) 2.  A repository of versioned components 3.  Automated retrieval at build time 4.  A manifest of what was retrieved 5.  Intelligent build triggering for CI 25
26. The plan for today 26 §  The challenge §  Basics of a dependency solution §  Best practices for CI at scale §  Managing 3rd party dependencies
27. Best Practices for CI at Scale §  Early in dev cycle rules should point to “latest” -  Bill of Materials must still know what the latest is §  When nearing release, lock versions §  In maintenance: sub-projects are branched as necessary using main project numbering #1 – Dynamic rules early, fixed later
28. Best Practices for CI at Scale Push builds from the bottom of the graph A B C
29. Best Practices for CI at Scale §  Use multiple build machines §  Independent builds run in parallel Distribute the work
30. Best Practices for CI at Scale §  Big graphs can take a while to build -  Introduce race conditions as new changes come in §  Consistent sets: -  Fetch code from same date / time -  Fetch from a snapshot / label / baseline Use a consistent set of code
31. The plan for today §  The challenge §  Basics of a dependency solution §  Best practices for CI at scale §  Managing 3rd party dependencies 31
32. Managing 3rd party libraries §  Do we really need another XML parser? -  Be suspicious of new libraries §  Don’t reference external repositories §  Is that the real version? -  Require some seniority to load a new versions §  Lifecycle libraries -  Flag them as under test, approved, or deprecated as your repo allows General concerns
33. Managing 3rd party libraries §  Some open source licenses are risky §  Involve legal without wishing you didn’t -  Get approval for (versioned) licenses, not specific libs -  Make approved & rejected licenses well known Open source and licensing No,  I’m  not  a  lawyer   㽈
34. Managing 3rd party libraries §  Compliance with licensing agreements: tricky §  Build a reverse BOM -  What is this library used by? -  Consider restricting permissions to it Commercial licensing No,  I’m  not  a  lawyer  
35. Key take-aways §  Build time dependencies require -  Having the files -  Knowing the dependency rules -  Resolving those dependency rules -  Recording a bill of materials §  CI exposes subtle challenges with dependencies §  3rd Party libraries require special care
36. Reference Material Urbancode.com/resources § White Papers -  Enterprise CD Maturity Model -  Lean Build & Deployment Automation -  Continuous Integration vs Build Management § Stay in touch: Blogs.urbancode.com Twitter.com/UrbanCodeSoft Twitter.com/EricMinick Slideshare.net/Urbancode
37. Yes, we’re a products company § uBuild -  Build automation and CI that scales -  Integrated dependency repository -  Integrated dependency definitions § uDeploy -  Deployment and release management § uRelease -  Release & Environment management / planning
38. Q&A Contact Eric Minick Eric@Urbancode.com @UrbanCodeSoft @EricMinick www.urbancode.com!