JUC Paris 2012 — Jenkins @ Nuxeo

Jenkins User Conference Paris 2012
Jenkins at Nuxeo: an history of continuous integration.

Transcript

1. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf Julien Carsique, DevOps https://qa.nuxeo.org/jenkins
2. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 2/25 The Nuxeo company — www.nuxeo.com • Founded in 2000 • ~ 50 employees (mainly France & USA) • Open Source Enterprise Content Management • No-charge software licensing with LGPL • Nuxeo Connect subscription offering for support, maintenance, consulting, training and services: • Nuxeo Studio • Nuxeo Marketplace • Nuxeo Cloud
3. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 3/25 The Nuxeo products Document and Case Management
4. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 4/25 The Nuxeo products Digital Assets Management
5. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 5/25 The Nuxeo products Web sites
6. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 6/25 The Nuxeo products Public facing applications, RIA and much more …
7. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 7/25 The Nuxeo products Open Source ECM platform, applications and services • Nuxeo Platform • Java and standards based • Component based architecture • Extensible and configurable ECM services • UI building blocks and High level frameworks • A lot of integration levels and APIs • Testing, benchmarking and packaging tools • Development tools (Nuxeo Studio, Nuxeo IDE) • Ready to use pre-packaged applications
8. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 8/25 The challenge of The source code • 200+ SubVersion, Mercurial and Git repositories • 1M+ LoC, 500+ Maven modules • 5 application modules and 60+ addons • 500+ components in the Platform • 2500+ unit tests, 2000+ Selenium and WebDriver tests • 10+ languages (Java, XML, JavaScript, HTML, CSS, Python, Shell, SQL, Groovy, …) and tools (Maven, Ant, Shell, …) • 30~50 active committers and contributors • ~1000 commits/month • 1 development branch, 3~5 maintenance branches
9. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 9/25 The challenge of The human constraints • Follow and maintain quality • Save developers time • Doing more checks than they can • Providing a safety net for their changes • No need to build everything themselves • Save testers time • Automate tests in various environments • Warn on regression • Easy to download latest deliverables • Save integrators time • Work on Nuxeo development branch with more confidence • Get guarantees on code quality
10. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 10/25 The challenge of The deployment constraints • Multiple deployment target • WAR, JEE or bare Java • OS (Linux, Unix, Windows, MacOS) • DB (PostgreSQL, Oracle, SQLServer, MySQL, …) • BlobStore backend (FS, NAS, S3, SQLDB, ...) • Scalability and performance • Cluster support • Multi-DB scale-out (several repositories and/or DB) • Raw performances (100M docs benchmark on a single server; 200 pages/s output on complex business apps) • Easy maintenance • Backup, restore, upgrade • export/import
11. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 11/25 The history of The beginning • Before 2007: BuildBot • 2007 — Nuxeo 5 switch to Java • Trying CruiseControl, TeamCity, Continuum, … Hudson! • Goals for Nuxeo 5.1 • Code analysis (Quality Assurance) • Build and unit test Nuxeo against default environment – Linux, JBoss, Sun JDK 5, H2 – Smoke test at JBoss start – Run Selenium suite • Automate Nuxeo release process • Automate generation and test of documentation • Involve developers
12. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 12/25 The history of Nuxeo 5.1 — CI architecture UPLOAD DOWNLOAD RELEASES RELEASES UPLOAD RELEASES & SNAPSHOTS RELEASES and SNAPSHOTS from outside RELEASES Public Internal Nexus Private Internal Nexus Nexus Preprod Public Jenkins Master Private Jenkins Master UPLOAD DOWNLOAD UPLOAD DOWNLOAD Maven.nuxeo.org
13. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 13/25 The history of Nuxeo 5.1 — CI architecture and usage • Speed up build time • Organize teams resources • Ensure publicly available artifacts quality • Isolate internal and external build chains • Enforce rules for developers • Early use of CI • Think about what kind of build (Maven, Shell, Ant, …) and what must be checked for validation (unit tests, errors in logs, …)? • Automated build: nothing specific or contextual to the slave • Make the build self-testing • Stay tuned, be aware of CI activity • Never let a project in an unstable status
14. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 14/25 The history of Nuxeo 5.1 to 5.3 — making heavy use of CI • Release process improvement • Automated tasks • Nightly / On demand • Build promotion • More and more tests and environments • More performance and functional tests • Tests with Jetty, JBoss and Tomcat • Matrix testing jobs with OS and databases • Testing development and administration tools • Seriously increasing number of builds • New packages and installers (Debian, VM, Windows, Multi-OS, …) • New applications (Android, Nuxeo CMF, Nuxeo DAM, …) • New plugins for Firefox, IE, OpenOffice/LibreOffice • Accumulating versions to maintain
15. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 15/25 The history of Nuxeo 5.3 to 5.4 (2010-2011) — growing CI issues • CI performance issues • CPU, network and disk usage • Slave usage strategy • Full cycle duration • CI usage issues • Build waiting time • Feedback delay • Feedback accuracy • Jobs' dependencies
16. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 16/25 The history of Nuxeo 5.4 (2011) — CI improvements, Jenkins analysis • Build cycles, workflows and durations • Use of Jenkviz (homemade tool), Dependency Graph View and (enhanced) Build Timeout plugins
17. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 17/25 The history of Nuxeo 5.4 (2011) — CI improvements, solutions • Work on source code • GWT optimizations (reduce build time) • Review testing suites, prefer use of WebDriver over Selenium • Work on CI • Improve slaves usage (using labels), local vs remote • Give priority to jobs with low duration (earlier feedback) • New short and permanent cycle • Create independent jobs for specific purposes • Create new views to help developers • Add CPU and/or memory on some slaves • Improve Nexus infrastructure (Maven build time) • Fix jobs triggering issue
18. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 18/25 The history of Nuxeo 5.5 — the current status
19. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 19/25 The history of Nuxeo 5.5 — the current status • Great coverage and good delays • 4 kinds of CI cycles per Nuxeo version maintained – Triggered by commit (300 jobs) – Full build and tests per day (10 jobs) – Independent cycle per 3 hours (11 jobs) – Nightly build (40 jobs) • Less than 10 minutes to 3 hours max for main feedback • Code and applications tested on multi-DB/OS/JDK/servers • 6 hours for the longest build cycle • About 450 jobs on 25 servers (2 masters), 200~?K builds/day • 6 database engines tested on 5 operating systems • 7 applications tested in all environments • 50K~100K tests ran including all products and versions under CI
20. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 20/25 The history of Nuxeo 5.5 — the current status, major CI cycles Commit Mercurial/GitHub Sources checkout Build & Unit tests Job trigger SCM trigger Packaging 7 apps, 2 servers Sources checkout daily trigger Packaging multi-OS Smoke tests Sources checkout 3h trigger Full build (no tests) Packaging all applications Functional tests Permanent cycle Merge « stable » branch (GitHub) Sources checkout Full build (no tests) Prepare release Packaging all applications Merge «5.6-SNAPSHOT» branch (GitHub) Deploy «5.6-SNAPSHOT» Maven artifacts Build installers (Debian, Win, multi-OS, ...) Test and upload installers Performance tests (référence) Perform release Batch task Push release tag (GitHub) Deploy release artifacts Functional tests multi-DB/OS/JDK/servers, cluster Functional tests multi-OS/DB Performance tests multi-DB Various addons functional tests Full build & tests multi-JDK Full build & tests multi-DB Linux Full build & tests multi-DB Win Nightly Manual
21. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 21/25 The history of Nuxeo 5.5 — main Jenkins plugins used • Required • Subversion, Mercurial, MSBuild, GIT, GitHub, Android Emulator • Useful for Jenkins management • Priority Sorter, Parameterized Trigger, Retry Failed Builds, Rebuilder, Hudson Post build task, Matrix Reloaded, next-executions, Bulk Builder, Join, Promoted Builds, batch task • Change Log History, Maven Deployment Linker, SCM Sync Configuration, Multiple SCMs, Static Analysis Utilities, Timestamper, build timeout, Hudson Setenv, java.io.tmpdir cleaner, Hudson global-build-stats, Node and Label parameter, Monitoring, SSH Slaves, Dependency Graph Viewer, Performance, Disk Usage • Useful reports • Checkstyle, FindBugs, Task Scanner, PMD, Warnings, Hudson Seleniumhq, Selenium HTML report, Cobertura, TextFinder, JUnit Attachments, Duplicate Code Scanner • For end-users • Dashboard View, Claim, Radiator View, Hudson iPhoneView, Continuous Integration game, instant-messaging, Jabber notifier, built-on-column, Compact Columns, Version Column, Email-ext, View Job Filters, Extra Columns, Simple Theme
22. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 22/25 The history of Nuxeo 5.6 — the near future • Better quality testing with Sonar • Integration with Jira • Automated merges from QA dedicated branches to the development branch • Automated Jenkins infrastructure deployment on cloud • Custom slaves on demand (Chef / Puppet, chroot / LXC, …) • Work on Nuxeo Cloud & Compatible One R&D • CloudBees • Nuxeo CI (partial or full) migration to the cloud
23. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 23/25 The history of Nuxeo 5.6 — the near future • Jenkins enhancements (plugins) • More abstraction to avoid plugins compliance • Multiple SCM & Git • SCM change log feeder • Matrix jobs improvements • Node Label & Parameterized trigger • Dependency Graph • Infrastructure failures (Check job prerequisites++) • Jobs workflow management • …
24. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf 17/04/2012 24/25 Conclusion • Jenkins • Easy maintenance • Good scalability • Great community and plugins catalog • Powerful extensibility • Efficient contribution process • Cloud capabilities • Nuxeo • CI & QA on Nuxeo Platform frameworks • Stable APIs with backward and forward compatibility • Stable applications with good performances • Automated processes with very few manual interventions
25. Jenkins User Conference Paris 17 April 2012 #jenkinsconf 17/04/2012 25 Thank You To Our Sponsors Platinum Sponsor Silver Sponsor Gold Sponsor