Comment Terradue utilise Jenkins et Artifactory pour automatiser le déploiement d’applications en sciences de la Terre

Dans cette présentation, Emmanuel présente comment Terradue utilise Jenkins avec Artifactory pour simplifier la construction, le packaging et le déploiement d’applications scientifiques depuis leur intégration dans une Sandbox jusqu’à leur exploitation en cluster évolutif en utilisant Docker. En prenant un exemple d’application développée dans le cadre d’un projet d’environnement de recherche virtuel de la commission européenne EVER-EST, il s’agira de présenter l’utilisation des plugins Jenkins tels que ‘pipeline’, ‘organisations’, ‘docker’ et de publication Artifactory. La présentation démontrera finalement comment tous ces outils permettent ensuite un déploiement et une exploitation de ces applications scientifiques sur des ressources Cloud hétérogènes.

Transcript

1. Jenkins et artifactory automatise le déploiement d'applications en science de la terre à Terradue 11 juillet 2017 / Paris Emmanuel Mathot Research and Innovation
3. User support Catalogue Production center Metadata Harvesters Data Hoovers Coordinators Workflows Cloud Platform Sandboxes Marketplace
12. Cloud Platform Integration layers: ▪  Connecting scientists to the Web and Cloud resources empowering their research ▪  Tailored for Earth Sciences data integration & analysis Partnership mindset: ▪  Dedicated entry points for different types of value adders Since 2013, we’ve supported more than 40 companies and research institutes in accessing earth observation data & deploying processing services
13. Cloud Platform A workplace & APIs to connect Data Providers, Developers, Integrators and information producers Mission: ▶  Rapid prototyping and benchmarking of algorithms ▶  Seamless data access whatever stage of the process ▶  Automated data processing ▶  Scalable resource management ▶  Information conveyance
14. Assumptions: ü  Computer systems are tools ü  Software as a result of years of research (legacy) ü  Earth scientist ≠ geek computer scientist ü  Goals : Research, monitoring of trends, publication Earth Science
15. Build Development Integrate/Test Package Dockerize Code Commit Scan Repository Production Continuous integration/Deployment Code Upload Application Release
16. Development Integrate/Test Code Commit Code Upload
18. Welcome on board demo1! [demo1@sb-10-15-36-38 ~]$ sudo yum install -y snap Loaded plugins: fastestmirror, presto Setting up Install Process Loading mirror speeds from cached hostfile epel/metalink | 25 kB 00:00 atomic | 3.4 kB 00:00 base | 3.7 kB 00:00 ciop-stable | 1.4 kB 00:00 cloudera-cdh3 | 951 B 00:00 epel | 4.3 kB 00:00 extras | 3.4 kB 00:00 updates | 3.4 kB 00:00 updates/primary_db | 2.0 MB 00:00 Resolving Dependencies --> Running transaction check ---> Package snap.x86_64 0:5.0-1.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================= Installing: snap x86_64 5.0-1.el6 ciop-stable 475 M Transaction Summary ============================================================================================================================================= Install 1 Package(s) Total size: 475 M Installed size: 476 M Downloading Packages: Transaction Test Succeeded Running Transaction Installing : snap-5.0-1.el6.x86_64 [#######################################################################################################] Installed: snap.x86_64 0:5.0-1.el6 Complete!
19. [demo1@sb-10-15-36-38 ~]$ wget http://download.osgeo.org/gdal/CURRENT/gdal-2.2.1.tar.gz --2017-06-30 16:50:30-- http://download.osgeo.org/gdal/CURRENT/gdal-2.2.1.tar.gz Resolving download.osgeo.org... 140.211.15.132 Connecting to download.osgeo.org|140.211.15.132|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 13724990 (13M) [application/x-gzip] Saving to: “gdal-2.2.1.tar.gz” 100%[========================================================================================================>] 13,724,990 1.13M/s in 13s 2017-06-30 16:50:43 (1.02 MB/s) - “gdal-2.2.1.tar.gz” saved [13724990/13724990] [demo1@sb-10-15-36-38 gdal-2.2.1]$ make (cd port; make) make[1]: Entering directory `/home/demo1/gdal-2.2.1/port' /bin/sh /home/demo1/gdal-2.2.1/libtool --mode=compile --tag=CXX g++ -I/home/demo1/gdal-2.2.1/port -I/home/demo1/gdal-2.2.1/gcore -I/home/demo1/ gdal-2.2.1/alg -I/home/demo1/gdal-2.2.1/ogr -I/home/demo1/gdal-2.2.1/ogr/ogrsf_frmts -I/home/demo1/gdal-2.2.1/gnm -I/home/demo1/gdal-2.2.1/apps - DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2 -Wall -Wextra -Winit-self -Wunused-parameter -Wformat - Werror=format-security -Wno-format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno- operator-names -DGNM_ENABLED -I/home/demo1/gdal-2.2.1/port -DGDAL_COMPILATION -I../frmts/zlib -DINST_DATA=\"/usr/local/share/gdal\" - DHAVE_LIBZ -DUSE_ONLY_CRYPTODLL_ALG -c -o cpl_conv.lo cpl_conv.cpp libtool: compile: g++ -I/home/demo1/gdal-2.2.1/port -I/home/demo1/gdal-2.2.1/gcore -I/home/demo1/gdal-2.2.1/alg -I/home/demo1/gdal-2.2.1/ogr -I/ home/demo1/gdal-2.2.1/ogr/ogrsf_frmts -I/home/demo1/gdal-2.2.1/gnm -I/home/demo1/gdal-2.2.1/apps -DHAVE_AVX_AT_COMPILE_TIME - DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2 -Wall -Wextra -Winit-self -Wunused-parameter -Wformat -Werror=format-security -Wno- format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno-operator-names -DGNM_ENABLED -I/home/demo1/gdal-2.2.1/port -DGDAL_COMPILATION -I../frmts/zlib -DINST_DATA=\"/usr/local/share/gdal\" -DHAVE_LIBZ -DUSE_ONLY_CRYPTODLL_ALG -c cpl_conv.cpp -fPIC -DPIC -o .libs/cpl_conv.o libtool: compile: g++ -I/home/demo1/gdal-2.2.1/port -I/home/demo1/gdal-2.2.1/gcore -I/home/demo1/gdal-2.2.1/alg -I/home/demo1/gdal-2.2.1/ogr -I/ home/demo1/gdal-2.2.1/ogr/ogrsf_frmts -I/home/demo1/gdal-2.2.1/gnm -I/home/demo1/gdal-2.2.1/apps -DHAVE_AVX_AT_COMPILE_TIME - DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2 -Wall -Wextra -Winit-self -Wunused-parameter -Wformat -Werror=format-security -Wno- format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno-operator-names -DGNM_ENABLED -I/home/demo1/gdal-2.2.1/port -DGDAL_COMPILATION -I../frmts/zlib -DINST_DATA=\"/usr/local/share/gdal\" -DHAVE_LIBZ -DUSE_ONLY_CRYPTODLL_ALG -c cpl_conv.cpp -o cpl_conv.o >/dev/null 2>&1 /bin/sh /home/demo1/gdal-2.2.1/libtool --mode=compile --tag=CXX g++ -I/home/demo1/gdal-2.2.1/port -I/home/demo1/gdal-2.2.1/gcore -I/home/demo1/ gdal-2.2.1/alg -I/home/demo1/gdal-2.2.1/ogr -I/home/demo1/gdal-2.2.1/ogr/ogrsf_frmts -I/home/demo1/gdal-2.2.1/gnm -I/home/demo1/gdal-2.2.1/apps - DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2 -Wall -Wextra -Winit-self -Wunused-parameter -Wformat - Werror=format-security -Wno-format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno- operator-names -DGNM_ENABLED -I/home/demo1/gdal-2.2.1/port -DGDAL_COMPILATION -I../frmts/zlib -DINST_DATA=\"/usr/local/share/gdal\" - DHAVE_LIBZ -DUSE_ONLY_CRYPTODLL_ALG -c -o cpl_error.lo cpl_error.cpp libtool: compile: g++ -I/home/demo1/gdal-2.2.1/port -I/home/demo1/gdal-2.2.1/gcore -I/home/demo1/gdal-2.2.1/alg -I/home/demo1/gdal-2.2.1/ogr -I/ home/demo1/gdal-2.2.1/ogr/ogrsf_frmts -I/home/demo1/gdal-2.2.1/gnm -I/home/demo1/gdal-2.2.1/apps -DHAVE_AVX_AT_COMPILE_TIME - DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2 -Wall -Wextra -Winit-self -Wunused-parameter -Wformat -Werror=format-security -Wno- format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno-operator-names -DGNM_ENABLED -I/home/demo1/gdal-2.2.1/port -DGDAL_COMPILATION -I../frmts/zlib -DINST_DATA=\"/usr/local/share/gdal\" -DHAVE_LIBZ -DUSE_ONLY_CRYPTODLL_ALG -c cpl_error.cpp -fPIC -DPIC -o .libs/cpl_error.o
20. [demo1@sb-10-15-36-37 dcs-sencor-demo1]$ tree . !"" Jenkinsfile !"" pom.xml !"" README.md !"" src # !"" main # # %"" app-resources # # !"" application.xml # # !"" dependencies # # # %"" python # # # %"" packages.list # # !"" etc # # # !"" pa.bbox # # # %"" pa.projwin # # !"" query # # # !"" lib # # # # %"" functions.sh # # # %"" run # # %"" sen2cor # # !"" bin # # # %"" gdalcopyproj.py # # !"" etc # # # !"" eop-template.xml # # # %"" md-template.xml # # !"" lib # # # %"" functions.sh # # %"" run # %"" test # !"" __pycache__ # # %"" test_util.cpython-27-PYTEST.pyc # !"" run_tests.sh # %"" test_util.py %"" target %"" antrun %"" build-main.xml 16 directories, 18 files
22. Build Development Integrate/Test Code Commit Scan Repository Code Upload
25. Build Development Integrate/Test Package Dockerize Code Commit Scan Repository Code Upload Application Release
26. def artserver = Artifactory.server('store.terradue.com') 
 def buildInfo = Artifactory.newBuildInfo() 
 buildInfo.env.capture = true 
 
 pipeline { 
 
 parameters{ 
 // TODO: release parameter for end user 
 booleanParam(name: 'RELEASE', defaultValue: false, description: 'If this parameter is set, the build will publish the processor as a release ready for production.', ) 
 } 
 
 options { 
 // Keep 5 builds history 
 buildDiscarder(logRotator(numToKeepStr: '5')) 
 } 
 
 agent {
 node {
 // docker community builder
 label 'ci-community-docker’
 }
 }
 
 stages {
 // Let's go!
 stage(’Build, Package & Dockerize') {
 steps {
 withMaven(
 // Maven installation declared in the Jenkins "Global Tool Configuration”
 maven: 'apache-maven-3.0.5' ) {
 sh 'mvn -B deploy’
 }
 script {
 artserver.publishBuildInfo buildInfo
 }
 }
 }
 }
 }
33. Build Development Integrate/Test Package Dockerize Code Commit Scan Repository Production Code Upload Application Release
36. Build Development Integrate/Test Package Dockerize Code Commit Scan Repository Production Code Upload Application Release
37. Dedicated VM CentOS 6 12 CPUs 48G RAM 2.60 Pipeline Ar?factory Maven Pipeline Docker Pipeline Blue Ocean Mul?branch Pipeline build Jenkins … store Dedicated Server CentOS 7 12 CPUs 64G RAM
40. * hLps://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
41. ü  Non computer scientist integrating software in CI pipeline ü  Fully automated workflow from code commit to docker push ü  30% operations time gained Recap
43. Thank you! 11 juillet 2017 / Paris @emmanuelmathot Emmanuel Mathot