Orchestrating Jenkins Pipelines With Jenkins and Python API - Jenkins User Conference - Santa Clara

Do you have complex pipeline scripts that you run for your continuous integration (CI)? Do you want to access Jenkins from those scripts? What about searching and accessing the artifacts? Or would you like to block a job until something has finished correctly? Interact with slaves? Would you like to have some kind of synchronization between your jobs and pipeline scripts and yet keep the your job configurations as simple as possible? This talk is about you getting more programmatic control of your Jenkins instance from your pipeline scripts using the Python API. We will look at the power of the API by showing working code examples and demoing the results. This talk will be very friendly to Jenkins beginners and intermediates alike. We will walk through the concepts and actual code during the presentation.

Transcript

1. Orchestrating Pipelines With Jenkins Python API Pradeepto K Bhattacharya Santa Clara, CA, September 2-3
2. #jenkinsconf Footer #whoami Pradeepto Bhattacharya pradeeptob@gmail.com @pradeepto
3. #jenkinsconf Footer Agenda • Why? • Installation • Library Modules • Code Examples • Documentation • Q & A
4. #jenkinsconf Motivation 4
5. #jenkinsconf Footer Why? • Fine Grained Control • Complicated Pipelines • Jenkins REST API • Python Wrapper for the Jenkins Rest API https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API http://yourjenkinsserver/api/
6. #jenkinsconf Footer Words • Create Jobs • Invoke Jobs • Monitor and query Builds/Jobs/Nodes • Search and manipulate artifacts • Create Views
7. #jenkinsconf Footer Conversations • Scripts talking to one or more jobs • Taking actions based on conditions
8. #jenkinsconf Installation 8
9. #jenkinsconf Footer Installation • pip install jenkinsapi • easy_install jenkinsapi • sudo apt-get install python-jenkinsapi Note : There are other similar libraries. I found this one best for all my use cases. It definitely was the most exhaustive. Note : Tested with Python 2.6, 2.7, 3.3
10. #jenkinsconf Modules And Code 10
11. #jenkinsconf Footer jenkinsapi Modules • Jenkins : The Jenkins instance • Job(s) : Represents Jenkins jobs. • User API : Consists of helpful high-level functions • Build : This module encapsulates the single run of a Jenkins job • Artifact : Artifacts are created by Jenkins build. This module encapsulates that. • Node : Encapsulates Jenkins Node
12. #jenkinsconf Footer Connect To Jenkins Instance >>> from jenkinsapi.jenkins import Jenkins >>> jenkins = Jenkins('http://10.10.20.200:8080') >>> jenkins <jenkinsapi.jenkins.Jenkins Jenkins server at http://10.10.20.200:8080> >>> print jenkins Jenkins server at http://10.10.20.200:8080 >>> type(jenkins) <class 'jenkinsapi.jenkins.Jenkins'>
13. #jenkinsconf Footer Simple Helper >>> def get_jenkins_instance(): ... return Jenkins('http://10.10.20.200:8080') ... >>> jenkins = get_jenkins_instance()
14. #jenkinsconf Job 14
15. #jenkinsconf Footer Fetching Jobs >>> jenkins.keys() ['Long Running Job', 'Simple Job'] >>> jobs = jenkins.get_jobs() >>> jobs <generator object get_jobs at 0x7f7eea8a51e0> >>> for job in jobs: ... print job ... ('Long Running Job', <jenkinsapi.job.Job Long Running Job>) ('Simple Job', <jenkinsapi.job.Job Simple Job>)
16. #jenkinsconf Footer Status Reports >>> job = jenkins.get_job('Simple Job') >>> job <jenkinsapi.job.Job Simple Job> >>> job.is_enabled() True >>> job.is_running() False
17. #jenkinsconf Footer Basic Build Information For A Job >>> job.get_last_stable_build() <jenkinsapi.build.Build Simple Job #53> >>> job.get_last_stable_buildnumber() 53 >>> job.get_last_failed_buildnumber() 49 >>> job.get_last_completed_buildnumber() 53
18. #jenkinsconf Footer Invoke A Job * >>> job.invoke() <jenkinsapi.invocation.Invocation object at 0x7f7eea858690> >>> job.is_running() True >>> # Waiting until the job execution is over. … >>> job.is_running() False
19. #jenkinsconf Footer Invoke With Parameters >>> job = jenkins.get_job('Parameterized Build') >>> jobs.get_params() <generator object get_params at 0x7f7eea8ffb90> >>> jobs.get_params_list() ['SOMESTRING', 'RELEASEBUILD'] >>> params = {'SOMESTRING' : 'Value', 'RELEASEBUILD' : True} >>> build_params = {'SOMESTRING' : 'Value', 'RELEASEBUILD' : True} >>> job.invoke(build_params=build_params) <jenkinsapi.invocation.Invocation object at 0x7f7eeda227d0>
20. #jenkinsconf Footer Enable / Disable Job >>> job.disable() <Response [200]> >>> job.is_enabled() False >>> job.enable() <Response [200]> >>> job.is_enabled() True
21. #jenkinsconf Footer SCM Information >>> job.get_scm_type() 'git' >>> job.get_scm_branch() ['*/master'] >>> job.get_scm_url() ['https://github.com/pradeepto/mesos-cluster-marathon']
22. #jenkinsconf Build 22
23. #jenkinsconf Footer Basic Build Information >>> build = job.get_build(71) >>> build <jenkinsapi.build.Build Simple Job #71> >>> build.is_running() False >>> build.get_revision() 'b47c0dc1b0b7ec15077ae840e01f74702b07b0ee' >>> build.name 'Simple Job #71' >>> build.get_status() 'SUCCESS'
24. #jenkinsconf Footer More Build Information >>> job.invoke() <jenkinsapi.invocation.Invocation object at 0x7f7eea858fd0> >>> job = jenkins.get_job('Simple Job') >>> build = job.get_build(77) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/dist-packages/jenkinsapi/job.py", line 353, in get_build url = self.get_build_dict()[buildnumber] KeyError: 77 >>> job = jenkins.get_job('Simple Job') >>> build = job.get_build(77) >>> build.is_running() True
25. #jenkinsconf Footer Console Log >>> build.get_console() 'Started by user anonymous\nBuilding in workspace /var/lib/jenkins/jobs/Simple Job/workspace\n[WS- CLEANUP] Deleting project workspace...\n[WS-CLEANUP] Done\nCloning the remote Git repository\nCloning repository https://github.com/pradeepto/mesos-cluster-marathon\n > git init /var/lib/jenkins/jobs/Simple Job/workspace # timeout=10\nFetching upstream changes from https://github.com/pradeepto/mesos-cluster- marathon\n > git --version # timeout=10\n > git -c core.askpass=true fetch --tags --progress https://github.com/pradeepto/mesos-cluster-marathon +refs/heads/*:refs/remotes/origin/*\n > git config remote.origin.url https://github.com/pradeepto/mesos-cluster-marathon # timeout=10\n > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10\n > git config remote.origin.url https://github.com/pradeepto/mesos-cluster-marathon # timeout=10\nFetching upstream changes from https://github.com/pradeepto/mesos-cluster-marathon\n > git -c core.askpass=true fetch --tags --progress https://github.com/pradeepto/mesos-cluster-marathon +refs/heads/*:refs/remotes/origin/*\n > git rev-parse refs/remotes/origin/master^{commit} # timeout=10\n > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10\nChecking out Revision b47c0dc1b0b7ec15077ae840e01f74702b07b0ee (refs/remotes/origin/master)\n > git config core.sparsecheckout # timeout=10\n > git checkout -f b47c0dc1b0b7ec15077ae840e01f74702b07b0ee\n > git rev-list b47c0dc1b0b7ec15077ae840e01f74702b07b0ee # timeout=10\n[workspace] $ /bin/sh -xe /tmp/hudson7335543422446213294.sh\n+ echo Running Simple Job\nRunning Simple Job\n+ sleep 10\n+ echo Done.\nDone.\nFinished: SUCCESS\n'
26. #jenkinsconf Artifacts 26
27. #jenkinsconf Footer Fetching Artifacts >>> build = job.get_build(81) >>> build.get_artifact_dict() {'artifact.txt': <jenkinsapi.artifact.Artifact http://10.10.20.200:8080/job/Simple%20Job/81/artifact/artifact.tx t>} >>> build.get_artifact_dict()['artifact.txt'] <jenkinsapi.artifact.Artifact http://10.10.20.200:8080/job/Simple%20Job/81/artifact/artifact.tx t> >>> build.get_artifact_dict()['artifact.txt'].get_data() 'This is an artifact\n'
28. #jenkinsconf Footer Artifacts Information artifacts = build.get_artifact_dict() >>> a1 = artifacts['artifact.txt'] >>> a1.build <jenkinsapi.build.Build Simple Job #81> >>> a1.filename 'artifact.txt' >>> a1.url 'http://10.10.20.200:8080/job/Simple%20Job/81/artifact/artifact.t xt'
29. #jenkinsconf User API 29
30. #jenkinsconf Footer Convenience >>> jenkins_url = 'http://10.10.20.100:8080' >>> from jenkinsapi.api import * >>> get_latest_build(jenkins_url, 'Simple Job') <jenkinsapi.build.Build Simple Job #81> >>> get_latest_complete_build(jenkins_url, 'Beefy Job') <jenkinsapi.build.Build Beefy Job #81> >>> get_artifacts(jenkins_url,'Simple Job', 81) {'artifact.txt': <jenkinsapi.artifact.Artifact http://10.10.20.200:8080/job/Simple%20Job/81/artifact/artifact.tx t>}
31. #jenkinsconf Node 31
32. #jenkinsconf Footer Query Nodes >>> for node in jenkins.get_nodes().keys(): ... print node ... master debian-1 windows-1 android-1 >>> master = jenkins.get_nodes()['master'] >>> master.is_online() True
33. #jenkinsconf Plugins 33
34. #jenkinsconf Footer Plugins >>> plugins = jenkins.get_plugins() >>> plugins <jenkinsapi.plugins.Plugins ['ant', 'antisamy-markup-formatter', 'credentials', 'cvs', 'external-monitor-job', 'git', 'git-client', 'git-parameter', 'javadoc', 'junit', 'ldap', 'mailer', 'matrix-auth', 'matrix-project', 'maven-plugin', 'pam-auth', 'scm-api', 'script- security', 'ssh-credentials', 'ssh-slaves', 'subversion', 'translation', 'windows-slaves', 'ws-cleanup']> >>> jenkins.get_plugins().keys() ['git-client', 'matrix-auth', 'maven-plugin', 'javadoc', 'external- monitor-job', 'git-parameter', 'ant', 'ssh-slaves', 'pam-auth', 'windows-slaves', 'git', 'scm-api', 'subversion', 'antisamy-markup- formatter', 'ldap', 'junit', 'mailer', 'credentials', 'translation', 'ssh-credentials', 'ws-cleanup', 'matrix-project', 'cvs', 'script- security']
35. #jenkinsconf Exceptions 35
36. #jenkinsconf Footer Using Exceptions • This library comes with bunch of built-in exceptions. Highly recommend that you use them in your Python scripts. JenkinsAPIException, UnknownJob, UnknownView, UnknownNode, UnknownPlugin, NoBuildData, NoResults, NotConfiguredSCM
37. #jenkinsconf Footer Documentation And Contributions • Few modules are documented. • Few examples. • Great opportunity to contribute to this wonderful library. • https://github.com/salimfadhley/jenkinsapi
38. #jenkinsconf Footer Other Libraries • Python Jenkins (https://pypi.python.org/pypi/python-jenkins/) • Jenkins API Client (Ruby) (https://rubygems.org/gems/jenkins_api_client)
39. #jenkinsconf Footer One more version, less space Questions? pradeeptob@gmail.com
40. #jenkinsconf Footer Coupon Code “cfjucuswe”
41. #jenkinsconf Footer Thanks • CloudBees and organisers of JUC 2015 • Contributors of Jenkins and its ecosystem • Girija and Rupali (my partners in CI related crimes)