JW2016 Hackathon. External Build Logging. Prototype Demo

This presentation summarizes results of our project from the Jenkins World 2016 hackaton. The prototype implements reporting of build logs to Logstash/Elasticsearch instead of the system disk and their further visualization within the Jenkins Web UI.

The approach allows to reduce network and IO load on Jenkins master and thus avoid performance bottlenecks on large-scale instances.

Transcript

1. Jenkins World #JenkinsWorld External Build Log Storage. Prototype demo Oleg Nenashev1, Xing Yan2 © 2016 CloudBees, Inc. All Rights Reserved HACKATHON * 1 – Jenkins Project && CloudBees, Inc., 2 - Google, Inc.
2. Jenkins World #JenkinsWorld Presenter Info Oleg Nenashev • Jenkins core contributor • JAM and Jenkins Online Meetup organizer • Maintainer of plugins • Jenkins Developer at CloudBees, Inc. @oleg_nenashev oleg-nenashev onenashev
3. Jenkins World #JenkinsWorld Problem statement • All Jenkins agents report logs to master • High disk and network throughput on big instances • Users view Console logs => Master has to load them ÞAdditional load • (+) Logrotate, backups, etc., etc. © 2016 CloudBees, Inc. All Rights Reserved Logging may be a performance bottleneck
4. Jenkins World #JenkinsWorld Possible solution – Logging to remote logging system • System logs – trivial (log appenders) • Build logs – complex • Custom implementation of the channel • Different run implementation (AbstractProject, Pipeline) • Dependent extension points (Console Annotator) © 2016 CloudBees, Inc. All Rights Reserved
5. Jenkins World #JenkinsWorld External build logs. Prototyping approach © 2016 CloudBees, Inc. All Rights Reserved Master Node Logstash Elasticsearch Kibana Logs from on- master operations Logs from remote invocations Build Console • No build logs on the disk • No log loading from nodes • Reduced load on master Storage Visualizer
6. Jenkins World #JenkinsWorld Prototype. Scope of change © 2016 CloudBees, Inc. All Rights Reserved Jenkins Core (2.23-SNAPSHOT) Logstash Plugin (1.2.1-SNAPSHOT) Workflow Job Plugin (2.7-SNAPSHOT) • LoggingMethod abstraction layer • LoggingMethodLocator Extension Point • Hack for Console Annotator markers (no-special-symbols) dep. dep. dep. • Parent POM update • Pipeline Support • LoggingMethod implementation • External log browsers • Logging Method hook
7. Jenkins World #JenkinsWorld Logging Method Factory © 2016 CloudBees, Inc. All Rights Reserved Fine-grain behavior for runs Default implementation in the core
8. Jenkins World #JenkinsWorld Logging Method © 2016 CloudBees, Inc. All Rights Reserved Process invocations On-master operations
9. Jenkins World #JenkinsWorld Prototype. Supported reporters © 2016 CloudBees, Inc. All Rights Reserved 4 indexers (from Logstash plugin) Basic Auth to destination servers
10. Jenkins World #JenkinsWorld Reporting. AbstractProject-based jobs © 2016 CloudBees, Inc. All Rights Reserved Master Node Logstash Elasticsearch Logs from on- master operations Logs from remote invocations Storage Current state: • Master still saves data to disk (easy to fix) • Logs from nodes go directly to Elasticsearch
11. Jenkins World #JenkinsWorld Reporting. Pipeline jobs © 2016 CloudBees, Inc. All Rights Reserved Master Node Logstash Elasticsearch Logs from on- master operations Logs from remote invocations Storage Current state: • Master reports data to both disk and Elasticsearch • A change in Durable task is required to redirect logs
12. Jenkins World #JenkinsWorld Build Log Browsing • Two modes are supported • Mode #1. “Kibana” – Embeds Kibana query layout – Request to Kibana goes from browser => Less load on master – Issue: Kibana with Authentication • Mode #2. “Elasticsearch” – Jenkins queries Elasticsearch and renders results – Pros: o Support of basic auth o Log annotation can be implemented – Cons: Request go through the master server © 2016 CloudBees, Inc. All Rights Reserved
13. Jenkins World #JenkinsWorld Build Log Browsing. Kibana © 2016 CloudBees, Inc. All Rights Reserved Kibana. Want to have
14. Jenkins World #JenkinsWorld Build Log Browsing. Kibana © 2016 CloudBees, Inc. All Rights Reserved Kibana. Now Garbage info before log
15. Jenkins World #JenkinsWorld Build Log Browsing. Elasticsearch queries © 2016 CloudBees, Inc. All Rights Reserved
16. Jenkins World #JenkinsWorld Next steps 1. Cleanup leftover tasks –https://docs.google.com/document/d/1hXBj2QIMQZcXuAPHm hvH8B1ynNmRih8vrPoGchu6N_8/edit#heading=h.dd7uzhbqscs6 2. Create Jenkins JIRA EPIC for further steps 3. Write design for the mergeable implementation –Better core API (Support of tasks, etc.) –RemoteConsoleAnnotator extension point –Changes in Pipeline to be done –etc. © 2016 CloudBees, Inc. All Rights Reserved
17. Jenkins World #JenkinsWorld Sources • Logstash-based PoC: –https://github.com/oleg-nenashev/logstash-plugin/tree/jw- hackaton-remote-logging • Core with extensions: –https://github.com/oleg-nenashev/jenkins/tree/jw-hackaton- external-logging • Pipeline Job: –https://github.com/oleg-nenashev/workflow-job- plugin/tree/jw-hackaton-external-logging © 2016 CloudBees, Inc. All Rights Reserved
18. Jenkins World #JenkinsWorld © 2016 CloudBees, Inc. All Rights Reserved