Jenkins World #JenkinsWorld External Build Log Storage. Prototype demo Oleg Nenashev1, Xing Yan2 © 2016 CloudBees, Inc. All Rights Reserved HACKATHON * 1 – Jenkins Project && CloudBees, Inc., 2 - Google, Inc.

Jenkins World #JenkinsWorld Presenter Info Oleg Nenashev • Jenkins core contributor • JAM and Jenkins Online Meetup organizer • Maintainer of plugins • Jenkins Developer at CloudBees, Inc. @oleg_nenashev oleg-nenashev onenashev

Jenkins World #JenkinsWorld Problem statement • All Jenkins agents report logs to master • High disk and network throughput on big instances • Users view Console logs => Master has to load them ÞAdditional load • (+) Logrotate, backups, etc., etc. © 2016 CloudBees, Inc. All Rights Reserved Logging may be a performance bottleneck

Jenkins World #JenkinsWorld Possible solution – Logging to remote logging system • System logs – trivial (log appenders) • Build logs – complex • Custom implementation of the channel • Different run implementation (AbstractProject, Pipeline) • Dependent extension points (Console Annotator) © 2016 CloudBees, Inc. All Rights Reserved

Jenkins World #JenkinsWorld External build logs. Prototyping approach © 2016 CloudBees, Inc. All Rights Reserved Master Node Logstash Elasticsearch Kibana Logs from on- master operations Logs from remote invocations Build Console • No build logs on the disk • No log loading from nodes • Reduced load on master Storage Visualizer

Jenkins World #JenkinsWorld Prototype. Scope of change © 2016 CloudBees, Inc. All Rights Reserved Jenkins Core (2.23-SNAPSHOT) Logstash Plugin (1.2.1-SNAPSHOT) Workflow Job Plugin (2.7-SNAPSHOT) • LoggingMethod abstraction layer • LoggingMethodLocator Extension Point • Hack for Console Annotator markers (no-special-symbols) dep. dep. dep. • Parent POM update • Pipeline Support • LoggingMethod implementation • External log browsers • Logging Method hook

Jenkins World #JenkinsWorld Logging Method Factory © 2016 CloudBees, Inc. All Rights Reserved Fine-grain behavior for runs Default implementation in the core

Jenkins World #JenkinsWorld Logging Method © 2016 CloudBees, Inc. All Rights Reserved Process invocations On-master operations

Jenkins World #JenkinsWorld Prototype. Supported reporters © 2016 CloudBees, Inc. All Rights Reserved 4 indexers (from Logstash plugin) Basic Auth to destination servers

Jenkins World #JenkinsWorld Reporting. AbstractProject-based jobs © 2016 CloudBees, Inc. All Rights Reserved Master Node Logstash Elasticsearch Logs from on- master operations Logs from remote invocations Storage Current state: • Master still saves data to disk (easy to fix) • Logs from nodes go directly to Elasticsearch

Jenkins World #JenkinsWorld Reporting. Pipeline jobs © 2016 CloudBees, Inc. All Rights Reserved Master Node Logstash Elasticsearch Logs from on- master operations Logs from remote invocations Storage Current state: • Master reports data to both disk and Elasticsearch • A change in Durable task is required to redirect logs

Jenkins World #JenkinsWorld Build Log Browsing • Two modes are supported • Mode #1. “Kibana” – Embeds Kibana query layout – Request to Kibana goes from browser => Less load on master – Issue: Kibana with Authentication • Mode #2. “Elasticsearch” – Jenkins queries Elasticsearch and renders results – Pros: o Support of basic auth o Log annotation can be implemented – Cons: Request go through the master server © 2016 CloudBees, Inc. All Rights Reserved

Jenkins World #JenkinsWorld Build Log Browsing. Kibana © 2016 CloudBees, Inc. All Rights Reserved Kibana. Want to have

Jenkins World #JenkinsWorld Build Log Browsing. Kibana © 2016 CloudBees, Inc. All Rights Reserved Kibana. Now Garbage info before log

Jenkins World #JenkinsWorld Build Log Browsing. Elasticsearch queries © 2016 CloudBees, Inc. All Rights Reserved

Jenkins World #JenkinsWorld Next steps 1. Cleanup leftover tasks –https://docs.google.com/document/d/1hXBj2QIMQZcXuAPHm hvH8B1ynNmRih8vrPoGchu6N_8/edit#heading=h.dd7uzhbqscs6 2. Create Jenkins JIRA EPIC for further steps 3. Write design for the mergeable implementation –Better core API (Support of tasks, etc.) –RemoteConsoleAnnotator extension point –Changes in Pipeline to be done –etc. © 2016 CloudBees, Inc. All Rights Reserved

Jenkins World #JenkinsWorld Sources • Logstash-based PoC: –https://github.com/oleg-nenashev/logstash-plugin/tree/jw- hackaton-remote-logging • Core with extensions: –https://github.com/oleg-nenashev/jenkins/tree/jw-hackaton- external-logging • Pipeline Job: –https://github.com/oleg-nenashev/workflow-job- plugin/tree/jw-hackaton-external-logging © 2016 CloudBees, Inc. All Rights Reserved

Jenkins World #JenkinsWorld © 2016 CloudBees, Inc. All Rights Reserved