Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JW2016 Hackathon. External Build Logging. Prototype Demo

568e3391c8b528f2b255443e4cca27ca?s=47 Oleg Nenashev
September 18, 2016

JW2016 Hackathon. External Build Logging. Prototype Demo

This presentation summarizes results of our project from the Jenkins World 2016 hackaton. The prototype implements reporting of build logs to Logstash/Elasticsearch instead of the system disk and their further visualization within the Jenkins Web UI.

The approach allows to reduce network and IO load on Jenkins master and thus avoid performance bottlenecks on large-scale instances.

568e3391c8b528f2b255443e4cca27ca?s=128

Oleg Nenashev

September 18, 2016
Tweet

Transcript

  1. Jenkins World #JenkinsWorld External Build Log Storage. Prototype demo Oleg

    Nenashev1, Xing Yan2 © 2016 CloudBees, Inc. All Rights Reserved HACKATHON * 1 – Jenkins Project && CloudBees, Inc., 2 - Google, Inc.
  2. Jenkins World #JenkinsWorld Presenter Info Oleg Nenashev • Jenkins core

    contributor • JAM and Jenkins Online Meetup organizer • Maintainer of plugins • Jenkins Developer at CloudBees, Inc. @oleg_nenashev oleg-nenashev onenashev
  3. Jenkins World #JenkinsWorld Problem statement • All Jenkins agents report

    logs to master • High disk and network throughput on big instances • Users view Console logs => Master has to load them ÞAdditional load • (+) Logrotate, backups, etc., etc. © 2016 CloudBees, Inc. All Rights Reserved Logging may be a performance bottleneck
  4. Jenkins World #JenkinsWorld Possible solution – Logging to remote logging

    system • System logs – trivial (log appenders) • Build logs – complex • Custom implementation of the channel • Different run implementation (AbstractProject, Pipeline) • Dependent extension points (Console Annotator) © 2016 CloudBees, Inc. All Rights Reserved
  5. Jenkins World #JenkinsWorld External build logs. Prototyping approach © 2016

    CloudBees, Inc. All Rights Reserved Master Node Logstash Elasticsearch Kibana Logs from on- master operations Logs from remote invocations Build Console • No build logs on the disk • No log loading from nodes • Reduced load on master Storage Visualizer
  6. Jenkins World #JenkinsWorld Prototype. Scope of change © 2016 CloudBees,

    Inc. All Rights Reserved Jenkins Core (2.23-SNAPSHOT) Logstash Plugin (1.2.1-SNAPSHOT) Workflow Job Plugin (2.7-SNAPSHOT) • LoggingMethod abstraction layer • LoggingMethodLocator Extension Point • Hack for Console Annotator markers (no-special-symbols) dep. dep. dep. • Parent POM update • Pipeline Support • LoggingMethod implementation • External log browsers • Logging Method hook
  7. Jenkins World #JenkinsWorld Logging Method Factory © 2016 CloudBees, Inc.

    All Rights Reserved Fine-grain behavior for runs Default implementation in the core
  8. Jenkins World #JenkinsWorld Logging Method © 2016 CloudBees, Inc. All

    Rights Reserved Process invocations On-master operations
  9. Jenkins World #JenkinsWorld Prototype. Supported reporters © 2016 CloudBees, Inc.

    All Rights Reserved 4 indexers (from Logstash plugin) Basic Auth to destination servers
  10. Jenkins World #JenkinsWorld Reporting. AbstractProject-based jobs © 2016 CloudBees, Inc.

    All Rights Reserved Master Node Logstash Elasticsearch Logs from on- master operations Logs from remote invocations Storage Current state: • Master still saves data to disk (easy to fix) • Logs from nodes go directly to Elasticsearch
  11. Jenkins World #JenkinsWorld Reporting. Pipeline jobs © 2016 CloudBees, Inc.

    All Rights Reserved Master Node Logstash Elasticsearch Logs from on- master operations Logs from remote invocations Storage Current state: • Master reports data to both disk and Elasticsearch • A change in Durable task is required to redirect logs
  12. Jenkins World #JenkinsWorld Build Log Browsing • Two modes are

    supported • Mode #1. “Kibana” – Embeds Kibana query layout – Request to Kibana goes from browser => Less load on master – Issue: Kibana with Authentication • Mode #2. “Elasticsearch” – Jenkins queries Elasticsearch and renders results – Pros: o Support of basic auth o Log annotation can be implemented – Cons: Request go through the master server © 2016 CloudBees, Inc. All Rights Reserved
  13. Jenkins World #JenkinsWorld Build Log Browsing. Kibana © 2016 CloudBees,

    Inc. All Rights Reserved Kibana. Want to have
  14. Jenkins World #JenkinsWorld Build Log Browsing. Kibana © 2016 CloudBees,

    Inc. All Rights Reserved Kibana. Now Garbage info before log
  15. Jenkins World #JenkinsWorld Build Log Browsing. Elasticsearch queries © 2016

    CloudBees, Inc. All Rights Reserved
  16. Jenkins World #JenkinsWorld Next steps 1. Cleanup leftover tasks –https://docs.google.com/document/d/1hXBj2QIMQZcXuAPHm

    hvH8B1ynNmRih8vrPoGchu6N_8/edit#heading=h.dd7uzhbqscs6 2. Create Jenkins JIRA EPIC for further steps 3. Write design for the mergeable implementation –Better core API (Support of tasks, etc.) –RemoteConsoleAnnotator extension point –Changes in Pipeline to be done –etc. © 2016 CloudBees, Inc. All Rights Reserved
  17. Jenkins World #JenkinsWorld Sources • Logstash-based PoC: –https://github.com/oleg-nenashev/logstash-plugin/tree/jw- hackaton-remote-logging •

    Core with extensions: –https://github.com/oleg-nenashev/jenkins/tree/jw-hackaton- external-logging • Pipeline Job: –https://github.com/oleg-nenashev/workflow-job- plugin/tree/jw-hackaton-external-logging © 2016 CloudBees, Inc. All Rights Reserved
  18. Jenkins World #JenkinsWorld © 2016 CloudBees, Inc. All Rights Reserved