JW2016 Hackathon. External Build Logging. Prototype Demo

This presentation summarizes results of our project from the Jenkins World 2016 hackaton. The prototype implements reporting of build logs to Logstash/Elasticsearch instead of the system disk and their further visualization within the Jenkins Web UI.

The approach allows to reduce network and IO load on Jenkins master and thus avoid performance bottlenecks on large-scale instances.

Transcript

1. Jenkins World
   #JenkinsWorld
   External Build Log Storage. Prototype demo
   Oleg Nenashev1, Xing Yan2
   © 2016 CloudBees, Inc. All Rights Reserved
   HACKATHON
   * 1 – Jenkins Project && CloudBees, Inc., 2 - Google, Inc.
   

   View Slide

2. Jenkins World
   #JenkinsWorld
   Presenter Info
   Oleg Nenashev
   • Jenkins core contributor
   • JAM and Jenkins Online Meetup organizer
   • Maintainer of plugins
   • Jenkins Developer at CloudBees, Inc.
   @oleg_nenashev
   oleg-nenashev
   onenashev
   

   View Slide

3. Jenkins World
   #JenkinsWorld
   Problem statement
   • All Jenkins agents report logs to master
   • High disk and network throughput on big instances
   • Users view Console logs
   => Master has to load them
   ÞAdditional load
   • (+) Logrotate, backups, etc., etc.
   © 2016 CloudBees, Inc. All Rights Reserved
   Logging may be a
   performance bottleneck
   

   View Slide

4. Jenkins World
   #JenkinsWorld
   Possible solution –
   Logging to remote logging system
   • System logs – trivial (log appenders)
   • Build logs – complex
   • Custom implementation of the channel
   • Different run implementation (AbstractProject, Pipeline)
   • Dependent extension points (Console Annotator)
   © 2016 CloudBees, Inc. All Rights Reserved
   

   View Slide

5. Jenkins World
   #JenkinsWorld
   External build logs. Prototyping approach
   © 2016 CloudBees, Inc. All Rights Reserved
   Master
   Node
   Logstash
   Elasticsearch
   Kibana
   Logs from on-
   master operations
   Logs from remote
   invocations
   Build Console
   • No build logs
   on the disk
   • No log loading
   from nodes
   • Reduced load
   on master
   Storage
   Visualizer
   

   View Slide

6. Jenkins World
   #JenkinsWorld
   Prototype. Scope of change
   © 2016 CloudBees, Inc. All Rights Reserved
   Jenkins Core
   (2.23-SNAPSHOT)
   Logstash Plugin
   (1.2.1-SNAPSHOT)
   Workflow Job Plugin
   (2.7-SNAPSHOT)
   • LoggingMethod abstraction layer
   • LoggingMethodLocator Extension Point
   • Hack for Console Annotator markers
   (no-special-symbols)
   dep.
   dep.
   dep.
   • Parent POM update
   • Pipeline Support
   • LoggingMethod implementation
   • External log browsers
   • Logging Method hook
   

   View Slide

7. Jenkins World
   #JenkinsWorld
   Logging Method Factory
   © 2016 CloudBees, Inc. All Rights Reserved
   Fine-grain
   behavior for
   runs
   Default
   implementation
   in the core
   

   View Slide

8. Jenkins World
   #JenkinsWorld
   Logging Method
   © 2016 CloudBees, Inc. All Rights Reserved
   Process
   invocations
   On-master
   operations
   

   View Slide

9. Jenkins World
   #JenkinsWorld
   Prototype. Supported reporters
   © 2016 CloudBees, Inc. All Rights Reserved
   4 indexers (from
   Logstash plugin)
   Basic Auth to
   destination servers
   

   View Slide

10. Jenkins World
    #JenkinsWorld
    Reporting. AbstractProject-based jobs
    © 2016 CloudBees, Inc. All Rights Reserved
    Master
    Node
    Logstash
    Elasticsearch
    Logs from on-
    master operations
    Logs from remote
    invocations
    Storage
    Current state:
    • Master still saves data to disk (easy to fix)
    • Logs from nodes go directly to Elasticsearch
    

    View Slide

11. Jenkins World
    #JenkinsWorld
    Reporting. Pipeline jobs
    © 2016 CloudBees, Inc. All Rights Reserved
    Master
    Node
    Logstash
    Elasticsearch
    Logs from on-
    master operations
    Logs from remote
    invocations
    Storage
    Current state:
    • Master reports data to both disk and Elasticsearch
    • A change in Durable task is required to redirect logs
    

    View Slide

12. Jenkins World
    #JenkinsWorld
    Build Log Browsing
    • Two modes are supported
    • Mode #1. “Kibana”
    – Embeds Kibana query layout
    – Request to Kibana goes from browser => Less load on master
    – Issue: Kibana with Authentication
    • Mode #2. “Elasticsearch”
    – Jenkins queries Elasticsearch and renders results
    – Pros:
    o Support of basic auth
    o Log annotation can be implemented
    – Cons: Request go through the master server
    © 2016 CloudBees, Inc. All Rights Reserved
    

    View Slide

13. Jenkins World
    #JenkinsWorld
    Build Log Browsing. Kibana
    © 2016 CloudBees, Inc. All Rights Reserved
    Kibana.
    Want to have
    

    View Slide

14. Jenkins World
    #JenkinsWorld
    Build Log Browsing. Kibana
    © 2016 CloudBees, Inc. All Rights Reserved
    Kibana.
    Now
    Garbage
    info
    before
    log
    

    View Slide

15. Jenkins World
    #JenkinsWorld
    Build Log Browsing. Elasticsearch queries
    © 2016 CloudBees, Inc. All Rights Reserved
    

    View Slide

16. Jenkins World
    #JenkinsWorld
    Next steps
    1. Cleanup leftover tasks
    –https://docs.google.com/document/d/1hXBj2QIMQZcXuAPHm
    hvH8B1ynNmRih8vrPoGchu6N_8/edit#heading=h.dd7uzhbqscs6
    2. Create Jenkins JIRA EPIC for further steps
    3. Write design for the mergeable implementation
    –Better core API (Support of tasks, etc.)
    –RemoteConsoleAnnotator extension point
    –Changes in Pipeline to be done
    –etc.
    © 2016 CloudBees, Inc. All Rights Reserved
    

    View Slide

17. Jenkins World
    #JenkinsWorld
    Sources
    • Logstash-based PoC:
    –https://github.com/oleg-nenashev/logstash-plugin/tree/jw-
    hackaton-remote-logging
    • Core with extensions:
    –https://github.com/oleg-nenashev/jenkins/tree/jw-hackaton-
    external-logging
    • Pipeline Job:
    –https://github.com/oleg-nenashev/workflow-job-
    plugin/tree/jw-hackaton-external-logging
    © 2016 CloudBees, Inc. All Rights Reserved
    

    View Slide

18. Jenkins World
    #JenkinsWorld
    © 2016 CloudBees, Inc. All Rights Reserved
    

    View Slide