Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JW2016 Hackathon. External Build Logging. Prototype Demo

Oleg Nenashev
September 18, 2016

JW2016 Hackathon. External Build Logging. Prototype Demo

This presentation summarizes results of our project from the Jenkins World 2016 hackaton. The prototype implements reporting of build logs to Logstash/Elasticsearch instead of the system disk and their further visualization within the Jenkins Web UI.

The approach allows to reduce network and IO load on Jenkins master and thus avoid performance bottlenecks on large-scale instances.

Oleg Nenashev

September 18, 2016
Tweet

More Decks by Oleg Nenashev

Other Decks in Programming

Transcript

  1. Jenkins World
    #JenkinsWorld
    External Build Log Storage. Prototype demo
    Oleg Nenashev1, Xing Yan2
    © 2016 CloudBees, Inc. All Rights Reserved
    HACKATHON
    * 1 – Jenkins Project && CloudBees, Inc., 2 - Google, Inc.

    View Slide

  2. Jenkins World
    #JenkinsWorld
    Presenter Info
    Oleg Nenashev
    • Jenkins core contributor
    • JAM and Jenkins Online Meetup organizer
    • Maintainer of plugins
    • Jenkins Developer at CloudBees, Inc.
    @oleg_nenashev
    oleg-nenashev
    onenashev

    View Slide

  3. Jenkins World
    #JenkinsWorld
    Problem statement
    • All Jenkins agents report logs to master
    • High disk and network throughput on big instances
    • Users view Console logs
    => Master has to load them
    ÞAdditional load
    • (+) Logrotate, backups, etc., etc.
    © 2016 CloudBees, Inc. All Rights Reserved
    Logging may be a
    performance bottleneck

    View Slide

  4. Jenkins World
    #JenkinsWorld
    Possible solution –
    Logging to remote logging system
    • System logs – trivial (log appenders)
    • Build logs – complex
    • Custom implementation of the channel
    • Different run implementation (AbstractProject, Pipeline)
    • Dependent extension points (Console Annotator)
    © 2016 CloudBees, Inc. All Rights Reserved

    View Slide

  5. Jenkins World
    #JenkinsWorld
    External build logs. Prototyping approach
    © 2016 CloudBees, Inc. All Rights Reserved
    Master
    Node
    Logstash
    Elasticsearch
    Kibana
    Logs from on-
    master operations
    Logs from remote
    invocations
    Build Console
    • No build logs
    on the disk
    • No log loading
    from nodes
    • Reduced load
    on master
    Storage
    Visualizer

    View Slide

  6. Jenkins World
    #JenkinsWorld
    Prototype. Scope of change
    © 2016 CloudBees, Inc. All Rights Reserved
    Jenkins Core
    (2.23-SNAPSHOT)
    Logstash Plugin
    (1.2.1-SNAPSHOT)
    Workflow Job Plugin
    (2.7-SNAPSHOT)
    • LoggingMethod abstraction layer
    • LoggingMethodLocator Extension Point
    • Hack for Console Annotator markers
    (no-special-symbols)
    dep.
    dep.
    dep.
    • Parent POM update
    • Pipeline Support
    • LoggingMethod implementation
    • External log browsers
    • Logging Method hook

    View Slide

  7. Jenkins World
    #JenkinsWorld
    Logging Method Factory
    © 2016 CloudBees, Inc. All Rights Reserved
    Fine-grain
    behavior for
    runs
    Default
    implementation
    in the core

    View Slide

  8. Jenkins World
    #JenkinsWorld
    Logging Method
    © 2016 CloudBees, Inc. All Rights Reserved
    Process
    invocations
    On-master
    operations

    View Slide

  9. Jenkins World
    #JenkinsWorld
    Prototype. Supported reporters
    © 2016 CloudBees, Inc. All Rights Reserved
    4 indexers (from
    Logstash plugin)
    Basic Auth to
    destination servers

    View Slide

  10. Jenkins World
    #JenkinsWorld
    Reporting. AbstractProject-based jobs
    © 2016 CloudBees, Inc. All Rights Reserved
    Master
    Node
    Logstash
    Elasticsearch
    Logs from on-
    master operations
    Logs from remote
    invocations
    Storage
    Current state:
    • Master still saves data to disk (easy to fix)
    • Logs from nodes go directly to Elasticsearch

    View Slide

  11. Jenkins World
    #JenkinsWorld
    Reporting. Pipeline jobs
    © 2016 CloudBees, Inc. All Rights Reserved
    Master
    Node
    Logstash
    Elasticsearch
    Logs from on-
    master operations
    Logs from remote
    invocations
    Storage
    Current state:
    • Master reports data to both disk and Elasticsearch
    • A change in Durable task is required to redirect logs

    View Slide

  12. Jenkins World
    #JenkinsWorld
    Build Log Browsing
    • Two modes are supported
    • Mode #1. “Kibana”
    – Embeds Kibana query layout
    – Request to Kibana goes from browser => Less load on master
    – Issue: Kibana with Authentication
    • Mode #2. “Elasticsearch”
    – Jenkins queries Elasticsearch and renders results
    – Pros:
    o Support of basic auth
    o Log annotation can be implemented
    – Cons: Request go through the master server
    © 2016 CloudBees, Inc. All Rights Reserved

    View Slide

  13. Jenkins World
    #JenkinsWorld
    Build Log Browsing. Kibana
    © 2016 CloudBees, Inc. All Rights Reserved
    Kibana.
    Want to have

    View Slide

  14. Jenkins World
    #JenkinsWorld
    Build Log Browsing. Kibana
    © 2016 CloudBees, Inc. All Rights Reserved
    Kibana.
    Now
    Garbage
    info
    before
    log

    View Slide

  15. Jenkins World
    #JenkinsWorld
    Build Log Browsing. Elasticsearch queries
    © 2016 CloudBees, Inc. All Rights Reserved

    View Slide

  16. Jenkins World
    #JenkinsWorld
    Next steps
    1. Cleanup leftover tasks
    –https://docs.google.com/document/d/1hXBj2QIMQZcXuAPHm
    hvH8B1ynNmRih8vrPoGchu6N_8/edit#heading=h.dd7uzhbqscs6
    2. Create Jenkins JIRA EPIC for further steps
    3. Write design for the mergeable implementation
    –Better core API (Support of tasks, etc.)
    –RemoteConsoleAnnotator extension point
    –Changes in Pipeline to be done
    –etc.
    © 2016 CloudBees, Inc. All Rights Reserved

    View Slide

  17. Jenkins World
    #JenkinsWorld
    Sources
    • Logstash-based PoC:
    –https://github.com/oleg-nenashev/logstash-plugin/tree/jw-
    hackaton-remote-logging
    • Core with extensions:
    –https://github.com/oleg-nenashev/jenkins/tree/jw-hackaton-
    external-logging
    • Pipeline Job:
    –https://github.com/oleg-nenashev/workflow-job-
    plugin/tree/jw-hackaton-external-logging
    © 2016 CloudBees, Inc. All Rights Reserved

    View Slide

  18. Jenkins World
    #JenkinsWorld
    © 2016 CloudBees, Inc. All Rights Reserved

    View Slide