My friend, the support bundle And how to keep healthy your Jenkins instances

© 2018 All Rights Reserved. !2 Arnaud Héritier • Long time Jenkins user and community member • August 2007: My first discussion on the mailing list and my first ticket about Jenkins born Hudson 1.131 • (ex-)Contributor on various plugins: • Maven, Xcode, Appaloosa, … • Member of the Jenkins infra team • I love to reboot confluence ☺ • Support Delivery Manager at CloudBees, Inc • I help our customers to get the best from Jenkins and CloudBees products

© 2018 All Rights Reserved. !3 Why this session? In this talk: • How to get a support bundle? • What do we find in a support bundle? • How to use its content to keep healthy a Jenkins instance?

How to get a support bundle ? My friend, the support bundle

© 2018 All Rights Reserved. !5 Support Core plugin

© 2018 All Rights Reserved. !6 On-demand generation UI • Go to /support/

© 2018 All Rights Reserved. !7 throw new TooMuchChoicesException() • More than 30 options to generate a bundle

© 2018 All Rights Reserved. !8 On-demand generation CLI • Go to /cli/command/support/

© 2018 All Rights Reserved. !9 Automatic generation • Automatically saved under: $JENKINS_HOME/support • Starting few seconds after Jenkins starts the plugin • i.e. this may still be generated even if Jenkins will not fully start. • Then every hour • The automatic bundles are retained using an exponential ageing strategy

© 2018 All Rights Reserved. !10 Support Bundle Anonymization • Since 2.48 (July 2018), must be activated in the global configuration settings. • What is redacted: • agent names → node_XXX • agent computer names → computer_XXX • agent labels → label_XXX • view names → view_XXX • job names → item_XXX • usernames → user_XXX • and IP addresses (both IPv4 and IPv6) → ip_XXX • Data are mapped to randomly generated fake names which are saved to Jenkins master. • A Jenkins administrator can view these mappings by going to 
 Manage Jenkins › Support Bundle Anonymization.

© 2018 All Rights Reserved. !11

© 2018 All Rights Reserved. !12 Support Bundle Anonymization • All files written to the support bundle by this and all extensions of the support-core plugin will replace all instances of the original values with their anonymized counterpart. • Exceptions in the anonymization • Stop words: terms are ignored when filtering names (case-insensitive full match) • Files containing only Jenkins plugins and other Java libraries using version numbers that are indistinguishable from IP addresses • active plugins (plugins/active.txt) • disabled plugins (plugins/disabled.txt) • failed plugins (plugins/failed.txt) • Dockerfile (docker/Dockerfile)

Inside the Support Bundle Which mysteries does it keep? My friend, the support bundle

© 2018 All Rights Reserved. !14 Inside Jenkins - A quick overview • About Jenkins • about.md • docker/Dockerfile • items.md • nodes.md • plugins/active.txt • plugins/disabled.txt • plugins/failed.txt

© 2018 All Rights Reserved. !15 Who are you? • About browser - useful for rendering issues • browser.md • About user (basic authentication details only) • user.md

© 2018 All Rights Reserved. !16 Inside Jenkins - What’s running? #ornot • Build queue • buildqueue.md • Thread dumps • nodes/master/thread-dump.txt • nodes/slave//thread-dump.txt • Thread dumps of running Pipeline builds • nodes/master/pipeline-thread-dump.txt

© 2018 All Rights Reserved. !17 Inside Jenkins - What happened? Logs, logs, logs …. • Master Log Recorders (If logs are managed by the embedded application server) • nodes/master/logs/**/*.log • other-logs/*.log • Agent Log Recorders (Disabled by default) • nodes/slave//jenkins.log • nodes/slave//logs/*.log

© 2018 All Rights Reserved. !18 Inside Jenkins - Statistics and performances • Load Statistics • load-stats/label//gnuplot • load-stats/label//hour.csv • load-stats/label//min.csv • load-stats/label//sec10.csv • load-stats/no-label/gnuplot • load-stats/no-label/hour.csv • load-stats/no-label/min.csv • load-stats/no-label/sec10.csv • load-stats/overall/gnuplot • load-stats/overall/hour.csv • load-stats/overall/min.csv • load-stats/overall/sec10.csv

© 2018 All Rights Reserved. !19 Inside Jenkins - Statistics and performances • Timing data about recently completed Pipeline builds • nodes/master/pipeline-timings.txt • Metrics (from the Metrics plugin) • nodes/master/metrics.json • Node monitors • node-monitors.md • Agent Command Statistics • nodes/slave//command-stats.md

© 2018 All Rights Reserved. !20 Inside Jenkins - Potential or detected issues • Slow Request Records • slow-requests/*.txt • Administrative monitors • admin-monitors.md • Deadlock Records • deadlocks/DeadlockDetected-yyyyMMdd-hhmmss.txt

© 2018 All Rights Reserved. !21 Inside Jenkins - The configuration • Agents config files (Encrypted secrets are redacted) (Disabled by default) • nodes/slave//config.xml • Jenkins Global Configuration File (Encrypted secrets are redacted) (Disabled by default) • jenkins-root-configuration-files/config.xml • Other Jenkins Configuration Files (Encrypted secrets are redacted) (Disabled by default) • jenkins-root-configuration-files/*.xml • All loggers currently enabled • loggers.md

© 2018 All Rights Reserved. !22 Inside Jenkins - The configuration • Update Center (and web proxy settings) • update-center.md • System properties • nodes/master/system.properties • nodes/slave//system.properties • Root CAs (Disabled by default) - Useful to diagnose certificates authorities related issues • nodes/master/RootCA.txt • nodes/slave//RootCA.txt

© 2018 All Rights Reserved. !23 The JVM - memory management • Garbage Collection Logs (If configured at the JVM level) • /nodes/master/logs/gc/gc-*.log.* • Master Heap Histogram (Disabled by default) • nodes/master/heap-histogram.txt • Requires to set the system property com.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false to be activated • Out Of Memory Errors • /nodes/master/oome.md • Dump agent export tables (could reveal some memory leaks) • nodes/slave//exportTable.txt

© 2018 All Rights Reserved. !24 Operating system, network and more … • Master system configuration (Linux only) • nodes/master/dmesg.txt • nodes/master/dmi.txt • nodes/master/proc/cpuinfo.txt • nodes/master/proc/mounts.txt • nodes/master/proc/net/rpc/nfs.txt • nodes/master/proc/net/rpc/nfsd.txt • nodes/master/proc/swaps.txt • nodes/master/proc/system-uptime.txt • nodes/master/sysctl.txt • nodes/master/userid.txt

© 2018 All Rights Reserved. !25 Operating system, network and more … • Agent system configuration (Linux only) • nodes/slave//dmesg.txt • nodes/slave//dmi.txt • nodes/slave//proc/cpuinfo.txt • nodes/slave//proc/mounts.txt • nodes/slave//proc/net/rpc/nfs.txt • nodes/slave//proc/net/rpc/nfsd.txt • nodes/slave//proc/swaps.txt • nodes/slave//proc/system-uptime.txt • nodes/slave//sysctl.txt • nodes/slave//userid.txt

© 2018 All Rights Reserved. !26 Operating system, network and more … • Master JVM process system metrics (Linux only) • nodes/master/proc/meminfo.txt • nodes/master/proc/self/cmdline • nodes/master/proc/self/environ • nodes/master/proc/self/limits.txt • nodes/master/proc/self/mountstats.txt • nodes/master/proc/self/status.txt

© 2018 All Rights Reserved. !27 Operating system, network and more … • Agent JVM process system metrics (Linux only) (Disabled by default) • nodes/slave//proc/meminfo.txt • nodes/slave//proc/self/cmdline • nodes/slave//proc/self/environ • nodes/slave//proc/self/limits.txt • nodes/slave//proc/self/mountstats.txt • nodes/slave//proc/self/status.txt

© 2018 All Rights Reserved. !28 Operating system, network and more … • Environment variables • nodes/master/environment.txt • nodes/slave//environment.txt • File descriptors (Unix only) • nodes/master/file-descriptors.txt • nodes/slave//file-descriptors.txt • Networking Interface • nodes/master/networkInterface.md • nodes/slave//networkInterface.md

© 2018 All Rights Reserved. !29 If you are using a CloudBees product • CloudBees Assurance Program • cap/beekeeper.md • cap/properties.md • Client Masters Connection Logs (for an Operations Center) • client-masters-connection-logs/ID-.log • Operations Center Connector Logs (for a Client Master) • operations-center-connector-logs/connection-details.md • operations-center-connector-logs/operations-center- connector.log

© 2018 All Rights Reserved. !30 404 - Not found • You won’t find in a bundle any data related about jobs/items: • configuration • build logs • build results

How to keep your instance healthy? Examples of issues you can easily diagnose My friend, the support bundle

© 2018 All Rights Reserved. !32 Sanity checks of your environment • Verify that you don’t have some errors in the logs which aren’t explained. • Verify the most obvious issues like the presence of deadlocks (in deadlocks folder and in thread-dump.txt files). • Verify how your instance usage is evolving (number of jobs, agents/labels statistics). • And update it regularly to get the latests improvements in term of stability / performances and to keep it secured.

© 2018 All Rights Reserved. !33 Unattended behaviour • Verify in the logs if you find some relevant errors. • Verify you don’t have don’t have a problem with your plugins and some of them aren’t loaded correctly (`plugins/failed.txt`) • In recent Jenkins versions (2.x) you should have a warning/monitor in the Manage Jenkins screen about such errors • If you are using a CloudBees product, enable CAP and configure it to automatically upgrade your plugins • Verify that you don’t have some network issues which could impact the connection between the master and its agents, and in the case of CloudBees products it doesn’t impact also the connection between the Operations Center and a master.

© 2018 All Rights Reserved. !34 Stability issues • Verify in the logs some recurrent errors (especially with SEVERE level) • Verify your GC logs (you have various tools for that like Eclipse GCMV or gceasy.io) to see if you don’t have to tune your memory settings and the GC • See our Knowledge Base Prepare Jenkins for Support • Verify your system • ulimit config for the Jenkins system account
 ulimit -n should be set to 4096 (soft) and 8192 (hard)
 ulimit -u should be set to 30654 (soft) and 30654 (hard) • System logs (dmesg.txt) should not show any java process frozen or killed. It shouldn’t should also any connectivity issues when using NFS.

© 2018 All Rights Reserved. !35 Performances issues • Verify the JVM Memory and GC configuration • Are you using the right settings? • See our Knowledge Base Prepare Jenkins for Support • Verify the loggers configuration • No default empty loggers or some too verbose loggers • Compare your metrics with a previous bundle • Do you have more jobs? more builds? • Verify the slow-requests • it’s generally not normal to take more than 20 seconds to process an end-user requests

Conclusion Subheading My friend, the support bundle

© 2018 All Rights Reserved. !37 Conclusion • Proactively and regularly review your bundles to understand more your instance. Don’t wait to have an important outage! • Archive them regularly, to compare them over the time (especially before/after an upgrade).

© 2018 All Rights Reserved. !38 Conclusion • If you are facing an issue that you don’t succeed to explain • Our product CloudBees Core is awaiting for you (it includes obviously our support) and if you prefer to stay on Jenkins LTS we’ll support you with pleasure with our new offer CloudBees Jenkins Support. • Setup CloudBees Devoptics for free to monitor your platform usage and then to measure, manage and optimize your software delivery. If you don’t (yet ) have a CloudBees subscription

© 2018 All Rights Reserved. !39 Conclusion • If you are facing an issue that you don’t succeed to explain • Just open a support case and • If it’s not yet done, install our service CloudBees Jenkins Advisor which will automatically analyse your bundles with several hundred of probes based on all issues we are seeing on our support side. If you already have a subscription

© 2018 All Rights Reserved. !40 Share Your Feedback Great session? Needs improvement? LET ME KNOW! Complete this session’s survey (found under session details in the Schedule) to share your feedback on my session. The more surveys you complete, the better chance you have to win a Nintendo Switch! *Prize will be awarded during Thursday’s closing keynote session.

Thank you