Managing Java Applications with Chef

Slide 1

Slide 1 text

Showing the JVM Some Love Managing Java Apps with Chef Bryan W. Berry, Chefconf 2013

Slide 2

Slide 2 text

About Me ● Sr. DevOps Engineer at Cycle Computing ● Creator and co-host of FoodFightShow podcast ● Author of many java-related cookbooks ● bryanwb on twitter, github, IRC ● [email protected]

Slide 3

Slide 3 text

We're Hiring! Come see me or Rob Futrick

Slide 4

Slide 4 text

Hate the Language, Love the Runtime “Java is the new C”* image: https://wiki.smu.edu.sg/is200/Java_Virtual_Machine * http://www.slideshare.net/mobile/pcalcado/from-a-monolithic-ruby-on-rails-app-to-the-jvm

Slide 5

Slide 5 text

The JVM, A World of its Own Windows Linux Ruby

Slide 6

Slide 6 text

Headaches ● Distro package or tarball? ● Cargo-culting JAVA_OPTS, CATALINA_OPTS, HADOOP_OPTS, etc. ● Managing Resource limits ● Managing Giant XML files ● Monitoring JVMs with JMX ● Competing with Log4j to rotate and compress log files

Slide 7

Slide 7 text

Let's Walk Some Talk ● Let's write a simple java related cookbook ● Oracle JDK 7 ● Tomcat as the container ● Tweak some kernel settings ● Fetch dependencies with Maven ● Monitor it with JMX Let's call it jvm-love in honor of Ulf

Slide 8

Slide 8 text

In the beginning . . . ● You need to install the Java JDK ● The opscode Java cookbook is excellent

Slide 9

Slide 9 text

Installing a JDK To install openjdk include_recipe “java” To install Oracle set[:java][:install_flavor] = “oracle” set[:java][:accept_oracle_download_terms] = true include_recipe “java” To change JDK version node.set[:java][:jdk_version] = '7' # it is 6 by default

Slide 10

Slide 10 text

Distro Package Cons ● Often out-of-date ● LFS spreads files all over filesystem ● Includes shell scripts w/ unwanted side effects ● Init script can be ridiculously complicated

Slide 11

Slide 11 text

FHS can confuse non-professional linux geeks ● Config files in /etc/tomcat7 ● Webapps, logs, libs in /usr/share/tomcat7 ● Temp directory in /var/cache/tomcat7 ● Various binary commands in /usr/[s]bin/ ● logs in /var/lib/tomcat7/logs* *ubuntu 10.10

Slide 12

Slide 12 text

Freshness ● No Tomcat7 in the default CentOS/RHEL repos ● Ubuntu provides 7.0.30*, ~7 months old ● Neither provide packages for Glassfish3 or JBoss > 4.0 ● jpackage.org 6.0 provides tomcat7 rpms, sometimes** *as of 26 March 2013 **the site is sometimes down

Slide 13

Slide 13 text

Installing from (Binary) Tarball ● Java Packages are arch-independent, no `make` step needed ● The download, unpack, link process can be tedious, but there are tools to help

Slide 14

Slide 14 text

Meet Ark ark “tomcat” do version “7.0.37” checksum “3abc3434..” url ”http://..../tomcat.tar.gz” owner “jvmlover” action :install end

Slide 15

Slide 15 text

Ark's Aftermath ● Tomcat tarball downloaded from Apache site ● tarball unpacked to /usr/local/tomcat-#{version} ● /usr/local/tomcat symlinked to /usr/local/tomcat-#{version} ● jvm-lover user made owner of /usr/local/tomcat-#{version}

Slide 16

Slide 16 text

Ark Issues ● You can inadvertently DDOS your apache.org ● You should use http://apache.mirrors.tds.net but that site only hosts recent releases ● Use http://archive.apache.org, but DDOS problem again

Slide 17

Slide 17 text

Do it Yourself ● Host the tarball yourself OR ● Implement a mirror_file LWRP that randomly selects from a list of Apache mirrors

Slide 18

Slide 18 text

Upgrading Tomcat w/ Ark ● Change the version number ark “tomcat” do version “7.0.38” #previously 7.0.37 # other settings as before end /usr/local/tomcat now points to /usr/local/tomcat-7.0.38

Slide 19

Slide 19 text

JVM Vs. Linux Low level settings that are a must 1. Maximum number of open files 2. Maximum number of processes (threads) 3. Swapping to disk

Slide 20

Slide 20 text

Use the Ulimit LWRP include_recipe “ulimit” user_ulimit "jvmlover" do filehandle_limit 8192 process_limit 61504 end

Slide 21

Slide 21 text

Use the Syctl LWRP include_recipe “sysctl” node.override['sysctl_file'] = '/etc/sysctl.d/99chef.conf' sysctl "kernel.swappiness" do value 0 end

Slide 22

Slide 22 text

Init Scripts You have 3 choices ● Upstart ● System V ● Runit* *compiled from source

Slide 23

Slide 23 text

SystemV ● Understood by many ● Horrible spaghetti Bash involving multiple scripts ● It is unix black magic to detach the JVM process from the session & grab its process ID ● No process monitoring

Slide 24

Slide 24 text

Upstart ● Easier to write than system V ● Some naïve process supervision ● In my anecdotal experience, not as reliable as systemV ● Deprecated on CentOS/RHEL in favor of SystemD

Slide 25

Slide 25 text

Runit ● Rock solid ● Easy ● Awesome on Ubuntu/Debian ● No native packages for CentOS* ● Not as widely known as sysv or upstart *You can install from source Using opscode's runit cookbook

Slide 26

Slide 26 text

Serious Supervision ● Docker and/or SystemD ● contain processes with LXC namespaces rather than tracking a process ID ● Systemd currently available in debian, coming to CentOS/RHEL 7, ubuntu maybe never

Slide 27

Slide 27 text

Pragmatic Process Supervision ● If you primarily use Ubuntu, use runit ● Think you use debian or will adapt CentOS/RHEL7 quickly, use/wait for systemd or docker ● If you have to use sysv or upstart, try to couple it with docker

Slide 28

Slide 28 text

Scripting like it's 1970 JAVA_USER=jvmlover PID_FILE=”/var/run/jvmlover/jvmlover.pid” LOG=”/var/log/jvmlove/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT

Slide 29

Slide 29 text

Scripting like it's 1970 JAVA_USER=jvmlover PID_FILE=”/var/run/jvmlover/jvmlover. pid” LOG=”/var/log/jvmlove/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT Let's Use the Process ID to track our running application

Slide 30

Slide 30 text

Scripting like it's 1970 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT Redirect STDOUT and STDERR to the Log file

Slide 31

Slide 31 text

Daemonization 101 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT 1. fork a new process in a new shell 2. detach from that process Step 1 Step 2

Slide 32

Slide 32 text

PID Magic JAVA_USER=jvmlover PID_FILE=”/var/run/jvmlover/jvmlover.pid” LOG=”/var/log/jvmlove/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT Capture the Process ID

Slide 33

Slide 33 text

sysv or upstart with Docker docker u $JAVA_USER run $SCRIPT Replaces su $JAVA_USER c $SCRIPT Pretty simple! But with a lot of benefits

Slide 34

Slide 34 text

You can't escape JAVA_OPTS JAVA_USER=jvmlover PID_FILE=”/var/run/jvmlover/jvmlover.pid” LOG=”/var/log/jvmlove/catalina.out SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT

Slide 35

Slide 35 text

JAVA_OPTS Expands to XX:MaxPermSize=256M Xmx5G Xms2G server XX:+DisableExplicitGC XX:+UseParallelOldGC XX:NewRatio=2 XX:SoftRefLRUPolicyMSPerMB=36000 Dsun.rmi.dgc.server.gcInterval=3600000 XX:+UseBiasedLocking Xrs Djava.rmi.server.hostname=127.0.0.1 Dcom.sun.management.jmxremote Dcom.sun.management.jmxremote.port=9000 And this is a short example!

Slide 36

Slide 36 text

JAVA_OPTS Expands to XX:MaxPermSize=256M Xmx5G Xms2G server XX: +DisableExplicitGC XX:+UseParallelOldGC XX:NewRatio=2 XX:SoftRefLRUPolicyMSPerMB=36000 Dsun.rmi.dgc.server.gcInterval=3600000 XX:+UseBiasedLocking Xrs Djava.rmi.server.hostname=127.0.0.1 Dcom.sun.management.jmxremote Dcom.sun.management.jmxremote.port=9000 image: http://brain-cheese.blogspot.it/2009/05/sometimes-i-want-to-scream-like-homer.html

Slide 37

Slide 37 text

A parsing nightmare 4 kinds of options -server -Xmx5G -XX:MaxPermSize=256M -DJSON_ENABLE=true Standard Nonstandard Unstable/Experimental Directives

Slide 38

Slide 38 text

The JAVA_OPTS are a changin' -Xmx, -Xms, -XX:MaxPermSize can change wildly depending on the application and environment but typically have lower bounds image: http://www.tshirtbordello.com/What-The-Frak-T-Shirt

Slide 39

Slide 39 text

Jvmargs to the rescue ● A sane parser for insane JVM options ● A Ruby gem

Slide 40

Slide 40 text

jvmargs in action # environments/test.rb default_attributes( “jvmlove” => { “java_opts” => [ 'Xint', 'XX:+DisableExplicitGC'] } ) # jvmlove/recipes/default.rb chef_gem 'jvmargs' require 'jvmargs # minimum settings args = JVMArgs::Args.new('Xmx128M', 'Xms128M', 'XX:MaxPermSize=256M') args.add(node['jvmlove']['java_opts'])

Slide 41

Slide 41 text

jvmargs in action # jvmlove/recipes/default.rb template “/etc/init.d/jvmlove” do source “init.erb” mode 00775 variables(:args => args) end

Slide 42

Slide 42 text

jvmargs in action # templates/default/init.erb SCRIPT=”java #{@args} org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT

Slide 43

Slide 43 text

more examples args = JVMArgs::Args.new("Xmx256M") args.add("Xmx2G") args.add("XX:+DisableExplicitGC") # the args are now # "XX:+DisableExplicitGC Xmx2G"

Slide 44

Slide 44 text

helpers args = JVMArgs::Args.new do jmx true heap_size “40%” permgen “256M” newgen “32M” end

Slide 45

Slide 45 text

results Xmx819M XX:MaxNewSize=32M XX:MaxPermSize=256M Djava.rmi.server.hostname=127.0.0.1 Dcom.sun.management.jmxremote Dcom.sun.management.jmxremote.ssl=false Dcom.sun.management.jmxremote.port=9000

Slide 46

Slide 46 text

jvmargs limitations Can't currently handle classpath /tmp/foo.jar:/tmp/bar.jar or Xrunjdwp:transport=dt_socket,address=10 45 “-server” currently added by default

Slide 47

Slide 47 text

jvmargs rules ● Cargo-culting can lead to conflicting options ● Many preventable errors ● Why can't this be configurable? “Don't set the max heap smaller than the minimum heap”

Slide 48

Slide 48 text

jvmargs rules args.add_rule(:max_smaller_than_min) do |args| max_heap = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xmx"].value) min_heap = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xms"].value) if max_heap < min_heap raise ArgumentError, "Max heap #{max_heap} too small!" end end

Slide 49

Slide 49 text

jvmargs rules ● rules are run whenever arguments are added and when JVMArgs::Args#to_s called ● can raise errors, change arg values, or pretty much anything else

Slide 50

Slide 50 text

fetching dependencies ● Need the postgresql JDBC driver ● you should pull from maven.org ● you should use the maven LWRP

Slide 51

Slide 51 text

maven it include_recipe “maven” maven "postgresql" do group_id “postgresql” version “9.21002jdbc4” owner "jvmlover" dest '/usr/local/tomcat/lib' transitive true end creates /usr/local/tomcat/lib/postgresql.jar

Slide 52

Slide 52 text

Java <3 XML ● Tomcat not so verbose, other containers can be much worse, esp. J2EEEEEEEEE ● It is really hard to write a DSL for XML because it is arbitrarily complex ● Jboss 7 standalone-full.xml is 706 lines long!

Slide 53

Slide 53 text

Handling XML ● templates – limited flexibility ● write your own DSL (HARD) OR ● use template partials (EASY)

Slide 54

Slide 54 text

context.xml Note: This is heavily abridged

Slide 55

Slide 55 text

context.xml.erb

Slide 56

Slide 56 text

context.xml.erb

Slide 57

Slide 57 text

context.xml.erb w/ partials* <%= render 'data_source.xml.erb', @template_context %> <%= render 'jms_factory.xml.erb', @template_context %> <%= render 'jms_topic.xml.erb', @template_context %> <%= render 'jms_queue.xml.erb', @template_context %> *introduced in chef11

Slide 58

Slide 58 text

Monitoring your JVM include_recipe “collectd::generic_jmx” collectd_jmx “myapp” do port 8999 username “foo” Password “bar” end

Slide 59

Slide 59 text

Log Rotation and Compression ● logrotate or log4j? ● logrotate has far fewer features ● log4j can't manage catalina.out or console.log ● java devs like log4j ● log4j can't compress log files :( ● Why not use both? image: http://logging.apache.org/log4j/1.2/

Slide 60

Slide 60 text

Rotating catalina.out include “logrotate” logrotate_app "jvmlove" do path "/usr/local/tomcat/logs/catalina.out" options [ "compress", "copytruncate" ] rotate 7 create "664 jvmlove jvmlove" end

Slide 61

Slide 61 text

Cleaning up after logrotate include_recipe “janitor” janitor_logs "/usr/local/tomcat/logs" do include_files ["*.log"] max_age 30 min_age 1 action [:compress,:purge] end Compresses all files that do not end in .gz

Slide 62

Slide 62 text

The Smart Kids Use SLF4J + Logback, which can compress files image: http://www.alice1059.com/Smart-kids-make-us-sick/11315931?pid=305825&archive=1

Slide 63

Slide 63 text

Local Logs are Dumb Logs Send 'em to Logstash ● Use gelf4j or log4j Socket Appender ● Together w/ the log4j or gelf logstash input on logstash indexer

Slide 64

Slide 64 text

Picking a Logstash Agent 1. Stock Agent, written in Jruby, sizeable footprint but can process logs before shipping 2. Lightweight shippers, no processing, ship everything I recommend starting w/ the stock agent

Slide 65

Slide 65 text

Still Needed ● mirror_file LWRP ● Keystore LWRP ● templating “shifting” xml files ● Validating XML document with nokogiri before updating a configuration file ● more full-featured jvmargs

Slide 66

Slide 66 text

Thanks To ● Xabier de Zuazo (@zuazo) ● John Vincent (@lusis) ● Nathen Harvey ● Julian Dunn ● Peter Donald

Slide 67

Slide 67 text

Resources ● berkshelf ● test-kitchen ● ark ● docker ● systemd ● java cookbook ● maven cookbook ● jvmargs ● logrotate ● chef-janitor ● FoodFightShow ● chef-collectd