Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Managing Java Applications with Chef

Managing Java Applications with Chef

How to manage java applications with Chef


Bryan Berry

April 27, 2013

More Decks by Bryan Berry

Other Decks in How-to & DIY


  1. Showing the JVM Some Love Managing Java Apps with Chef

    Bryan W. Berry, Chefconf 2013
  2. About Me • Sr. DevOps Engineer at Cycle Computing •

    Creator and co-host of FoodFightShow podcast • Author of many java-related cookbooks • bryanwb on twitter, github, IRC • bberry@cyclecomputing.com
  3. We're Hiring! Come see me or Rob Futrick

  4. Hate the Language, Love the Runtime “Java is the new

    C”* image: https://wiki.smu.edu.sg/is200/Java_Virtual_Machine * http://www.slideshare.net/mobile/pcalcado/from-a-monolithic-ruby-on-rails-app-to-the-jvm
  5. The JVM, A World of its Own Windows Linux Ruby

  6. Headaches • Distro package or tarball? • Cargo-culting JAVA_OPTS, CATALINA_OPTS,

    HADOOP_OPTS, etc. • Managing Resource limits • Managing Giant XML files • Monitoring JVMs with JMX • Competing with Log4j to rotate and compress log files
  7. Let's Walk Some Talk • Let's write a simple java

    related cookbook • Oracle JDK 7 • Tomcat as the container • Tweak some kernel settings • Fetch dependencies with Maven • Monitor it with JMX Let's call it jvm-love in honor of Ulf
  8. In the beginning . . . • You need to

    install the Java JDK • The opscode Java cookbook is excellent
  9. Installing a JDK To install openjdk include_recipe “java” To install

    Oracle set[:java][:install_flavor] = “oracle” set[:java][:accept_oracle_download_terms] = true include_recipe “java” To change JDK version node.set[:java][:jdk_version] = '7' # it is 6 by default
  10. Distro Package Cons • Often out-of-date • LFS spreads files

    all over filesystem • Includes shell scripts w/ unwanted side effects • Init script can be ridiculously complicated
  11. FHS can confuse non-professional linux geeks • Config files in

    /etc/tomcat7 • Webapps, logs, libs in /usr/share/tomcat7 • Temp directory in /var/cache/tomcat7 • Various binary commands in /usr/[s]bin/ • logs in /var/lib/tomcat7/logs* *ubuntu 10.10
  12. Freshness • No Tomcat7 in the default CentOS/RHEL repos •

    Ubuntu provides 7.0.30*, ~7 months old • Neither provide packages for Glassfish3 or JBoss > 4.0 • jpackage.org 6.0 provides tomcat7 rpms, sometimes** *as of 26 March 2013 **the site is sometimes down
  13. Installing from (Binary) Tarball • Java Packages are arch-independent, no

    `make` step needed • The download, unpack, link process can be tedious, but there are tools to help
  14. Meet Ark ark “tomcat” do version “7.0.37” checksum “3abc3434..” url

    ”http://..../tomcat.tar.gz” owner “jvm­lover” action :install end
  15. Ark's Aftermath • Tomcat tarball downloaded from Apache site •

    tarball unpacked to /usr/local/tomcat-#{version} • /usr/local/tomcat symlinked to /usr/local/tomcat-#{version} • jvm-lover user made owner of /usr/local/tomcat-#{version}
  16. Ark Issues • You can inadvertently DDOS your apache.org •

    You should use http://apache.mirrors.tds.net but that site only hosts recent releases • Use http://archive.apache.org, but DDOS problem again
  17. Do it Yourself • Host the tarball yourself OR •

    Implement a mirror_file LWRP that randomly selects from a list of Apache mirrors
  18. Upgrading Tomcat w/ Ark • Change the version number ark

    “tomcat” do version “7.0.38” #previously 7.0.37 # other settings as before end /usr/local/tomcat now points to /usr/local/tomcat-7.0.38
  19. JVM Vs. Linux Low level settings that are a must

    1. Maximum number of open files 2. Maximum number of processes (threads) 3. Swapping to disk
  20. Use the Ulimit LWRP include_recipe “ulimit” user_ulimit "jvm­lover" do filehandle_limit

    8192 process_limit 61504 end
  21. Use the Syctl LWRP include_recipe “sysctl” node.override['sysctl_file'] = '/etc/sysctl.d/99­chef.conf' sysctl

    "kernel.swappiness" do value 0 end
  22. Init Scripts You have 3 choices • Upstart • System

    V • Runit* *compiled from source
  23. SystemV • Understood by many • Horrible spaghetti Bash involving

    multiple scripts • It is unix black magic to detach the JVM process from the session & grab its process ID • No process monitoring
  24. Upstart • Easier to write than system V • Some

    naïve process supervision • In my anecdotal experience, not as reliable as systemV • Deprecated on CentOS/RHEL in favor of SystemD
  25. Runit • Rock solid • Easy • Awesome on Ubuntu/Debian

    • No native packages for CentOS* • Not as widely known as sysv or upstart *You can install from source Using opscode's runit cookbook
  26. Serious Supervision • ​Docker and/or SystemD • contain processes with

    LXC namespaces rather than tracking a process ID • Systemd currently available in debian, coming to CentOS/RHEL 7, ubuntu maybe never
  27. Pragmatic Process Supervision • If you primarily use Ubuntu, use

    runit • Think you use debian or will adapt CentOS/RHEL7 quickly, use/wait for systemd or docker • If you have to use sysv or upstart, try to couple it with docker
  28. Scripting like it's 1970 JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap

    start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
  29. Scripting like it's 1970 JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover. pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS

    org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT Let's Use the Process ID to track our running application
  30. Scripting like it's 1970 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap

    start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT Redirect STDOUT and STDERR to the Log file
  31. Daemonization 101 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \

    > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT 1. fork a new process in a new shell 2. detach from that process Step 1 Step 2
  32. PID Magic JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \

    > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT Capture the Process ID
  33. sysv or upstart with Docker docker ­u $JAVA_USER run $SCRIPT

    Replaces su $JAVA_USER ­c $SCRIPT Pretty simple! But with a lot of benefits
  34. You can't escape JAVA_OPTS JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap

    start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
  35. JAVA_OPTS Expands to ­XX:MaxPermSize=256M ­Xmx5G ­Xms2G ­server ­XX:+DisableExplicitGC ­XX:+UseParallelOldGC ­XX:NewRatio=2

    ­XX:SoftRefLRUPolicyMSPerMB=36000 ­Dsun.rmi.dgc.server.gcInterval=3600000 ­XX:+UseBiasedLocking ­Xrs ­Djava.rmi.server.hostname= ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.port=9000 And this is a short example!
  36. JAVA_OPTS Expands to ­XX:MaxPermSize=256M ­Xmx5G ­Xms2G ­server ­XX: +DisableExplicitGC ­XX:+UseParallelOldGC

    ­XX:NewRatio=2 ­XX:SoftRefLRUPolicyMSPerMB=36000 ­Dsun.rmi.dgc.server.gcInterval=3600000 ­XX:+UseBiasedLocking ­Xrs ­Djava.rmi.server.hostname= ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.port=9000 image: http://brain-cheese.blogspot.it/2009/05/sometimes-i-want-to-scream-like-homer.html
  37. A parsing nightmare 4 kinds of options -server -Xmx5G -XX:MaxPermSize=256M

    -DJSON_ENABLE=true Standard Nonstandard Unstable/Experimental Directives
  38. The JAVA_OPTS are a changin' -Xmx, -Xms, -XX:MaxPermSize can change

    wildly depending on the application and environment but typically have lower bounds image: http://www.tshirtbordello.com/What-The-Frak-T-Shirt
  39. Jvmargs to the rescue • A sane parser for insane

    JVM options • A Ruby gem
  40. jvmargs in action # environments/test.rb default_attributes( “jvm­love” => { “java_opts”

    => [ 'Xint', 'XX:+DisableExplicitGC'] } ) # jvm­love/recipes/default.rb chef_gem 'jvmargs' require 'jvmargs # minimum settings args = JVMArgs::Args.new('Xmx128M', 'Xms128M', 'XX:MaxPermSize=256M') args.add(node['jvm­love']['java_opts'])
  41. jvmargs in action # jvm­love/recipes/default.rb template “/etc/init.d/jvm­love” do source “init.erb”

    mode 00775 variables(:args => args) end
  42. jvmargs in action # templates/default/init.erb SCRIPT=”java #{@args} org.apache.Bootstrap start \

    > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
  43. more examples args = JVMArgs::Args.new("Xmx256M") args.add("Xmx2G") args.add("XX:+DisableExplicitGC") # the args

    are now # "­XX:+DisableExplicitGC ­Xmx2G"
  44. helpers args = JVMArgs::Args.new do jmx true heap_size “40%” permgen

    “256M” newgen “32M” end
  45. results ­Xmx819M ­XX:MaxNewSize=32M ­XX:MaxPermSize=256M ­Djava.rmi.server.hostname= ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.ssl=false ­Dcom.sun.management.jmxremote.port=9000

  46. jvmargs limitations Can't currently handle ­classpath /tmp/foo.jar:/tmp/bar.jar or ­Xrunjdwp:transport=dt_socket,address=10 45

    “-server” currently added by default
  47. jvmargs rules • Cargo-culting can lead to conflicting options •

    Many preventable errors • Why can't this be configurable? “Don't set the max heap smaller than the minimum heap”
  48. jvmargs rules args.add_rule(:max_smaller_than_min) do |args| max_heap = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xmx"].value) min_heap

    = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xms"].value) if max_heap < min_heap raise ArgumentError, "Max heap #{max_heap} too small!" end end
  49. jvmargs rules • rules are run whenever arguments are added

    and when JVMArgs::Args#to_s called • can raise errors, change arg values, or pretty much anything else
  50. fetching dependencies • Need the postgresql JDBC driver • you

    should pull from maven.org • you should use the maven LWRP
  51. maven it include_recipe “maven” maven "postgresql" do group_id “postgresql” version

    “9.2­1002­jdbc4” owner "jvm­lover" dest '/usr/local/tomcat/lib' transitive true end creates /usr/local/tomcat/lib/postgresql.jar
  52. Java <3 XML • Tomcat not so verbose, other containers

    can be much worse, esp. J2EEEEEEEEE • It is really hard to write a DSL for XML because it is arbitrarily complex • Jboss 7 standalone-full.xml is 706 lines long!
  53. Handling XML • templates – limited flexibility • write your

    own DSL (HARD) OR • use template partials (EASY)
  54. context.xml <Context path=”” crossContext=”true”> <Resource name="jdbc/com.teradata.terajms" type=”javax.sql.DataSource” /> <Resource name="jms/ConnectionFactory"

    auth="Container" type="org.apache.activemq.ActiveMQConnectionFactory" /> <Resource name="jms/SampleTopic" physicalName="SAMPLE.TOPIC"/> <Resource name="jms/SampleQueue" type="org.apache.activemq.command.ActiveMQQueue" /> <!­­ Many more resources to follow ­­> </Context> Note: This is heavily abridged
  55. context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>”

    driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <!­­ additional attributes ­­> /> <!­­ continue for n more Resources ­­> </Context>
  56. context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>”

    driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <!­­ additional attributes ­­> /> <!­­ continue for n more Resources ­­> </Context> image: http://www.trenddelacreme.com/2010/02/when-unicorn-vomits-below-belt.html
  57. context.xml.erb w/ partials* <Context path=”” crossContext=”true”> <%= render 'data_source.xml.erb', @template_context

    %> <%= render 'jms_factory.xml.erb', @template_context %> <%= render 'jms_topic.xml.erb', @template_context %> <%= render 'jms_queue.xml.erb', @template_context %> </Context> *introduced in chef11
  58. Monitoring your JVM include_recipe “collectd::generic_jmx” collectd_jmx “myapp” do port 8999

    username “foo” Password “bar” end
  59. Log Rotation and Compression • logrotate or log4j? • logrotate

    has far fewer features • log4j can't manage catalina.out or console.log • java devs like log4j • log4j can't compress log files :( • Why not use both? image: http://logging.apache.org/log4j/1.2/
  60. Rotating catalina.out include “logrotate” logrotate_app "jvm­love" do path "/usr/local/tomcat/logs/catalina.out" options

    [ "compress", "copytruncate" ] rotate 7 create "664 jvm­love jvm­love" end
  61. Cleaning up after logrotate include_recipe “janitor” janitor_logs "/usr/local/tomcat/logs" do include_files

    ["*.log"] max_age 30 min_age 1 action [:compress,:purge] end Compresses all files that do not end in .gz
  62. The Smart Kids Use SLF4J + Logback, which can compress

    files image: http://www.alice1059.com/Smart-kids-make-us-sick/11315931?pid=305825&archive=1
  63. Local Logs are Dumb Logs Send 'em to Logstash •

    Use gelf4j or log4j Socket Appender • Together w/ the log4j or gelf logstash input on logstash indexer
  64. Picking a Logstash Agent 1. Stock Agent, written in Jruby,

    sizeable footprint but can process logs before shipping 2. Lightweight shippers, no processing, ship everything I recommend starting w/ the stock agent
  65. Still Needed • mirror_file LWRP • Keystore LWRP • templating

    “shifting” xml files • Validating XML document with nokogiri before updating a configuration file • more full-featured jvmargs
  66. Thanks To • Xabier de Zuazo (@zuazo) • John Vincent

    (@lusis) • Nathen Harvey • Julian Dunn • Peter Donald
  67. Resources • berkshelf • test-kitchen • ark • docker •

    systemd • java cookbook • maven cookbook • jvmargs • logrotate • chef-janitor • FoodFightShow • chef-collectd