Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Managing Java Applications with Chef

Managing Java Applications with Chef

How to manage java applications with Chef

3b2917349b488b1f7511918ab0c409d4?s=128

Bryan Berry

April 27, 2013
Tweet

More Decks by Bryan Berry

Other Decks in How-to & DIY

Transcript

  1. Showing the JVM Some Love Managing Java Apps with Chef

    Bryan W. Berry, Chefconf 2013
  2. About Me • Sr. DevOps Engineer at Cycle Computing •

    Creator and co-host of FoodFightShow podcast • Author of many java-related cookbooks • bryanwb on twitter, github, IRC • bberry@cyclecomputing.com
  3. We're Hiring! Come see me or Rob Futrick

  4. Hate the Language, Love the Runtime “Java is the new

    C”* image: https://wiki.smu.edu.sg/is200/Java_Virtual_Machine * http://www.slideshare.net/mobile/pcalcado/from-a-monolithic-ruby-on-rails-app-to-the-jvm
  5. The JVM, A World of its Own Windows Linux Ruby

  6. Headaches • Distro package or tarball? • Cargo-culting JAVA_OPTS, CATALINA_OPTS,

    HADOOP_OPTS, etc. • Managing Resource limits • Managing Giant XML files • Monitoring JVMs with JMX • Competing with Log4j to rotate and compress log files
  7. Let's Walk Some Talk • Let's write a simple java

    related cookbook • Oracle JDK 7 • Tomcat as the container • Tweak some kernel settings • Fetch dependencies with Maven • Monitor it with JMX Let's call it jvm-love in honor of Ulf
  8. In the beginning . . . • You need to

    install the Java JDK • The opscode Java cookbook is excellent
  9. Installing a JDK To install openjdk include_recipe “java” To install

    Oracle set[:java][:install_flavor] = “oracle” set[:java][:accept_oracle_download_terms] = true include_recipe “java” To change JDK version node.set[:java][:jdk_version] = '7' # it is 6 by default
  10. Distro Package Cons • Often out-of-date • LFS spreads files

    all over filesystem • Includes shell scripts w/ unwanted side effects • Init script can be ridiculously complicated
  11. FHS can confuse non-professional linux geeks • Config files in

    /etc/tomcat7 • Webapps, logs, libs in /usr/share/tomcat7 • Temp directory in /var/cache/tomcat7 • Various binary commands in /usr/[s]bin/ • logs in /var/lib/tomcat7/logs* *ubuntu 10.10
  12. Freshness • No Tomcat7 in the default CentOS/RHEL repos •

    Ubuntu provides 7.0.30*, ~7 months old • Neither provide packages for Glassfish3 or JBoss > 4.0 • jpackage.org 6.0 provides tomcat7 rpms, sometimes** *as of 26 March 2013 **the site is sometimes down
  13. Installing from (Binary) Tarball • Java Packages are arch-independent, no

    `make` step needed • The download, unpack, link process can be tedious, but there are tools to help
  14. Meet Ark ark “tomcat” do version “7.0.37” checksum “3abc3434..” url

    ”http://..../tomcat.tar.gz” owner “jvm­lover” action :install end
  15. Ark's Aftermath • Tomcat tarball downloaded from Apache site •

    tarball unpacked to /usr/local/tomcat-#{version} • /usr/local/tomcat symlinked to /usr/local/tomcat-#{version} • jvm-lover user made owner of /usr/local/tomcat-#{version}
  16. Ark Issues • You can inadvertently DDOS your apache.org •

    You should use http://apache.mirrors.tds.net but that site only hosts recent releases • Use http://archive.apache.org, but DDOS problem again
  17. Do it Yourself • Host the tarball yourself OR •

    Implement a mirror_file LWRP that randomly selects from a list of Apache mirrors
  18. Upgrading Tomcat w/ Ark • Change the version number ark

    “tomcat” do version “7.0.38” #previously 7.0.37 # other settings as before end /usr/local/tomcat now points to /usr/local/tomcat-7.0.38
  19. JVM Vs. Linux Low level settings that are a must

    1. Maximum number of open files 2. Maximum number of processes (threads) 3. Swapping to disk
  20. Use the Ulimit LWRP include_recipe “ulimit” user_ulimit "jvm­lover" do filehandle_limit

    8192 process_limit 61504 end
  21. Use the Syctl LWRP include_recipe “sysctl” node.override['sysctl_file'] = '/etc/sysctl.d/99­chef.conf' sysctl

    "kernel.swappiness" do value 0 end
  22. Init Scripts You have 3 choices • Upstart • System

    V • Runit* *compiled from source
  23. SystemV • Understood by many • Horrible spaghetti Bash involving

    multiple scripts • It is unix black magic to detach the JVM process from the session & grab its process ID • No process monitoring
  24. Upstart • Easier to write than system V • Some

    naïve process supervision • In my anecdotal experience, not as reliable as systemV • Deprecated on CentOS/RHEL in favor of SystemD
  25. Runit • Rock solid • Easy • Awesome on Ubuntu/Debian

    • No native packages for CentOS* • Not as widely known as sysv or upstart *You can install from source Using opscode's runit cookbook
  26. Serious Supervision • ​Docker and/or SystemD • contain processes with

    LXC namespaces rather than tracking a process ID • Systemd currently available in debian, coming to CentOS/RHEL 7, ubuntu maybe never
  27. Pragmatic Process Supervision • If you primarily use Ubuntu, use

    runit • Think you use debian or will adapt CentOS/RHEL7 quickly, use/wait for systemd or docker • If you have to use sysv or upstart, try to couple it with docker
  28. Scripting like it's 1970 JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap

    start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
  29. Scripting like it's 1970 JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover. pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS

    org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT Let's Use the Process ID to track our running application
  30. Scripting like it's 1970 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap

    start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT Redirect STDOUT and STDERR to the Log file
  31. Daemonization 101 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \

    > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT 1. fork a new process in a new shell 2. detach from that process Step 1 Step 2
  32. PID Magic JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \

    > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT Capture the Process ID
  33. sysv or upstart with Docker docker ­u $JAVA_USER run $SCRIPT

    Replaces su $JAVA_USER ­c $SCRIPT Pretty simple! But with a lot of benefits
  34. You can't escape JAVA_OPTS JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap

    start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
  35. JAVA_OPTS Expands to ­XX:MaxPermSize=256M ­Xmx5G ­Xms2G ­server ­XX:+DisableExplicitGC ­XX:+UseParallelOldGC ­XX:NewRatio=2

    ­XX:SoftRefLRUPolicyMSPerMB=36000 ­Dsun.rmi.dgc.server.gcInterval=3600000 ­XX:+UseBiasedLocking ­Xrs ­Djava.rmi.server.hostname=127.0.0.1 ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.port=9000 And this is a short example!
  36. JAVA_OPTS Expands to ­XX:MaxPermSize=256M ­Xmx5G ­Xms2G ­server ­XX: +DisableExplicitGC ­XX:+UseParallelOldGC

    ­XX:NewRatio=2 ­XX:SoftRefLRUPolicyMSPerMB=36000 ­Dsun.rmi.dgc.server.gcInterval=3600000 ­XX:+UseBiasedLocking ­Xrs ­Djava.rmi.server.hostname=127.0.0.1 ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.port=9000 image: http://brain-cheese.blogspot.it/2009/05/sometimes-i-want-to-scream-like-homer.html
  37. A parsing nightmare 4 kinds of options -server -Xmx5G -XX:MaxPermSize=256M

    -DJSON_ENABLE=true Standard Nonstandard Unstable/Experimental Directives
  38. The JAVA_OPTS are a changin' -Xmx, -Xms, -XX:MaxPermSize can change

    wildly depending on the application and environment but typically have lower bounds image: http://www.tshirtbordello.com/What-The-Frak-T-Shirt
  39. Jvmargs to the rescue • A sane parser for insane

    JVM options • A Ruby gem
  40. jvmargs in action # environments/test.rb default_attributes( “jvm­love” => { “java_opts”

    => [ 'Xint', 'XX:+DisableExplicitGC'] } ) # jvm­love/recipes/default.rb chef_gem 'jvmargs' require 'jvmargs # minimum settings args = JVMArgs::Args.new('Xmx128M', 'Xms128M', 'XX:MaxPermSize=256M') args.add(node['jvm­love']['java_opts'])
  41. jvmargs in action # jvm­love/recipes/default.rb template “/etc/init.d/jvm­love” do source “init.erb”

    mode 00775 variables(:args => args) end
  42. jvmargs in action # templates/default/init.erb SCRIPT=”java #{@args} org.apache.Bootstrap start \

    > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
  43. more examples args = JVMArgs::Args.new("Xmx256M") args.add("Xmx2G") args.add("XX:+DisableExplicitGC") # the args

    are now # "­XX:+DisableExplicitGC ­Xmx2G"
  44. helpers args = JVMArgs::Args.new do jmx true heap_size “40%” permgen

    “256M” newgen “32M” end
  45. results ­Xmx819M ­XX:MaxNewSize=32M ­XX:MaxPermSize=256M ­Djava.rmi.server.hostname=127.0.0.1 ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.ssl=false ­Dcom.sun.management.jmxremote.port=9000

  46. jvmargs limitations Can't currently handle ­classpath /tmp/foo.jar:/tmp/bar.jar or ­Xrunjdwp:transport=dt_socket,address=10 45

    “-server” currently added by default
  47. jvmargs rules • Cargo-culting can lead to conflicting options •

    Many preventable errors • Why can't this be configurable? “Don't set the max heap smaller than the minimum heap”
  48. jvmargs rules args.add_rule(:max_smaller_than_min) do |args| max_heap = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xmx"].value) min_heap

    = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xms"].value) if max_heap < min_heap raise ArgumentError, "Max heap #{max_heap} too small!" end end
  49. jvmargs rules • rules are run whenever arguments are added

    and when JVMArgs::Args#to_s called • can raise errors, change arg values, or pretty much anything else
  50. fetching dependencies • Need the postgresql JDBC driver • you

    should pull from maven.org • you should use the maven LWRP
  51. maven it include_recipe “maven” maven "postgresql" do group_id “postgresql” version

    “9.2­1002­jdbc4” owner "jvm­lover" dest '/usr/local/tomcat/lib' transitive true end creates /usr/local/tomcat/lib/postgresql.jar
  52. Java <3 XML • Tomcat not so verbose, other containers

    can be much worse, esp. J2EEEEEEEEE • It is really hard to write a DSL for XML because it is arbitrarily complex • Jboss 7 standalone-full.xml is 706 lines long!
  53. Handling XML • templates – limited flexibility • write your

    own DSL (HARD) OR • use template partials (EASY)
  54. context.xml <Context path=”” crossContext=”true”> <Resource name="jdbc/com.teradata.terajms" type=”javax.sql.DataSource” /> <Resource name="jms/ConnectionFactory"

    auth="Container" type="org.apache.activemq.ActiveMQConnectionFactory" /> <Resource name="jms/SampleTopic" physicalName="SAMPLE.TOPIC"/> <Resource name="jms/SampleQueue" type="org.apache.activemq.command.ActiveMQQueue" /> <!­­ Many more resources to follow ­­> </Context> Note: This is heavily abridged
  55. context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>”

    driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <!­­ additional attributes ­­> /> <!­­ continue for n more Resources ­­> </Context>
  56. context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>”

    driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <!­­ additional attributes ­­> /> <!­­ continue for n more Resources ­­> </Context> image: http://www.trenddelacreme.com/2010/02/when-unicorn-vomits-below-belt.html
  57. context.xml.erb w/ partials* <Context path=”” crossContext=”true”> <%= render 'data_source.xml.erb', @template_context

    %> <%= render 'jms_factory.xml.erb', @template_context %> <%= render 'jms_topic.xml.erb', @template_context %> <%= render 'jms_queue.xml.erb', @template_context %> </Context> *introduced in chef11
  58. Monitoring your JVM include_recipe “collectd::generic_jmx” collectd_jmx “myapp” do port 8999

    username “foo” Password “bar” end
  59. Log Rotation and Compression • logrotate or log4j? • logrotate

    has far fewer features • log4j can't manage catalina.out or console.log • java devs like log4j • log4j can't compress log files :( • Why not use both? image: http://logging.apache.org/log4j/1.2/
  60. Rotating catalina.out include “logrotate” logrotate_app "jvm­love" do path "/usr/local/tomcat/logs/catalina.out" options

    [ "compress", "copytruncate" ] rotate 7 create "664 jvm­love jvm­love" end
  61. Cleaning up after logrotate include_recipe “janitor” janitor_logs "/usr/local/tomcat/logs" do include_files

    ["*.log"] max_age 30 min_age 1 action [:compress,:purge] end Compresses all files that do not end in .gz
  62. The Smart Kids Use SLF4J + Logback, which can compress

    files image: http://www.alice1059.com/Smart-kids-make-us-sick/11315931?pid=305825&archive=1
  63. Local Logs are Dumb Logs Send 'em to Logstash •

    Use gelf4j or log4j Socket Appender • Together w/ the log4j or gelf logstash input on logstash indexer
  64. Picking a Logstash Agent 1. Stock Agent, written in Jruby,

    sizeable footprint but can process logs before shipping 2. Lightweight shippers, no processing, ship everything I recommend starting w/ the stock agent
  65. Still Needed • mirror_file LWRP • Keystore LWRP • templating

    “shifting” xml files • Validating XML document with nokogiri before updating a configuration file • more full-featured jvmargs
  66. Thanks To • Xabier de Zuazo (@zuazo) • John Vincent

    (@lusis) • Nathen Harvey • Julian Dunn • Peter Donald
  67. Resources • berkshelf • test-kitchen • ark • docker •

    systemd • java cookbook • maven cookbook • jvmargs • logrotate • chef-janitor • FoodFightShow • chef-collectd