Managing Java Applications with Chef

How to manage java applications with Chef

Transcript

1. Showing the JVM Some Love Managing Java Apps with Chef Bryan W. Berry, Chefconf 2013
2. About Me • Sr. DevOps Engineer at Cycle Computing • Creator and co-host of FoodFightShow podcast • Author of many java-related cookbooks • bryanwb on twitter, github, IRC • bberry@cyclecomputing.com
3. We're Hiring! Come see me or Rob Futrick
4. Hate the Language, Love the Runtime “Java is the new C”* image: https://wiki.smu.edu.sg/is200/Java_Virtual_Machine * http://www.slideshare.net/mobile/pcalcado/from-a-monolithic-ruby-on-rails-app-to-the-jvm
5. The JVM, A World of its Own Windows Linux Ruby
6. Headaches • Distro package or tarball? • Cargo-culting JAVA_OPTS, CATALINA_OPTS, HADOOP_OPTS, etc. • Managing Resource limits • Managing Giant XML files • Monitoring JVMs with JMX • Competing with Log4j to rotate and compress log files
7. Let's Walk Some Talk • Let's write a simple java related cookbook • Oracle JDK 7 • Tomcat as the container • Tweak some kernel settings • Fetch dependencies with Maven • Monitor it with JMX Let's call it jvm-love in honor of Ulf
8. In the beginning . . . • You need to install the Java JDK • The opscode Java cookbook is excellent
9. Installing a JDK To install openjdk include_recipe “java” To install Oracle set[:java][:install_flavor] = “oracle” set[:java][:accept_oracle_download_terms] = true include_recipe “java” To change JDK version node.set[:java][:jdk_version] = '7' # it is 6 by default
10. Distro Package Cons • Often out-of-date • LFS spreads files all over filesystem • Includes shell scripts w/ unwanted side effects • Init script can be ridiculously complicated
11. FHS can confuse non-professional linux geeks • Config files in /etc/tomcat7 • Webapps, logs, libs in /usr/share/tomcat7 • Temp directory in /var/cache/tomcat7 • Various binary commands in /usr/[s]bin/ • logs in /var/lib/tomcat7/logs* *ubuntu 10.10
12. Freshness • No Tomcat7 in the default CentOS/RHEL repos • Ubuntu provides 7.0.30*, ~7 months old • Neither provide packages for Glassfish3 or JBoss > 4.0 • jpackage.org 6.0 provides tomcat7 rpms, sometimes** *as of 26 March 2013 **the site is sometimes down
13. Installing from (Binary) Tarball • Java Packages are arch-independent, no `make` step needed • The download, unpack, link process can be tedious, but there are tools to help
14. Meet Ark ark “tomcat” do version “7.0.37” checksum “3abc3434..” url ”http://..../tomcat.tar.gz” owner “jvm­lover” action :install end
15. Ark's Aftermath • Tomcat tarball downloaded from Apache site • tarball unpacked to /usr/local/tomcat-#{version} • /usr/local/tomcat symlinked to /usr/local/tomcat-#{version} • jvm-lover user made owner of /usr/local/tomcat-#{version}
16. Ark Issues • You can inadvertently DDOS your apache.org • You should use http://apache.mirrors.tds.net but that site only hosts recent releases • Use http://archive.apache.org, but DDOS problem again
17. Do it Yourself • Host the tarball yourself OR • Implement a mirror_file LWRP that randomly selects from a list of Apache mirrors
18. Upgrading Tomcat w/ Ark • Change the version number ark “tomcat” do version “7.0.38” #previously 7.0.37 # other settings as before end /usr/local/tomcat now points to /usr/local/tomcat-7.0.38
19. JVM Vs. Linux Low level settings that are a must 1. Maximum number of open files 2. Maximum number of processes (threads) 3. Swapping to disk
20. Use the Ulimit LWRP include_recipe “ulimit” user_ulimit "jvm­lover" do filehandle_limit 8192 process_limit 61504 end
21. Use the Syctl LWRP include_recipe “sysctl” node.override['sysctl_file'] = '/etc/sysctl.d/99­chef.conf' sysctl "kernel.swappiness" do value 0 end
22. Init Scripts You have 3 choices • Upstart • System V • Runit* *compiled from source
23. SystemV • Understood by many • Horrible spaghetti Bash involving multiple scripts • It is unix black magic to detach the JVM process from the session & grab its process ID • No process monitoring
24. Upstart • Easier to write than system V • Some naïve process supervision • In my anecdotal experience, not as reliable as systemV • Deprecated on CentOS/RHEL in favor of SystemD
25. Runit • Rock solid • Easy • Awesome on Ubuntu/Debian • No native packages for CentOS* • Not as widely known as sysv or upstart *You can install from source Using opscode's runit cookbook
26. Serious Supervision • ​Docker and/or SystemD • contain processes with LXC namespaces rather than tracking a process ID • Systemd currently available in debian, coming to CentOS/RHEL 7, ubuntu maybe never
27. Pragmatic Process Supervision • If you primarily use Ubuntu, use runit • Think you use debian or will adapt CentOS/RHEL7 quickly, use/wait for systemd or docker • If you have to use sysv or upstart, try to couple it with docker
28. Scripting like it's 1970 JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
29. Scripting like it's 1970 JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover. pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT Let's Use the Process ID to track our running application
30. Scripting like it's 1970 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT Redirect STDOUT and STDERR to the Log file
31. Daemonization 101 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT 1. fork a new process in a new shell 2. detach from that process Step 1 Step 2
32. PID Magic JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT Capture the Process ID
33. sysv or upstart with Docker docker ­u $JAVA_USER run $SCRIPT Replaces su $JAVA_USER ­c $SCRIPT Pretty simple! But with a lot of benefits
34. You can't escape JAVA_OPTS JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover.pid” LOG=”/var/log/jvm­love/catalina.out SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
35. JAVA_OPTS Expands to ­XX:MaxPermSize=256M ­Xmx5G ­Xms2G ­server ­XX:+DisableExplicitGC ­XX:+UseParallelOldGC ­XX:NewRatio=2 ­XX:SoftRefLRUPolicyMSPerMB=36000 ­Dsun.rmi.dgc.server.gcInterval=3600000 ­XX:+UseBiasedLocking ­Xrs ­Djava.rmi.server.hostname=127.0.0.1 ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.port=9000 And this is a short example!
36. JAVA_OPTS Expands to ­XX:MaxPermSize=256M ­Xmx5G ­Xms2G ­server ­XX: +DisableExplicitGC ­XX:+UseParallelOldGC ­XX:NewRatio=2 ­XX:SoftRefLRUPolicyMSPerMB=36000 ­Dsun.rmi.dgc.server.gcInterval=3600000 ­XX:+UseBiasedLocking ­Xrs ­Djava.rmi.server.hostname=127.0.0.1 ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.port=9000 image: http://brain-cheese.blogspot.it/2009/05/sometimes-i-want-to-scream-like-homer.html
37. A parsing nightmare 4 kinds of options -server -Xmx5G -XX:MaxPermSize=256M -DJSON_ENABLE=true Standard Nonstandard Unstable/Experimental Directives
38. The JAVA_OPTS are a changin' -Xmx, -Xms, -XX:MaxPermSize can change wildly depending on the application and environment but typically have lower bounds image: http://www.tshirtbordello.com/What-The-Frak-T-Shirt
39. Jvmargs to the rescue • A sane parser for insane JVM options • A Ruby gem
40. jvmargs in action # environments/test.rb default_attributes( “jvm­love” => { “java_opts” => [ 'Xint', 'XX:+DisableExplicitGC'] } ) # jvm­love/recipes/default.rb chef_gem 'jvmargs' require 'jvmargs # minimum settings args = JVMArgs::Args.new('Xmx128M', 'Xms128M', 'XX:MaxPermSize=256M') args.add(node['jvm­love']['java_opts'])
41. jvmargs in action # jvm­love/recipes/default.rb template “/etc/init.d/jvm­love” do source “init.erb” mode 00775 variables(:args => args) end
42. jvmargs in action # templates/default/init.erb SCRIPT=”java #{@args} org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT
43. more examples args = JVMArgs::Args.new("Xmx256M") args.add("Xmx2G") args.add("XX:+DisableExplicitGC") # the args are now # "­XX:+DisableExplicitGC ­Xmx2G"
44. helpers args = JVMArgs::Args.new do jmx true heap_size “40%” permgen “256M” newgen “32M” end
45. results ­Xmx819M ­XX:MaxNewSize=32M ­XX:MaxPermSize=256M ­Djava.rmi.server.hostname=127.0.0.1 ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.ssl=false ­Dcom.sun.management.jmxremote.port=9000
46. jvmargs limitations Can't currently handle ­classpath /tmp/foo.jar:/tmp/bar.jar or ­Xrunjdwp:transport=dt_socket,address=10 45 “-server” currently added by default
47. jvmargs rules • Cargo-culting can lead to conflicting options • Many preventable errors • Why can't this be configurable? “Don't set the max heap smaller than the minimum heap”
48. jvmargs rules args.add_rule(:max_smaller_than_min) do |args| max_heap = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xmx"].value) min_heap = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xms"].value) if max_heap < min_heap raise ArgumentError, "Max heap #{max_heap} too small!" end end
49. jvmargs rules • rules are run whenever arguments are added and when JVMArgs::Args#to_s called • can raise errors, change arg values, or pretty much anything else
50. fetching dependencies • Need the postgresql JDBC driver • you should pull from maven.org • you should use the maven LWRP
51. maven it include_recipe “maven” maven "postgresql" do group_id “postgresql” version “9.2­1002­jdbc4” owner "jvm­lover" dest '/usr/local/tomcat/lib' transitive true end creates /usr/local/tomcat/lib/postgresql.jar
52. Java <3 XML • Tomcat not so verbose, other containers can be much worse, esp. J2EEEEEEEEE • It is really hard to write a DSL for XML because it is arbitrarily complex • Jboss 7 standalone-full.xml is 706 lines long!
53. Handling XML • templates – limited flexibility • write your own DSL (HARD) OR • use template partials (EASY)
54. context.xml <Context path=”” crossContext=”true”> <Resource name="jdbc/com.teradata.terajms" type=”javax.sql.DataSource” /> <Resource name="jms/ConnectionFactory" auth="Container" type="org.apache.activemq.ActiveMQConnectionFactory" /> <Resource name="jms/SampleTopic" physicalName="SAMPLE.TOPIC"/> <Resource name="jms/SampleQueue" type="org.apache.activemq.command.ActiveMQQueue" /> <!­­ Many more resources to follow ­­> </Context> Note: This is heavily abridged
55. context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>” driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <!­­ additional attributes ­­> /> <!­­ continue for n more Resources ­­> </Context>
56. context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>” driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <!­­ additional attributes ­­> /> <!­­ continue for n more Resources ­­> </Context> image: http://www.trenddelacreme.com/2010/02/when-unicorn-vomits-below-belt.html
57. context.xml.erb w/ partials* <Context path=”” crossContext=”true”> <%= render 'data_source.xml.erb', @template_context %> <%= render 'jms_factory.xml.erb', @template_context %> <%= render 'jms_topic.xml.erb', @template_context %> <%= render 'jms_queue.xml.erb', @template_context %> </Context> *introduced in chef11
58. Monitoring your JVM include_recipe “collectd::generic_jmx” collectd_jmx “myapp” do port 8999 username “foo” Password “bar” end
59. Log Rotation and Compression • logrotate or log4j? • logrotate has far fewer features • log4j can't manage catalina.out or console.log • java devs like log4j • log4j can't compress log files :( • Why not use both? image: http://logging.apache.org/log4j/1.2/
60. Rotating catalina.out include “logrotate” logrotate_app "jvm­love" do path "/usr/local/tomcat/logs/catalina.out" options [ "compress", "copytruncate" ] rotate 7 create "664 jvm­love jvm­love" end
61. Cleaning up after logrotate include_recipe “janitor” janitor_logs "/usr/local/tomcat/logs" do include_files ["*.log"] max_age 30 min_age 1 action [:compress,:purge] end Compresses all files that do not end in .gz
62. The Smart Kids Use SLF4J + Logback, which can compress files image: http://www.alice1059.com/Smart-kids-make-us-sick/11315931?pid=305825&archive=1
63. Local Logs are Dumb Logs Send 'em to Logstash • Use gelf4j or log4j Socket Appender • Together w/ the log4j or gelf logstash input on logstash indexer
64. Picking a Logstash Agent 1. Stock Agent, written in Jruby, sizeable footprint but can process logs before shipping 2. Lightweight shippers, no processing, ship everything I recommend starting w/ the stock agent
65. Still Needed • mirror_file LWRP • Keystore LWRP • templating “shifting” xml files • Validating XML document with nokogiri before updating a configuration file • more full-featured jvmargs
66. Thanks To • Xabier de Zuazo (@zuazo) • John Vincent (@lusis) • Nathen Harvey • Julian Dunn • Peter Donald
67. Resources • berkshelf • test-kitchen • ark • docker • systemd • java cookbook • maven cookbook • jvmargs • logrotate • chef-janitor • FoodFightShow • chef-collectd