Share
Embed
Managing Java Applications with Chef
How to manage java applications with Chef
Bryan Berry
April 27, 2013
Transcript
67 slides
-
Showing the JVM Some Love Managing Java Apps with Chef
Bryan W. Berry, Chefconf 2013 -
About Me • Sr. DevOps Engineer at Cycle Computing •
Creator and co-host of FoodFightShow podcast • Author of many java-related cookbooks • bryanwb on twitter, github, IRC • [email protected] -
Hate the Language, Love the Runtime “Java is the new
C”* image: https://wiki.smu.edu.sg/is200/Java_Virtual_Machine * http://www.slideshare.net/mobile/pcalcado/from-a-monolithic-ruby-on-rails-app-to-the-jvm -
Headaches • Distro package or tarball? • Cargo-culting JAVA_OPTS, CATALINA_OPTS,
HADOOP_OPTS, etc. • Managing Resource limits • Managing Giant XML files • Monitoring JVMs with JMX • Competing with Log4j to rotate and compress log files -
Let's Walk Some Talk • Let's write a simple java
related cookbook • Oracle JDK 7 • Tomcat as the container • Tweak some kernel settings • Fetch dependencies with Maven • Monitor it with JMX Let's call it jvm-love in honor of Ulf -
In the beginning . . . • You need to
install the Java JDK • The opscode Java cookbook is excellent -
Installing a JDK To install openjdk include_recipe “java” To install
Oracle set[:java][:install_flavor] = “oracle” set[:java][:accept_oracle_download_terms] = true include_recipe “java” To change JDK version node.set[:java][:jdk_version] = '7' # it is 6 by default -
Distro Package Cons • Often out-of-date • LFS spreads files
all over filesystem • Includes shell scripts w/ unwanted side effects • Init script can be ridiculously complicated -
FHS can confuse non-professional linux geeks • Config files in
/etc/tomcat7 • Webapps, logs, libs in /usr/share/tomcat7 • Temp directory in /var/cache/tomcat7 • Various binary commands in /usr/[s]bin/ • logs in /var/lib/tomcat7/logs* *ubuntu 10.10 -
Freshness • No Tomcat7 in the default CentOS/RHEL repos •
Ubuntu provides 7.0.30*, ~7 months old • Neither provide packages for Glassfish3 or JBoss > 4.0 • jpackage.org 6.0 provides tomcat7 rpms, sometimes** *as of 26 March 2013 **the site is sometimes down -
Installing from (Binary) Tarball • Java Packages are arch-independent, no
`make` step needed • The download, unpack, link process can be tedious, but there are tools to help -
Meet Ark ark “tomcat” do version “7.0.37” checksum “3abc3434..” url
”http://..../tomcat.tar.gz” owner “jvmlover” action :install end -
Ark's Aftermath • Tomcat tarball downloaded from Apache site •
tarball unpacked to /usr/local/tomcat-#{version} • /usr/local/tomcat symlinked to /usr/local/tomcat-#{version} • jvm-lover user made owner of /usr/local/tomcat-#{version} -
Ark Issues • You can inadvertently DDOS your apache.org •
You should use http://apache.mirrors.tds.net but that site only hosts recent releases • Use http://archive.apache.org, but DDOS problem again -
Do it Yourself • Host the tarball yourself OR •
Implement a mirror_file LWRP that randomly selects from a list of Apache mirrors -
Upgrading Tomcat w/ Ark • Change the version number ark
“tomcat” do version “7.0.38” #previously 7.0.37 # other settings as before end /usr/local/tomcat now points to /usr/local/tomcat-7.0.38 -
JVM Vs. Linux Low level settings that are a must
1. Maximum number of open files 2. Maximum number of processes (threads) 3. Swapping to disk -
Use the Ulimit LWRP include_recipe “ulimit” user_ulimit "jvmlover" do filehandle_limit
8192 process_limit 61504 end -
Use the Syctl LWRP include_recipe “sysctl” node.override['sysctl_file'] = '/etc/sysctl.d/99chef.conf' sysctl
"kernel.swappiness" do value 0 end -
Init Scripts You have 3 choices • Upstart • System
V • Runit* *compiled from source -
SystemV • Understood by many • Horrible spaghetti Bash involving
multiple scripts • It is unix black magic to detach the JVM process from the session & grab its process ID • No process monitoring -
Upstart • Easier to write than system V • Some
naïve process supervision • In my anecdotal experience, not as reliable as systemV • Deprecated on CentOS/RHEL in favor of SystemD -
Runit • Rock solid • Easy • Awesome on Ubuntu/Debian
• No native packages for CentOS* • Not as widely known as sysv or upstart *You can install from source Using opscode's runit cookbook -
Serious Supervision • Docker and/or SystemD • contain processes with
LXC namespaces rather than tracking a process ID • Systemd currently available in debian, coming to CentOS/RHEL 7, ubuntu maybe never -
Pragmatic Process Supervision • If you primarily use Ubuntu, use
runit • Think you use debian or will adapt CentOS/RHEL7 quickly, use/wait for systemd or docker • If you have to use sysv or upstart, try to couple it with docker -
Scripting like it's 1970 JAVA_USER=jvmlover PID_FILE=”/var/run/jvmlover/jvmlover.pid” LOG=”/var/log/jvmlove/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap
start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT -
Scripting like it's 1970 JAVA_USER=jvmlover PID_FILE=”/var/run/jvmlover/jvmlover. pid” LOG=”/var/log/jvmlove/catalina.out” SCRIPT=”java $JAVA_OPTS
org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT Let's Use the Process ID to track our running application -
Scripting like it's 1970 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap
start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT Redirect STDOUT and STDERR to the Log file -
Daemonization 101 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \
> $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT 1. fork a new process in a new shell 2. detach from that process Step 1 Step 2 -
PID Magic JAVA_USER=jvmlover PID_FILE=”/var/run/jvmlover/jvmlover.pid” LOG=”/var/log/jvmlove/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \
> $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT Capture the Process ID -
sysv or upstart with Docker docker u $JAVA_USER run $SCRIPT
Replaces su $JAVA_USER c $SCRIPT Pretty simple! But with a lot of benefits -
You can't escape JAVA_OPTS JAVA_USER=jvmlover PID_FILE=”/var/run/jvmlover/jvmlover.pid” LOG=”/var/log/jvmlove/catalina.out SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap
start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT -
JAVA_OPTS Expands to XX:MaxPermSize=256M Xmx5G Xms2G server XX:+DisableExplicitGC XX:+UseParallelOldGC XX:NewRatio=2
XX:SoftRefLRUPolicyMSPerMB=36000 Dsun.rmi.dgc.server.gcInterval=3600000 XX:+UseBiasedLocking Xrs Djava.rmi.server.hostname=127.0.0.1 Dcom.sun.management.jmxremote Dcom.sun.management.jmxremote.port=9000 And this is a short example! -
JAVA_OPTS Expands to XX:MaxPermSize=256M Xmx5G Xms2G server XX: +DisableExplicitGC XX:+UseParallelOldGC
XX:NewRatio=2 XX:SoftRefLRUPolicyMSPerMB=36000 Dsun.rmi.dgc.server.gcInterval=3600000 XX:+UseBiasedLocking Xrs Djava.rmi.server.hostname=127.0.0.1 Dcom.sun.management.jmxremote Dcom.sun.management.jmxremote.port=9000 image: http://brain-cheese.blogspot.it/2009/05/sometimes-i-want-to-scream-like-homer.html -
A parsing nightmare 4 kinds of options -server -Xmx5G -XX:MaxPermSize=256M
-DJSON_ENABLE=true Standard Nonstandard Unstable/Experimental Directives -
The JAVA_OPTS are a changin' -Xmx, -Xms, -XX:MaxPermSize can change
wildly depending on the application and environment but typically have lower bounds image: http://www.tshirtbordello.com/What-The-Frak-T-Shirt -
Jvmargs to the rescue • A sane parser for insane
JVM options • A Ruby gem -
jvmargs in action # environments/test.rb default_attributes( “jvmlove” => { “java_opts”
=> [ 'Xint', 'XX:+DisableExplicitGC'] } ) # jvmlove/recipes/default.rb chef_gem 'jvmargs' require 'jvmargs # minimum settings args = JVMArgs::Args.new('Xmx128M', 'Xms128M', 'XX:MaxPermSize=256M') args.add(node['jvmlove']['java_opts']) -
jvmargs in action # jvmlove/recipes/default.rb template “/etc/init.d/jvmlove” do source “init.erb”
mode 00775 variables(:args => args) end -
jvmargs in action # templates/default/init.erb SCRIPT=”java #{@args} org.apache.Bootstrap start \
> $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER c $SCRIPT -
more examples args = JVMArgs::Args.new("Xmx256M") args.add("Xmx2G") args.add("XX:+DisableExplicitGC") # the args
are now # "XX:+DisableExplicitGC Xmx2G" -
helpers args = JVMArgs::Args.new do jmx true heap_size “40%” permgen
“256M” newgen “32M” end -
jvmargs limitations Can't currently handle classpath /tmp/foo.jar:/tmp/bar.jar or Xrunjdwp:transport=dt_socket,address=10 45
“-server” currently added by default -
jvmargs rules • Cargo-culting can lead to conflicting options •
Many preventable errors • Why can't this be configurable? “Don't set the max heap smaller than the minimum heap” -
jvmargs rules args.add_rule(:max_smaller_than_min) do |args| max_heap = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xmx"].value) min_heap
= JVMArgs::Util.get_raw_num( args[:nonstandard]["Xms"].value) if max_heap < min_heap raise ArgumentError, "Max heap #{max_heap} too small!" end end -
jvmargs rules • rules are run whenever arguments are added
and when JVMArgs::Args#to_s called • can raise errors, change arg values, or pretty much anything else -
fetching dependencies • Need the postgresql JDBC driver • you
should pull from maven.org • you should use the maven LWRP -
maven it include_recipe “maven” maven "postgresql" do group_id “postgresql” version
“9.21002jdbc4” owner "jvmlover" dest '/usr/local/tomcat/lib' transitive true end creates /usr/local/tomcat/lib/postgresql.jar -
Java <3 XML • Tomcat not so verbose, other containers
can be much worse, esp. J2EEEEEEEEE • It is really hard to write a DSL for XML because it is arbitrarily complex • Jboss 7 standalone-full.xml is 706 lines long! -
Handling XML • templates – limited flexibility • write your
own DSL (HARD) OR • use template partials (EASY) -
context.xml <Context path=”” crossContext=”true”> <Resource name="jdbc/com.teradata.terajms" type=”javax.sql.DataSource” /> <Resource name="jms/ConnectionFactory"
auth="Container" type="org.apache.activemq.ActiveMQConnectionFactory" /> <Resource name="jms/SampleTopic" physicalName="SAMPLE.TOPIC"/> <Resource name="jms/SampleQueue" type="org.apache.activemq.command.ActiveMQQueue" /> <! Many more resources to follow > </Context> Note: This is heavily abridged -
context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>”
driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <! additional attributes > /> <! continue for n more Resources > </Context> -
context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>”
driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <! additional attributes > /> <! continue for n more Resources > </Context> image: http://www.trenddelacreme.com/2010/02/when-unicorn-vomits-below-belt.html -
context.xml.erb w/ partials* <Context path=”” crossContext=”true”> <%= render 'data_source.xml.erb', @template_context
%> <%= render 'jms_factory.xml.erb', @template_context %> <%= render 'jms_topic.xml.erb', @template_context %> <%= render 'jms_queue.xml.erb', @template_context %> </Context> *introduced in chef11 -
Monitoring your JVM include_recipe “collectd::generic_jmx” collectd_jmx “myapp” do port 8999
username “foo” Password “bar” end -
Log Rotation and Compression • logrotate or log4j? • logrotate
has far fewer features • log4j can't manage catalina.out or console.log • java devs like log4j • log4j can't compress log files :( • Why not use both? image: http://logging.apache.org/log4j/1.2/ -
Rotating catalina.out include “logrotate” logrotate_app "jvmlove" do path "/usr/local/tomcat/logs/catalina.out" options
[ "compress", "copytruncate" ] rotate 7 create "664 jvmlove jvmlove" end -
Cleaning up after logrotate include_recipe “janitor” janitor_logs "/usr/local/tomcat/logs" do include_files
["*.log"] max_age 30 min_age 1 action [:compress,:purge] end Compresses all files that do not end in .gz -
The Smart Kids Use SLF4J + Logback, which can compress
files image: http://www.alice1059.com/Smart-kids-make-us-sick/11315931?pid=305825&archive=1 -
Local Logs are Dumb Logs Send 'em to Logstash •
Use gelf4j or log4j Socket Appender • Together w/ the log4j or gelf logstash input on logstash indexer -
Picking a Logstash Agent 1. Stock Agent, written in Jruby,
sizeable footprint but can process logs before shipping 2. Lightweight shippers, no processing, ship everything I recommend starting w/ the stock agent -
Still Needed • mirror_file LWRP • Keystore LWRP • templating
“shifting” xml files • Validating XML document with nokogiri before updating a configuration file • more full-featured jvmargs -
Thanks To • Xabier de Zuazo (@zuazo) • John Vincent
(@lusis) • Nathen Harvey • Julian Dunn • Peter Donald -
Resources • berkshelf • test-kitchen • ark • docker •
systemd • java cookbook • maven cookbook • jvmargs • logrotate • chef-janitor • FoodFightShow • chef-collectd