Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Managing Java Applications with Chef

Managing Java Applications with Chef

How to manage java applications with Chef

Bryan Berry

April 27, 2013
Tweet

More Decks by Bryan Berry

Other Decks in How-to & DIY

Transcript

  1. About Me • Sr. DevOps Engineer at Cycle Computing •

    Creator and co-host of FoodFightShow podcast • Author of many java-related cookbooks • bryanwb on twitter, github, IRC • [email protected]
  2. Hate the Language, Love the Runtime “Java is the new

    C”* image: https://wiki.smu.edu.sg/is200/Java_Virtual_Machine * http://www.slideshare.net/mobile/pcalcado/from-a-monolithic-ruby-on-rails-app-to-the-jvm
  3. Headaches • Distro package or tarball? • Cargo-culting JAVA_OPTS, CATALINA_OPTS,

    HADOOP_OPTS, etc. • Managing Resource limits • Managing Giant XML files • Monitoring JVMs with JMX • Competing with Log4j to rotate and compress log files
  4. Let's Walk Some Talk • Let's write a simple java

    related cookbook • Oracle JDK 7 • Tomcat as the container • Tweak some kernel settings • Fetch dependencies with Maven • Monitor it with JMX Let's call it jvm-love in honor of Ulf
  5. In the beginning . . . • You need to

    install the Java JDK • The opscode Java cookbook is excellent
  6. Installing a JDK To install openjdk include_recipe “java” To install

    Oracle set[:java][:install_flavor] = “oracle” set[:java][:accept_oracle_download_terms] = true include_recipe “java” To change JDK version node.set[:java][:jdk_version] = '7' # it is 6 by default
  7. Distro Package Cons • Often out-of-date • LFS spreads files

    all over filesystem • Includes shell scripts w/ unwanted side effects • Init script can be ridiculously complicated
  8. FHS can confuse non-professional linux geeks • Config files in

    /etc/tomcat7 • Webapps, logs, libs in /usr/share/tomcat7 • Temp directory in /var/cache/tomcat7 • Various binary commands in /usr/[s]bin/ • logs in /var/lib/tomcat7/logs* *ubuntu 10.10
  9. Freshness • No Tomcat7 in the default CentOS/RHEL repos •

    Ubuntu provides 7.0.30*, ~7 months old • Neither provide packages for Glassfish3 or JBoss > 4.0 • jpackage.org 6.0 provides tomcat7 rpms, sometimes** *as of 26 March 2013 **the site is sometimes down
  10. Installing from (Binary) Tarball • Java Packages are arch-independent, no

    `make` step needed • The download, unpack, link process can be tedious, but there are tools to help
  11. Meet Ark ark “tomcat” do version “7.0.37” checksum “3abc3434..” url

    ”http://..../tomcat.tar.gz” owner “jvm­lover” action :install end
  12. Ark's Aftermath • Tomcat tarball downloaded from Apache site •

    tarball unpacked to /usr/local/tomcat-#{version} • /usr/local/tomcat symlinked to /usr/local/tomcat-#{version} • jvm-lover user made owner of /usr/local/tomcat-#{version}
  13. Ark Issues • You can inadvertently DDOS your apache.org •

    You should use http://apache.mirrors.tds.net but that site only hosts recent releases • Use http://archive.apache.org, but DDOS problem again
  14. Do it Yourself • Host the tarball yourself OR •

    Implement a mirror_file LWRP that randomly selects from a list of Apache mirrors
  15. Upgrading Tomcat w/ Ark • Change the version number ark

    “tomcat” do version “7.0.38” #previously 7.0.37 # other settings as before end /usr/local/tomcat now points to /usr/local/tomcat-7.0.38
  16. JVM Vs. Linux Low level settings that are a must

    1. Maximum number of open files 2. Maximum number of processes (threads) 3. Swapping to disk
  17. Init Scripts You have 3 choices • Upstart • System

    V • Runit* *compiled from source
  18. SystemV • Understood by many • Horrible spaghetti Bash involving

    multiple scripts • It is unix black magic to detach the JVM process from the session & grab its process ID • No process monitoring
  19. Upstart • Easier to write than system V • Some

    naïve process supervision • In my anecdotal experience, not as reliable as systemV • Deprecated on CentOS/RHEL in favor of SystemD
  20. Runit • Rock solid • Easy • Awesome on Ubuntu/Debian

    • No native packages for CentOS* • Not as widely known as sysv or upstart *You can install from source Using opscode's runit cookbook
  21. Serious Supervision • ​Docker and/or SystemD • contain processes with

    LXC namespaces rather than tracking a process ID • Systemd currently available in debian, coming to CentOS/RHEL 7, ubuntu maybe never
  22. Pragmatic Process Supervision • If you primarily use Ubuntu, use

    runit • Think you use debian or will adapt CentOS/RHEL7 quickly, use/wait for systemd or docker • If you have to use sysv or upstart, try to couple it with docker
  23. Scripting like it's 1970 JAVA_USER=jvm­lover PID_FILE=”/var/run/jvm­lover/jvm­lover. pid” LOG=”/var/log/jvm­love/catalina.out” SCRIPT=”java $JAVA_OPTS

    org.apache.Bootstrap start \ > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER ­c $SCRIPT Let's Use the Process ID to track our running application
  24. Daemonization 101 JAVA_USER=jvm-lover PID_FILE=”/var/run/jvm-lover/jvm-lover.pid” LOG=”/var/log/jvm-love/catalina.out” SCRIPT=”java $JAVA_OPTS org.apache.Bootstrap start \

    > $LOG 2>&1 & echo $! > $PID_FILE” su $JAVA_USER -c $SCRIPT 1. fork a new process in a new shell 2. detach from that process Step 1 Step 2
  25. sysv or upstart with Docker docker ­u $JAVA_USER run $SCRIPT

    Replaces su $JAVA_USER ­c $SCRIPT Pretty simple! But with a lot of benefits
  26. JAVA_OPTS Expands to ­XX:MaxPermSize=256M ­Xmx5G ­Xms2G ­server ­XX:+DisableExplicitGC ­XX:+UseParallelOldGC ­XX:NewRatio=2

    ­XX:SoftRefLRUPolicyMSPerMB=36000 ­Dsun.rmi.dgc.server.gcInterval=3600000 ­XX:+UseBiasedLocking ­Xrs ­Djava.rmi.server.hostname=127.0.0.1 ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.port=9000 And this is a short example!
  27. JAVA_OPTS Expands to ­XX:MaxPermSize=256M ­Xmx5G ­Xms2G ­server ­XX: +DisableExplicitGC ­XX:+UseParallelOldGC

    ­XX:NewRatio=2 ­XX:SoftRefLRUPolicyMSPerMB=36000 ­Dsun.rmi.dgc.server.gcInterval=3600000 ­XX:+UseBiasedLocking ­Xrs ­Djava.rmi.server.hostname=127.0.0.1 ­Dcom.sun.management.jmxremote ­Dcom.sun.management.jmxremote.port=9000 image: http://brain-cheese.blogspot.it/2009/05/sometimes-i-want-to-scream-like-homer.html
  28. A parsing nightmare 4 kinds of options -server -Xmx5G -XX:MaxPermSize=256M

    -DJSON_ENABLE=true Standard Nonstandard Unstable/Experimental Directives
  29. The JAVA_OPTS are a changin' -Xmx, -Xms, -XX:MaxPermSize can change

    wildly depending on the application and environment but typically have lower bounds image: http://www.tshirtbordello.com/What-The-Frak-T-Shirt
  30. jvmargs in action # environments/test.rb default_attributes( “jvm­love” => { “java_opts”

    => [ 'Xint', 'XX:+DisableExplicitGC'] } ) # jvm­love/recipes/default.rb chef_gem 'jvmargs' require 'jvmargs # minimum settings args = JVMArgs::Args.new('Xmx128M', 'Xms128M', 'XX:MaxPermSize=256M') args.add(node['jvm­love']['java_opts'])
  31. jvmargs rules • Cargo-culting can lead to conflicting options •

    Many preventable errors • Why can't this be configurable? “Don't set the max heap smaller than the minimum heap”
  32. jvmargs rules args.add_rule(:max_smaller_than_min) do |args| max_heap = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xmx"].value) min_heap

    = JVMArgs::Util.get_raw_num( args[:nonstandard]["Xms"].value) if max_heap < min_heap raise ArgumentError, "Max heap #{max_heap} too small!" end end
  33. jvmargs rules • rules are run whenever arguments are added

    and when JVMArgs::Args#to_s called • can raise errors, change arg values, or pretty much anything else
  34. fetching dependencies • Need the postgresql JDBC driver • you

    should pull from maven.org • you should use the maven LWRP
  35. maven it include_recipe “maven” maven "postgresql" do group_id “postgresql” version

    “9.2­1002­jdbc4” owner "jvm­lover" dest '/usr/local/tomcat/lib' transitive true end creates /usr/local/tomcat/lib/postgresql.jar
  36. Java <3 XML • Tomcat not so verbose, other containers

    can be much worse, esp. J2EEEEEEEEE • It is really hard to write a DSL for XML because it is arbitrarily complex • Jboss 7 standalone-full.xml is 706 lines long!
  37. Handling XML • templates – limited flexibility • write your

    own DSL (HARD) OR • use template partials (EASY)
  38. context.xml <Context path=”” crossContext=”true”> <Resource name="jdbc/com.teradata.terajms" type=”javax.sql.DataSource” /> <Resource name="jms/ConnectionFactory"

    auth="Container" type="org.apache.activemq.ActiveMQConnectionFactory" /> <Resource name="jms/SampleTopic" physicalName="SAMPLE.TOPIC"/> <Resource name="jms/SampleQueue" type="org.apache.activemq.command.ActiveMQQueue" /> <!­­ Many more resources to follow ­­> </Context> Note: This is heavily abridged
  39. context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>”

    driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <!­­ additional attributes ­­> /> <!­­ continue for n more Resources ­­> </Context>
  40. context.xml.erb <Context path=”” crossContext=”true”> <Resource name="jdbc/<%= @resource[:data_source][:id]” type=”<%= @resource[:data_source][:type] %>”

    driverClassName=”<%= @resource[:data_source][:driver] %>” url=”<%= @resource[:data_source][:url] %>” username=”<%= @resource[:data_source][:username] %>” <!­­ additional attributes ­­> /> <!­­ continue for n more Resources ­­> </Context> image: http://www.trenddelacreme.com/2010/02/when-unicorn-vomits-below-belt.html
  41. context.xml.erb w/ partials* <Context path=”” crossContext=”true”> <%= render 'data_source.xml.erb', @template_context

    %> <%= render 'jms_factory.xml.erb', @template_context %> <%= render 'jms_topic.xml.erb', @template_context %> <%= render 'jms_queue.xml.erb', @template_context %> </Context> *introduced in chef11
  42. Log Rotation and Compression • logrotate or log4j? • logrotate

    has far fewer features • log4j can't manage catalina.out or console.log • java devs like log4j • log4j can't compress log files :( • Why not use both? image: http://logging.apache.org/log4j/1.2/
  43. Cleaning up after logrotate include_recipe “janitor” janitor_logs "/usr/local/tomcat/logs" do include_files

    ["*.log"] max_age 30 min_age 1 action [:compress,:purge] end Compresses all files that do not end in .gz
  44. The Smart Kids Use SLF4J + Logback, which can compress

    files image: http://www.alice1059.com/Smart-kids-make-us-sick/11315931?pid=305825&archive=1
  45. Local Logs are Dumb Logs Send 'em to Logstash •

    Use gelf4j or log4j Socket Appender • Together w/ the log4j or gelf logstash input on logstash indexer
  46. Picking a Logstash Agent 1. Stock Agent, written in Jruby,

    sizeable footprint but can process logs before shipping 2. Lightweight shippers, no processing, ship everything I recommend starting w/ the stock agent
  47. Still Needed • mirror_file LWRP • Keystore LWRP • templating

    “shifting” xml files • Validating XML document with nokogiri before updating a configuration file • more full-featured jvmargs
  48. Thanks To • Xabier de Zuazo (@zuazo) • John Vincent

    (@lusis) • Nathen Harvey • Julian Dunn • Peter Donald
  49. Resources • berkshelf • test-kitchen • ark • docker •

    systemd • java cookbook • maven cookbook • jvmargs • logrotate • chef-janitor • FoodFightShow • chef-collectd