The Song of JBoss and Chef

9b91521dd3d020edb72521181aa32c9f?s=47 Tomasz
November 22, 2016

The Song of JBoss and Chef

JBoss EAP is a heavy-duty Java application server. It's a perfect match for huge banking applications, but also perfect impedance mismatch with DevOps processes. This is the story how we persuaded the Behemoth to work with Chef automation. The goal was to manage multitude of our JBoss clusters and be able to rebuild the infrastructure on-demand.

First presented on DevOpsDays Warsaw 2016.
https://www.youtube.com/watch?v=Ev3pssBlVI4

9b91521dd3d020edb72521181aa32c9f?s=128

Tomasz

November 22, 2016
Tweet

Transcript

  1. The Song of JBoss and Chef The Song of JBoss

    and Chef DevOpsDays Warsaw 2016 Tomasz Torcz <tomasz.torcz@alior.pl>
  2. Agenda - who am I? - what is JBoss? What

    is Chef? - why is it complicated? - pro & cons
  3. Who am I?

  4. Banks. Banks are special

  5. JBoss? The JBoss Enterprise Application Platform is a Java EE-based

    application server runtime platform used for building, deploying, and hosting highly-transactional Java applications and services.
  6. Chef Configuration management tool written in Ruby and Erlang. It

    uses a pure-Ruby, domain-specific language for writing system configuration "recipes".
  7. The Song of The Song of JBoss and Chef JBoss

    and Chef
  8. The Song of The Song of JBoss and Chef JBoss

    and Chef
  9. Why did we care? - already automated lower levels of

    stack Paweł Bartusch @ https://youtu.be/U50w_ZoQygA - needed for full CI/CD pipeline Waldemar Piszczewiat @ https://youtu.be/9-X8qytNznA - crucial to have accounting git blame
  10. Architecture

  11. Architecture vroom! vrooom!

  12. Workflow - multiple teams, separation of duties - internal developers

    - external vendors
  13. How do you configure JBoss?

  14. 1. Web Console

  15. How do you configure JBoss? 1. Web Console 2. Command

    Line Interface
  16. How do you configure JBoss? 1. Web Console 2. Command

    Line Interface 3. XML files (domain.xml, host*.xml, etc.)
  17. How do you configure JBoss? 1. Web Console 2. Command

    Line Interface 3. XML files (domain.xml, host*.xml, etc.)
  18. JBoss CLI "logging": { … "root-logger": { "handlers": "[\"File_rotating\"]", "level":

    "INFO" ~/bin/jboss-cli.sh --controller=10.14.x.y:9999 --connect –commands='/profile=full- ha/subsystem=logging/root-logger=ROOT:write- attribute(name=handlers, value=["File_rotating"])'
  19. Reading the JBoss output { "outcome" => "failed", "failure-description" =>

    {"host-failure- descriptions" => {"vhost02" => "JBAS014749: Operation handler failed: null"}}, "rolled-back" => true }
  20. Reading the JBoss output eval(JBoss’ output) … success! => {"outcome"=>"failed",

    "failure-description"=>{ "host-failure-descriptions"=>{ "vhost02"=>"JBAS014749: Operation handler failed: null" } }, "rolled-back"=>true}
  21. Reading the JBoss output "extension" => { "org.jboss.as.webservices" => undefined,

    … NameError: undefined local variable or method `undefined' for main:Object
  22. Reading the JBoss output "extension" => { "org.jboss.as.webservices" => undefined,

    … NameError: undefined local variable or method `undefined' for main:Object sed 's/undefined/{}/g' … success!
  23. Reading the JBoss output "content" => [{"hash" => bytes {

    0x87, 0x17, 0x0a, 0xd7, 0xff, 0x72 }}] SyntaxError: (eval):1: syntax error, unexpected =>, expecting end-of-input "content" => [{"hash" => bytes { 0x87, 0x... ^ sed 's/{"hash" => bytes {//g' … success!
  24. JBoss output massaging function c.gsub!('=> undefined', '=> {}') c.gsub!(/([[:blank:]]*)(-?\b[0-9]+)L\b(,?)/, '\1\2\3')

    # 1234L -> "1234" c.gsub!('expression "', '"expression ') c.gsub!('{"hash" => bytes {', '') # \_ 'hash => bytes' to array c.gsub!(/([ \t]+)}}(\],?$)/,'\1\2') # / c.gsub!(/'/, '\\\\' + "'") # don't ask :/ c.gsub!(/([[:blank:]]*\()(". +"[[:blank:]]=\>[[:blank:]]".+")\)(,?)/, '\2\3') c.gsub!(/(.+\[)\((.+)\)(\].+)/,'\1\2\3') c.gsub!(/"/, '\'')
  25. JBoss output massaging function commit 3b54e51a0fc9149330e32c42fd784ad3b20850ad Author: Bartosz Trudnowski <bartosz.trudnowski@ge.com>

    Date: Tue Feb 23 10:00:58 2016 +0100 - c.gsub!(/([[:blank:]]\()(".+"[[:blank:]]=\>[[:blank:]]".+")\)(,?)/, '\2\3') + c.gsub!(/([[:blank:]]*\()(".+"[[:blank:]]=\>[[:blank:]]".+")\)(,?)/, '\2\3')
  26. JBoss output format

  27. Other JBoss annoyances - config keywords incoherence - punctuation frenzy

  28. The good and the bad What lessons did we learn?

  29. PROD cluster

  30. PROD cluster 1 hour later

  31. Handover bph_jboss_srv bph_jboss_ml bph_jboss_app

  32. Handover – number of commits bph_jboss_srv bph_jboss_ml bph_jboss_app 80 Tomasz

    Torcz 254 Tomasz Torcz 50 Tomasz Torcz 9 Waldemar Piszczewiat 67 Bartosz Trudnowski 10 Bartosz Trudnowski 4 Krzysztof Kowalczyk 17 Waldemar Piszczewiat 7 Waldemar Piszczewiat 4 Bartosz Trudnowski 4 Bartosz Dempc 6 Krzysztof Kowalczyk 3 Waldemar Buczak 1 Waldemar Buczak 1 Grzegorz Prusik 1 Grzegorz Krysik
  33. Data bags – number of commits dev test prod 218

    Waldemar Piszczewiat 273 Waldemar Piszczewiat 40 Bartosz Dempc 53 Tomasz Torcz 194 Bartosz Dempc 17 Paweł Rodziewicz 8 Bartosz Trudnowski 107 Grzegorz Prusik 13 Grzegorz Prusik 5 Technical CICD account 49 Rafał Markowski 6 Michał Szor 3 Waldemar Buczak 43 Grzegorz Krysik 6 Waldemar Piszczewiat 2 Bartosz Dempc 24 Grzegorz Słaby 5 Grzegorz Słaby … …
  34. Reading the Chef output * execute[Updating SG01 property spring.profiles.active to

    cbpm,sit,mockReport,localStorage] action run (skipped due to not_if)
  35. Managable config - many similar, but maybe different clusters? -

    json-diff - testing, moving configuration - copy&paste - easy to revert
  36. Diverging templates cicd-dev-apps/dev/ … /GsbTytan.properties.erb cicd-uat-apps/dk1/ … /GsbTytan.properties.erb cicd-uat-apps/dk2/ …

    /GsbTytan.properties.erb cicd-uat-apps/tg/ … /GsbTytan.properties.erb cicd-sit-apps/sit/ … /GsbTytan.properties.erb cicd-sit-apps/sitstab/ … /GsbTytan.properties.erb cicd-prod-apps/prod/ … /GsbTytan.properties.erb - moved to external git repos
  37. The Song of JBoss and Chef The Song of JBoss

    and Chef DevOpsDays Warsaw 2016 Tomasz Torcz <tomasz.torcz@alior.pl>