Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Song of JBoss and Chef

Tomasz
November 22, 2016

The Song of JBoss and Chef

JBoss EAP is a heavy-duty Java application server. It's a perfect match for huge banking applications, but also perfect impedance mismatch with DevOps processes. This is the story how we persuaded the Behemoth to work with Chef automation. The goal was to manage multitude of our JBoss clusters and be able to rebuild the infrastructure on-demand.

First presented on DevOpsDays Warsaw 2016.
https://www.youtube.com/watch?v=Ev3pssBlVI4

Tomasz

November 22, 2016
Tweet

More Decks by Tomasz

Other Decks in Technology

Transcript

  1. The Song of JBoss and Chef
    The Song of JBoss and Chef
    DevOpsDays Warsaw 2016
    Tomasz Torcz

    View full-size slide

  2. Agenda
    - who am I?
    - what is JBoss? What is Chef?
    - why is it complicated?
    - pro & cons

    View full-size slide

  3. Banks. Banks are special

    View full-size slide

  4. JBoss?
    The JBoss Enterprise Application Platform is a
    Java EE-based application server runtime
    platform used for building, deploying, and hosting
    highly-transactional Java applications and services.

    View full-size slide

  5. Chef
    Configuration management tool written in Ruby
    and Erlang. It uses a pure-Ruby, domain-specific
    language for writing system configuration
    "recipes".

    View full-size slide

  6. The Song of
    The Song of
    JBoss and Chef
    JBoss and Chef

    View full-size slide

  7. The Song of
    The Song of
    JBoss and Chef
    JBoss and Chef

    View full-size slide

  8. Why did we care?
    - already automated lower levels of stack
    Paweł Bartusch @ https://youtu.be/U50w_ZoQygA
    - needed for full CI/CD pipeline
    Waldemar Piszczewiat @ https://youtu.be/9-X8qytNznA
    - crucial to have accounting
    git blame

    View full-size slide

  9. Architecture

    View full-size slide

  10. Architecture
    vroom!
    vrooom!

    View full-size slide

  11. Workflow
    - multiple teams, separation of duties
    - internal developers
    - external vendors

    View full-size slide

  12. How do you configure JBoss?

    View full-size slide

  13. 1. Web Console

    View full-size slide

  14. How do you configure JBoss?
    1. Web Console
    2. Command Line Interface

    View full-size slide

  15. How do you configure JBoss?
    1. Web Console
    2. Command Line Interface
    3. XML files (domain.xml, host*.xml, etc.)

    View full-size slide

  16. How do you configure JBoss?
    1. Web Console
    2. Command Line Interface
    3. XML files (domain.xml, host*.xml, etc.)

    View full-size slide

  17. JBoss CLI
    "logging": {

    "root-logger": {
    "handlers": "[\"File_rotating\"]",
    "level": "INFO"
    ~/bin/jboss-cli.sh --controller=10.14.x.y:9999
    --connect –commands='/profile=full-
    ha/subsystem=logging/root-logger=ROOT:write-
    attribute(name=handlers, value=["File_rotating"])'

    View full-size slide

  18. Reading the JBoss output
    {
    "outcome" => "failed",
    "failure-description" => {"host-failure-
    descriptions" => {"vhost02" => "JBAS014749:
    Operation handler failed: null"}},
    "rolled-back" => true
    }

    View full-size slide

  19. Reading the JBoss output
    eval(JBoss’ output) … success!
    => {"outcome"=>"failed",
    "failure-description"=>{
    "host-failure-descriptions"=>{
    "vhost02"=>"JBAS014749: Operation handler
    failed: null"
    }
    },
    "rolled-back"=>true}

    View full-size slide

  20. Reading the JBoss output
    "extension" => {
    "org.jboss.as.webservices" => undefined, …
    NameError: undefined local variable or
    method `undefined' for main:Object

    View full-size slide

  21. Reading the JBoss output
    "extension" => {
    "org.jboss.as.webservices" => undefined, …
    NameError: undefined local variable or
    method `undefined' for main:Object
    sed 's/undefined/{}/g' … success!

    View full-size slide

  22. Reading the JBoss output
    "content" => [{"hash" => bytes { 0x87, 0x17,
    0x0a, 0xd7, 0xff, 0x72 }}]
    SyntaxError: (eval):1: syntax error,
    unexpected =>, expecting end-of-input
    "content" => [{"hash" => bytes { 0x87, 0x...
    ^
    sed 's/{"hash" => bytes {//g' … success!

    View full-size slide

  23. JBoss output massaging function
    c.gsub!('=> undefined', '=> {}')
    c.gsub!(/([[:blank:]]*)(-?\b[0-9]+)L\b(,?)/,
    '\1\2\3') # 1234L -> "1234"
    c.gsub!('expression "', '"expression ')
    c.gsub!('{"hash" => bytes {', '') # \_ 'hash =>
    bytes' to array
    c.gsub!(/([ \t]+)}}(\],?$)/,'\1\2') # /
    c.gsub!(/'/, '\\\\' + "'") # don't ask :/
    c.gsub!(/([[:blank:]]*\()(".
    +"[[:blank:]]=\>[[:blank:]]".+")\)(,?)/, '\2\3')
    c.gsub!(/(.+\[)\((.+)\)(\].+)/,'\1\2\3')
    c.gsub!(/"/, '\'')

    View full-size slide

  24. JBoss output massaging function
    commit 3b54e51a0fc9149330e32c42fd784ad3b20850ad
    Author: Bartosz Trudnowski
    Date: Tue Feb 23 10:00:58 2016 +0100
    - c.gsub!(/([[:blank:]]\()(".+"[[:blank:]]=\>[[:blank:]]".+")\)(,?)/, '\2\3')
    + c.gsub!(/([[:blank:]]*\()(".+"[[:blank:]]=\>[[:blank:]]".+")\)(,?)/, '\2\3')

    View full-size slide

  25. JBoss output format

    View full-size slide

  26. Other JBoss annoyances
    - config keywords incoherence
    - punctuation frenzy

    View full-size slide

  27. The good and the bad
    What lessons did we learn?

    View full-size slide

  28. PROD cluster

    View full-size slide

  29. PROD cluster 1 hour later

    View full-size slide

  30. Handover
    bph_jboss_srv bph_jboss_ml bph_jboss_app

    View full-size slide

  31. Handover – number of commits
    bph_jboss_srv bph_jboss_ml bph_jboss_app
    80 Tomasz Torcz 254 Tomasz Torcz 50 Tomasz Torcz
    9 Waldemar Piszczewiat 67 Bartosz Trudnowski 10 Bartosz Trudnowski
    4 Krzysztof Kowalczyk 17 Waldemar Piszczewiat 7 Waldemar Piszczewiat
    4 Bartosz Trudnowski 4 Bartosz Dempc 6 Krzysztof Kowalczyk
    3 Waldemar Buczak 1 Waldemar Buczak
    1 Grzegorz Prusik 1 Grzegorz Krysik

    View full-size slide

  32. Data bags – number of commits
    dev test prod
    218 Waldemar Piszczewiat 273 Waldemar Piszczewiat 40 Bartosz Dempc
    53 Tomasz Torcz 194 Bartosz Dempc 17 Paweł Rodziewicz
    8 Bartosz Trudnowski 107 Grzegorz Prusik 13 Grzegorz Prusik
    5 Technical CICD account 49 Rafał Markowski 6 Michał Szor
    3 Waldemar Buczak 43 Grzegorz Krysik 6 Waldemar Piszczewiat
    2 Bartosz Dempc 24 Grzegorz Słaby 5 Grzegorz Słaby
    … …

    View full-size slide

  33. Reading the Chef output
    * execute[Updating SG01 property
    spring.profiles.active to
    cbpm,sit,mockReport,localStorage] action run
    (skipped due to not_if)

    View full-size slide

  34. Managable config
    - many similar, but maybe different clusters?
    - json-diff
    - testing, moving configuration
    - copy&paste
    - easy to revert

    View full-size slide

  35. Diverging templates
    cicd-dev-apps/dev/ … /GsbTytan.properties.erb
    cicd-uat-apps/dk1/ … /GsbTytan.properties.erb
    cicd-uat-apps/dk2/ … /GsbTytan.properties.erb
    cicd-uat-apps/tg/ … /GsbTytan.properties.erb
    cicd-sit-apps/sit/ … /GsbTytan.properties.erb
    cicd-sit-apps/sitstab/ … /GsbTytan.properties.erb
    cicd-prod-apps/prod/ … /GsbTytan.properties.erb
    - moved to external git repos

    View full-size slide

  36. The Song of JBoss and Chef
    The Song of JBoss and Chef
    DevOpsDays Warsaw 2016
    Tomasz Torcz

    View full-size slide