Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Moving Past the monolith with Clojure

kidpollo
September 01, 2016
270

Moving Past the monolith with Clojure

Experience report of introducing Clojure in the enterprise for the Clojure PDX Meetup. https://gist.github.com/kidpollo/d6462feae7d52d3c0352754d45d5b0f9

kidpollo

September 01, 2016
Tweet

Transcript

  1. Don’t come empty handed! • Change is hard in larger

    orgs • It’s not going to sell itself • There is going to be fierce opposition • Put your foot in the door • Prototype! ◦ Collect Data ◦ Instrument Clojure code ◦ Take a data driven approach ◦ Have other success stories at hand (like this one :P)
  2. Bird’s eye view • Multi Terabytes db • ~17k db

    reads / ~1.3k writes per second • Risk hardware degradation • Small blips in traffic could cascade into total system failure • Non scaling background jobs running with MySQL backend • Code produces unnecessary db calls • The core (email processing) constantly in flames
  3. The core of the system IMAP Mailboxes Background Jobs (Monolith)

    Customer A Customer B Customer C MySQL DB • Most time spent on socket negotiation • ~87% of the bg jobs where processing email. • 20 servers with 99 rails processes • Only checking email once every ~45 seconds
  4. Background Jobs (Monolith) Mysql DB Amazon SQS + S3 IMAP

    Mailboxes Customer A Customer B Postman Nodes Zookeeper Postman Jobs DB (Redis)
  5. The Postman node 1. Get mailbox connection info 2. Open/handle

    imap connection 3. Check email 4. Download email 5. Push to SQS + S3 6. Wait 5s 7. Go to 3 *** For ~12k mailboxes concurrently! Ordacity/Zookeeper javax.mail Amazonica Java Thread Pool Executors
  6. Over 80% reduction in total bg job workload Over 50%

    reduction servers used Check email every 5s VS every 45s Postman
  7. Fin