Moving Past the monolith with Clojure

Experience report of introducing Clojure in the enterprise for the Clojure PDX Meetup. https://gist.github.com/kidpollo/d6462feae7d52d3c0352754d45d5b0f9

Transcript

1. Moving Past the Monolith with Clojure Francisco Viramontes @kidpollo
4. Yes, simple, but tell me how! https://www.youtube.com/watch?v=VSdnJDO-xdg Clojure, Made Simple
6. The barrier? http://www.clojurenewbieguide.com/
8. Don’t come empty handed! • Change is hard in larger orgs • It’s not going to sell itself • There is going to be fierce opposition • Put your foot in the door • Prototype! ◦ Collect Data ◦ Instrument Clojure code ◦ Take a data driven approach ◦ Have other success stories at hand (like this one :P)
11. ~40% more work ~70% less resources. Clojure API
14. Bird’s eye view • Multi Terabytes db • ~17k db reads / ~1.3k writes per second • Risk hardware degradation • Small blips in traffic could cascade into total system failure • Non scaling background jobs running with MySQL backend • Code produces unnecessary db calls • The core (email processing) constantly in flames
15. The core of the system IMAP Mailboxes Background Jobs (Monolith) Customer A Customer B Customer C MySQL DB • Most time spent on socket negotiation • ~87% of the bg jobs where processing email. • 20 servers with 99 rails processes • Only checking email once every ~45 seconds
16. Background Jobs (Monolith) Mysql DB Amazon SQS + S3 IMAP Mailboxes Customer A Customer B Postman Nodes Zookeeper Postman Jobs DB (Redis)
17. The Postman node 1. Get mailbox connection info 2. Open/handle imap connection 3. Check email 4. Download email 5. Push to SQS + S3 6. Wait 5s 7. Go to 3 *** For ~12k mailboxes concurrently! Ordacity/Zookeeper javax.mail Amazonica Java Thread Pool Executors
18. https://gist.github.com/kidpollo/d6462feae7d52d3c0352754d45d5b0f9 Demo
20. Over 80% reduction in total bg job workload Over 50% reduction servers used Check email every 5s VS every 45s Postman
21. Fin
22. Sources • www.Desk.com • http://clojure.org/about/rationale • http://3.bp.blogspot.com/-DC1MO0wrBwU/TpG-WtFZP7I/AAAAAAAAIPM/q TB6wZLQmG0/s800/IMG_3645.JPG • https://upload.wikimedia.org/wikipedia/en/thumb/e/e9/Ruby_on_Rails.svg /791px-Ruby_on_Rails.svg.png • http://www.magnasoma.com/content/images/Magnasoma-Monolith-3-01. jpg • http://emojipedia.org/pile-of-poo/