Moving Past the Monolith
with Clojure
Francisco Viramontes
@kidpollo
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Yes, simple, but tell me how!
https://www.youtube.com/watch?v=VSdnJDO-xdg
Clojure, Made Simple
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
The barrier?
http://www.clojurenewbieguide.com/
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
Don’t come empty handed!
● Change is hard in larger orgs
● It’s not going to sell itself
● There is going to be fierce opposition
● Put your foot in the door
● Prototype!
○ Collect Data
○ Instrument Clojure code
○ Take a data driven approach
○ Have other success stories at hand (like this
one :P)
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
~40% more work
~70% less resources.
Clojure API
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
Bird’s eye view
● Multi Terabytes db
● ~17k db reads / ~1.3k writes per second
● Risk hardware degradation
● Small blips in traffic could cascade into total system failure
● Non scaling background jobs running with MySQL backend
● Code produces unnecessary db calls
● The core (email processing) constantly in flames
Slide 15
Slide 15 text
The core of the system
IMAP Mailboxes
Background Jobs (Monolith)
Customer A Customer B Customer C
MySQL DB
● Most time spent on socket
negotiation
● ~87% of the bg jobs where
processing email.
● 20 servers with 99 rails
processes
● Only checking email once
every ~45 seconds
Slide 16
Slide 16 text
Background Jobs
(Monolith)
Mysql DB
Amazon SQS + S3
IMAP
Mailboxes
Customer A
Customer B
Postman
Nodes
Zookeeper Postman
Jobs DB
(Redis)
Slide 17
Slide 17 text
The Postman node
1. Get mailbox connection info
2. Open/handle imap connection
3. Check email
4. Download email
5. Push to SQS + S3
6. Wait 5s
7. Go to 3
*** For ~12k mailboxes concurrently!
Ordacity/Zookeeper
javax.mail
Amazonica
Java Thread Pool Executors