🔪 How we cut our AWS costs in half

E-commerce-search engines are resource eating monsters. Fast response times are key thus a lot of both heap and cpu is needed. For a customer we were challenged with reducing their overall AWS budget spent on driving their Solr cluster. In this project report we'll navigate through a classic cloud architecture way down to the foundations of garbage collection in the JVM. We'll explain the benefits of using ARM over Intel processors. We achieved a breakthrough by analysing the Solr internals using the Java Flight Recorder, upgrading the JDK to version 19 and experimenting with the Z garbage collector. In the end it worked: we cut the customers's AWS bill in half.

Transcript

1. @debe & @tboeghk 🔪 How we cut our AWS costs in half
2. @debe software engineer, architect and performance enthusiast #jvm #high performance #rust @tboeghk Freelance Search & Operations Engineer #solr #observability #ops #java 🤝
3. 📚 A holistic approach Project profile Architectural challenges Infrastructure challenges Software challenges Lessons learned Foto von Agnieszka Kowalczyk auf Unsplash
4. (1) Architectural challenges
5. 🛒 Project setting Foto von Mike Petrucci auf Unsplash
6. 🏡 Platform architecture
7. 🐳 Team system architecture
8. 🐳 Solr cluster topology Split API & shop traffic API cluster absorbes request peaks Medium cluster utilization 💰 1k per instance/month
9. 🐳 Solr Cluster topology Goal: get rid of redundant infrastructure How to enable Solr to handle request spikes?
10. (2) Operational challenges Foto von Chris Lawton auf Unsplash
11. 🦾 ARM our lord and savior Docker buildx to build 
 multi-arch builds Custom arm64 AMI Multi-Arch ASGs via MixedInstances in LaunchTemplate Graviton AMD Intel
12. 💰 ARM vs AMD Graviton2 AMD Linux load1 (less is better) AMD power management ⚡
13. 🏎 Graviton2 vs Graviton3 Linux load1 (less is better) Graviton3 Graviton2
14. 🏝 Our instance type journey
15. 🌱 monitor ecosystem innovation
16. (3) Software Challenges Foto von Alexander Hafemann auf Unsplash
17. 🙌 Why software challenges? Response time is tied to cpu utilization 😱 Foto von Jeremy Lapak auf Unsplash
18. 👨✈ Java Flight Recorder Event based tracing framework built into the JVM Very low overhead < 1% Designed for production use Free to use Foto von Richard Cartmell auf Unsplash
19. 👩✈ JDK Mission Control Java Object Locking problems
20. 👩✈ JDK Mission Control: Locking Monitor classes Thread wait time Call stack
21. 🛬 Flight recorder always on -XX:StartFlightRecording=disk=true, maxsize=512M, dumponexit=true, name=continuous, settings=default, filename=/tmp/jfr/search_REDACTED.jfr
22. 🛬 Flight recorder always on
23. 🧑💻 Analysis process
24. Flight Recorder Findings using 60s on-demand recordings Foto von Elias Maurer auf Unsplash
25. 🔐 RSA-Certificates
26. 🪵 Async logging
27. 🤦 Locking in Solr
28. 🧞 Deep dive into Solr code
29. 👌 Locking in Solr
30. 🛫 Achievement unlocked Scale out at 50% cpu usage possible 💪 Linux load1 
 (ideal @ 32)
31. 📉 Overall reductions 1/3rd 😱
32. (4) Lessons learned Foto von Kelli Tungay auf Unsplash
33. 🤓 Failures • Spot-Instances • Java Z garbage collector (for now)
34. 👴 G1 Garbage Collector Minimum pause collector 200ms max collection time Uses 90-105ms for a 32g heap
35. ♻ Z Garbage Collector Reduces collection time to 600µs for a 32g heap 😳 Uses Linux huge pages
36. 🐾 Lessons learned • Have strong observability in place • Involve yourself in the OSS software you use – yes maybe you are the first one having this exact problem • Test in prod or live a lie
37. Dennis Berger (@debe) Torsten Köster (@tboeghk) Questions?