A Hitchhikers Guide to the Cloud Friendly Migration of Apache Solr

This talk will cover the process of migrating a large-scale, multi-language Apache Solr infrastructure from on-premise servers to AWS cloud infrastructure. The presentation will detail the challenges and best practices involved in the migration, including considerations for instance sizing, data transfer, configuring autoscaling and load balancing, while maintaining optimal search performance. The talk will also highlight the benefits of the migration, including increased scalability, cost savings, and improved disaster recovery capabilities. Real-world examples will be used to illustrate the process, building blocks and tools used during the migration and the talk will conclude with a discussion of future optimizations.

Transcript

1. qaware.de A Hitchhikers Guide to the Cloud Friendly Migration of Apache Solr Mario-Leander Reimer mario-leander.reimer@qaware.de Managing Director | CTO QAware GmbH Wolfgang Klimt wolfgang.klimt@consol.de IT Troubleshooter ConSol Software GmbH
2. QAware | 2
3. 3 QAware
4. 4 QAware
5. Old Architecture QAware | 5 28x 28x 28x 8 Apache Instances Search Load 28x Loader
6. New Architecture, Step one QAware | 6 28 x 3(+) 28x 8 Apache Instances Search 28x Loader Load
7. New Architecture, Step one QAware | 7 28 x 3(+) 28x 8 Apache Instances Search 28x Loader Load Staging 28 x
8. Search Solr Go Live QAware | 8 8 Apache Instances 28x 28x
9. New Architecture, Step two QAware | 9 28 x 3(+) Search 8 Apache Instances 28 x Staging 28 x Load Loader Loader Loader Loader ? x Loader
10. Go Live of Step two QAware | 10 28 x 3(+) Search 8 Apache Instances 28 x Staging Loader Loader Loader Loader 28x 28x Loader
11. Go Live of Step two QAware | 11 28 x 3(+) Search 8 Apache Instances 28 x Staging Loader Loader Loader Loader 28x 28x Loader
12. Technologies QAware | 12 ▪ Mainly AWS EC2 machines and ALBs ▪ Terraform and Terragrunt used as Infrastructure as Code solution ▪ Hashicorp Packer used to create custom AMIs with Apache Solr distribution and AWS Corretto JDK pre-installed ▪ AWS Aurora but also Apache Ignite as staging databases ▪ Envoy Proxies to access on-premise datacenter from VPC, mainly to save IP addresses on direct connect ▪ EBS snapshots for faster startup of Solr instance on scale-up
13. Learnings and Tips QAware | 13 ▪ Plan a gradually Go Live ▪ Have a fallback plan ▪ Many small instances behave different to few big instances ▪ Reserve time for improvements after go live ▪ Test thoroughly, but don’t expect to have tested everything
14. Q&A
15. qaware.de QAware GmbH Aschauer Straße 32 81549 München Tel. +49 89 232315-0 info@qaware.de twitter.com/qaware linkedin.com/company/qaware-gmbh xing.com/companies/qawaregmbh slideshare.net/qaware github.com/qaware