Elasticsearch: when to host on premise and when to turn to SaaS

Elasticsearch: when to host on premise and when to turn to SaaS

You've been tasked with building a logging/streaming/text search product. You know you want to use Elasticsearch, but do you host on premise, or do you go for the cloud? The decision depends much on the use cases for the product, the resources of the company, the project time-frame and the strengths of your team. Drawing on my extensive experience with on-premise Elasticsearch at Loggly, and more recently, using AWS's hosted Elasticsearch at Stitch Fix (which I wrote about in this article: https://read.acloud.guru/things-you-should-know-before-using-awss-elasticsearch-service-7cd70c9afb4f), I'll discuss the pros and cons of each option and explain in which cases hosted Elasticsearch makes sense and when Elasticsearch as a service is the way to go.

Liz Bennett has specialized in the realm of Big Data and infrastructure since 2013. She got her start working on the News Feed at LinkedIn, then moved on to Loggly, a popular SaaS based log management company. There she was an integral member of the infrastructure team building their massive Kafka and Elasticsearch backed log processing pipeline. Now, she is a member of Stitch Fix's Data Infrastructure team where she is leading efforts to expand their streaming, logging and data integration capabilities.

https://www.meetup.com/Elasticsearch-San-Francisco/events/244064018/

Transcript

1. Liz Bennett - Data Platform Infrastructure Engineer Elasticsearch: When to Host On-Prem and when to turn to SaaS?
2. About Me Liz Bennett Currently @ Member of Compute Infrastructure Team Previously @ Senior Engineer - Data Infrastructure Previously @ Engineer - News Feed Team
3. We are a personal styling company Customers complete a “style profile” Stitch Fix sends them clothes Customers keep what they want, send back everything else Customers give us feedback for every item
4. Our 80+ Data Scientists own dozens of verticals in the business The Data Platform team builds infrastructure and tooling to support all things Data Science We use ES as: an ELK stack + sundry search use cases
5. Elasticsearch: When to Host On-Prem and when to turn to SaaS?
6. Today there’s a hosted solution for everything And Elasticsearch is no different Just because you can buy hosted ES, should you?
7. Before diving into specifics… ...my personal Elasticsearch journey
8. My ES Journey 1. LinkedIn: deployed an ELK stack on-premise 1. Loggly: maintained a massive, multi-clustered, on-premise ES deployment 1. Stitch Fix: hosted ES with AWS
9. In June 2017, I wrote Some Things you Should Know Before Using Amazon’s Elasticsearch Service on AWS
10. I got to thinking… ...why do so many people use ES as a service? ...when should you and when should you not use a hosted ES solution?
11. A tough decision regardless of technology Elasticsearch is unique Special considerations should be made when deciding between on-prem vs. SaaS
12. Main Considerations 1. Size of the dataset 2. Irregularity of the dataset 3. The skillsets of your team 4. Timeframe 5. Search’s impact to the business 6. Security Considerations 7. SLA requirements
13. Size of your dataset
14. Size of the Dataset Small datasets (<50GB) vs. very large datasets (>5TB) The size of a dataset matters because of these factors: 1. Cost in $$$ 2. Operational Burden 3. Effectiveness of a hosted solution
15. Size of the Dataset - Cost $$$ Cost will scale linearly based off of the size of your dataset The slope and intercept of the cost functions are different tho
16. Size of the Dataset - Cost $$$ Rough estimates of per node costs ( 1 node, 16GB Memory, 4 cores) AWS Raw Instance cost (m4.xlarge) $0.2 / hour = $146 / month AWS ES, per node cost (m4.xlarge.elasticsearch) $0.301 / hour = $220 / month Elastic Cloud (1 node cluster w/ 16GB mem) $0.5221 / hour = $382 / month HP ProLiant DL360 Quad-Core, 16GB mem $0.08 / hour = $58 / month ($2200 amortized over 3 yrs)
17. Size of the Dataset - Operational Burden ! The larger your dataset, the more maintenance the cluster requires ! A small cluster can be operated more manually ! With bigger clusters, maintenance needs to be automated ! A very small cluster can be easily hosted on premise but is also cheap to buy from a hosting service ! Other factors, like time-frame, will matter more if the dataset is small
18. Size of the Dataset - Effectiveness of a Hosted Solution ! Larger clusters may exceed the capabilities of a hosting service to provide a stable deployment ! Particularly with ES, large deployments can require complex, custom configurations ! Many hosted ES providers expose a limited set of configuration options, or none at all
19. Irregularity of your dataset
20. Irregularity of the data set An ES-specific factor that should not be underestimated Regular datasets: ! Documents have a well defined set of fields ! Indexing volume is steady ! Data can be easily distributed across all nodes in the cluster Irregular datasets: ! Documents can have any field, lots of dynamic fields ! Indexing volumes are sporadic and unpredictable ! Data is not evenly distributed across the cluster. Hot spots may occur
21. Irregularity of the data set ! Irregular datasets can cause a myriad of stability issues in ES ! Regardless of hosting on-premise or in the cloud ! Finding these issues, understanding, and fixing them requires an in-depth knowledge of ES ! You also need access to detailed dashboards and monitoring tools ! Regardless of on-premise vs. SaaS, you need to know how to model your data, how to structure your indexes, ES best-practices, dos and don’ts, etc. ! If your data is irregular, you need an understanding of ES and its internals in order to ensure the stability of your cluster ! Developing this knowledge is half the hurdle when it comes to operating and hosting ES yourself ! If your data is highly irregular, you could bite the bullet and host ES yourself
22. Irregularity of the data set ! Regular datasets will be far easier to manage ! The cluster will be more stable, easier to operate and to maintain ! Hosting yourself will be easier ! On the other hand, paying someone to host for you will offload the burden of having to know anything at all about Elasticsearch
23. Skills of your team
24. Skills of your team ! Is your team mostly front end engineers? You probably want hosted ES ! Mostly backend engineers? Maybe on-premise ! Is your team very small and strapped for cycles? Go with hosted ES ! Your team is small, strapped for cycles, and strapped for cash? Maybe on-premise DevOps Skills ! If you are considering using Elasticsearch, you already have a software stack ! Software built in-house, you already maintain, monitor, deploy, upgrade and operate ! You’ve maybe already invested in devops to some degree ! If you already have strong devops chops, the upfront cost of deploying your own infrastructure is much less ! For example, if your stack is mainly JVM with high SLAs, you likely already have the tools necessary to run your own ES cluster
25. Timeframe
26. Timeframe ! Clicking a button on a UI is always faster than hosting something yourself ! If time is your biggest concern, the cloud is the way to go ! Hosted solutions are great for prototyping ! This is especially true with AWS, if you are already on AWS ! Buying a hosted solution may involve a contract negotiation process
27. Search’s Impact to the Business
28. Search’s Impact to the Business ! If search is central to your business, it could be worth investing in on-premise ES ! Developing in-house Elasticsearch expertise will ensure your cluster is: ◦ More tailored to your use case ◦ Cheaper to operate ◦ More performant ◦ More stable ! If search is central to your business, having a better ES deployment than your competitors gives you a competitive advantage ! If you host ES on-prem, it is easier to get more involved with the open source community ! Easier to write and submit patches upstream ! If search is important to the business, it may be valuable to play an active role in open-source ! If you need search to power an internal tool, the ROI is less and a hosted solution could be better
29. Security Considerations
30. Security Considerations ! Because of security considerations, you may prefer on-premise software solutions. ! If you are storing sensitive data (such as PII) in your ES cluster, you may not want to send the data to a third party. ! Some regulations can make it harder to work with hosted solutions, i.e. HIPAA, EU GDPR ! If the data in your ES cluster was breached, how bad would it be for your company? ! How much bureaucracy is involved when you decide to work with a hosted solution? ! Would it be faster and easier to just host it yourself?
31. SLA Requirements
32. SLA Requirements ! SLA requirements break down into ! Query performance ! Indexing performance ! Uptime ! Query performance and indexing performance can generally be improved by: ! Adding more nodes ! Optimizing sharding strategies ! Optimizing use of filters and caching ! Things you’d have to do regardless of on-prem vs. SaaS ! Uptime is a different story
33. SLA Requirements - Uptime ! When it comes to Elasticsearch, expect downtime. ! Many ES stability issues happen because “you shot yourself in the foot”. ! They are self-inflicted by your query/indexing patterns, sharding, document structure, etc. ! What does this have to do with on-prem vs. SaaS? ! With most SaaS infrastructure, you offload maintenance, monitoring & technical expertise ! With ES, it’s harder to offload the burden of technical expertise ! Also, monitoring is more complex and circumstantial. You have to do a lot of it yourself ! Even if you’re using ES as a service, don’t expect plug n’ play ! Here’s the kicker ! You won’t always be able to fix your cluster yourself, because administrative and monitoring APIs may be gated by the hosting service ! You must work with support engineers ! Back and forth can take a long time, and periods of downtime can last hours ! If uptime is extremely important, hosting ES yourself may yield a more stable cluster
34. Some Examples
35. When hosted ES makes sense 1. ELK stack for a small to moderately sized team or department 2. To power search for a small and predictable dataset, i.e. inventory in a warehouse, blog posts, etc. 3. When prototyping a new search feature or product 4. To power search for an internal tool
36. When on-premise ES makes sense 1. You’re launching a search-centric business or product line, i.e. Yelp 2. Company-wide ELK deployment for mid to large sized company 3. You’re building an app that searches highly sensitive data
37. Why my team uses hosted ES 1. Our dataset is relatively small 2. We can tolerate downtime 3. The indexing and query patterns are somewhat predictable 4. It is only used to power internal tools 5. Having a better ES deployment is not a competitive advantage
38. Main Take-Aways
39. Main Take-Aways ! Hosted ES is faster to setup ! Hosted ES is more expensive ! Hosted ES will not offload the burden of technical expertise or monitoring ! Hosted ES does offload the burden of maintenance (upgrades, snapshots) ! On-prem ES will perform better and be more stable, especially for large clusters ! Cost of on-prem ES depends on the DevOps skills of your team ! Elasticsearch is not like other kinds of infrastructure. Do not expect plug & play
40. Some armchair thoughts… Many open-source companies (Elastic included) are increasingly relying on hosted solutions for revenue They are incentivized to convince you that hosting their software is onerous They are incentivized to get you to upgrade and use more resources They are incentivized to make it difficult for you to host their software Always think critically, consider your options deeply and understand the ROI
41. Thanks for coming! Tweet me at @zzbennett E-mail me at elizabeth.bennett@stitchfix.com (Also we’re hiring!!!!!)
42. ? ? ? ? ? ? ? ? ? ? Questions?