Quickwit - IT Press Tour #55 April 2024

Transcript

1. Quickwit Log Search at Petabyte Scale 1
2. Log search solutions are hard to scale 1) Infrastructure costs are prohibitive at terabyte and petabyte scale 2) Cluster management is overly complex 3) Indexing is slow and unstructured logs are poorly handled 2
3. Log datasets are growing exponentially In a data-driven world, terabyte datasets are the new standard. Large tech and cybersecurity companies already need to manage petabytes of logs. Splunk, Elastic, and other solutions eventually hurt companies' innovation: retention is limited, query time is too long, high TCO. 3
4. Quickwit: unparalleled cost-efficient log search engine We designed Quickwit to be fast, scalable, and reliable at petabyte-scale for a fraction of the costs: Built in Rust, and powered by Tantivy, a leading search engine library maintained by Quickwit. With decoupled compute and storage (all data on object storage) Stateless Schemaless Quickwit is at least 10x more cost-efficient than existing solutions and you don't need a whole team to manage petabytes. 4
5. Showcase: Quickwit indexing at petabyte scale Indexing 1 PB of logs per day with Quickwit with 200 pods with 6 vCPUs, 8GB of RAM per pod. 5
6. Showcase Quickwit search at petabyte scale Searching 40PB of logs from Grafana directly on object storage with 30 pods with 20 vCPUs, 40GB of RAM per pod. 6
7. We receive love from our community, from companies, users, and engineers who are using Quickwit 7
8. It's really new but I have a feeling Quickwit will take the logging scene by storm. DevOps on Reddit - 2024/04/04 8
9. Elastic was too time-consuming to maintain, and we wanted a more down-to-earth solution with an S3-compatible backend. Loïc Tosser - Co-Founder & CTO at Kalvad 9
10. Our darknet search engine for companies, leverages Quickwit for indexing darknet content. Quickwit's cost efficiency sets us apart from competitors, making Quickwit a game-changer for us Damien LESCOS, CEO of SitInCloud 10
11. Observability and security teams are our main target Observability and security teams managing more than 1TB of logs Cybersecurity companies: XDR/MSSP, Threat Intelligence Financial services companies: audit and transaction search 11
12. How Quickwit is different from other search technologies Fast search Inverted index Fast analytics Columnar storage Cost-efficient Object storage native Stateless architecture Elasticsearch/Splunk Grafana/Loki Quickwit Best of Both Worlds: Offers unparalleled fast search and analytics on object storage, setting a new standard in the industry. 12
13. Quickwit architecture is scalable & efficient Indexing process: Creates split files from JSON documents and uploads them to the object storage. Search process: Sends precise range bytes queries to splits. 13
14. Quickwit is sub-second from object storage Quickwit sets a new standard for efficient data retrieval from object storage Stored directly on object storage. Composed of 3 datastructures: Row-oriented storage: Enables fast document access Inverted Index: Enables fast lookups Columnar Storage: Enables fast analytics. 14
15. Case study #1 (Crypto company): Log search at petabyte scale Migration from OpenSearch to Quickwit divided CPU costs by 5, storage costs by 2 while increasing retention by 10. Cluster size: 40PB ingested, 5x10¹³ of log entries, 7.5 PB on S3, more than 500 instances. Quickwit sizing for ~1 PB/day: 200 pods with 6000 milliCPU, 8GB of RAM per pod. Exactly-once semantic thanks to Kafka native integration. 15
16. Case study #2 (Fly.io): Large-scale multitenant log search Migration from OpenSearch to Quickwit: From 40 instances to a couple of instances (not yet in production, sizing under evaluation) while increasing retention by 10. Cost-efficient multi-tenant setup thanks to Quickwit cooperative indexing and stateless search. Seamless integration in Grafana thanks to Quickwit plugin. 16
17. Case study #3 (OwlyScan): Searching the Darknet Cost-efficient darknet search engine for companies from 500TB to 1PB of data The cost-efficiency of Quickwit allows SitInCloud to offer a competitive pricing model. 17
18. Quickwit business model Quickwit is deployed on the customer's infrastructure. Privacy matters in log search. 2024: Double license AGPL / Commercial license to remove AGPL restrictions and provide enterprise support services. 2025: Open-Core with commercial license for enterprise features. Example: Encryption per tenant/bucket. 18
19. GoToMarket / Distribution Direct Sales: Targeting tech and cybersecurity companies with large log datasets. Partnerships: With cloud providers, large tech companies and consulting companies who recommend Quickwit to their clients. Community: Open-source version to build a community around Quickwit. 19
20. Quickwit: leading OSS cloud-native log search engine Launch & Foundation: Started 3 years ago; built on top of Tantivy library with over 6 years of development. Adoption: Used by > 200 companies (OSS version). Trusted by leading tech companies: Crypto company: Manages over 40 petabytes of logs across more than 500 instances. Fly.io: Supports over 100,000 tenants. YC Log SaaS Company: Manages over 1 petabyte. 20
21. Made by a team of passionate rust engineers Paul, Co-founder Creator of Tantivy Pascal, Senior Search Engineer > 10 years of XP Adrien, Co-founder > 10 years in Distributed Systems Trinity, Senior Rust Software engineer > 5 years of XP François, Co-founder > 10 years of XP as SSWE Rémi, Cloud & Data Senior Engineer > 10 years of XP Damien, UX/Rust Senior Engineer > 10 years of XP 21
22. Roadmap April 2024: Distributed ingest High-throughput indexing on tens of thousands of indexes. Q2 2024: OpenSearch Dashboard support Enable OpenSearch users to migrate seamlessly to Quickwit with their existing dashboards. Q3 2024: Pipe-based query language Introduction of a flexible and powerful query language similar to SPL (Splunk Query Language) Q4 2024 - 2025: Metrics support New storage engine optimized for time series data. 22
23. Quickwit long term goal: becoming the go-to solution for log search. Drop-in for Elasticsearch, Splunk, Datadog (log), Google Chronicle and other solutions. Innovative companies with petabytes of logs will use Quickwit or will have to implement their own solution. 23
24. Thank you! Contact francois@quickwit.io 24