PTSM - #2 - White Paper QuasarDB Technology

Transcript

1. The QuasarDB Techology From Big Data to Big Value October 2019 Intended audiences This document is intended for software architects, software developers, IT managers, and anyone curious about the QuasarDB technology. In this white paper we will contextualize the QuasarDB technology and then dig into the details of how it uniquely addresses the challenges of modern data management.
2. 2 The QuasarDB Technology Table of contents Part 3: What does QuasarDB give me? 06 06 07 07 07 08 Ingest all the data you need at the speed you require The model adapts to your need, not the other way around Powerful and instant insights on all your data Safely and securely manage your data Unlimited and efficient storage Low Total Cost of Ownership Part 1: The case for timeseries databses 03 04 04 05 Relational databases Numerical data and timeseries Specificities of timeseries data management Should I use a timeseries database? Part 2: QuasarDB’s principles Our vision The principles 05 06 Part 4: Technology deep dive Implementation Zero-overhead architecture Core engine Clustering Concurrency Efficient networking Smart indexing Transaction Storage Compression Queries Authentication and rights management Audit trail Security 08 08 09 09 10 11 11 11 12 13 14 15 15 15
3. The QuasarDB Technology Relational database Numerical data and timeseries 1. The case for timeseries database Annual size of the global Datasphere Source: Data Age 2025, sponsored by Seagate with data from IDC Global Datasphere, Nov 2018 Zetabytes 175 ZB 2010 0 20 40 60 80 100 120 140 160 180 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 Relational databases, and their companion language, SQL, are the safe, default choice when it comes to data management. Their flexibility and the solidity of the relational model enables them to cater to the typical needs of a business. It is no surprise that SQL became a standard for data querying and that so many businesses in the world depend on relational databases on a day to day basis. However, the constraints of the relational model become a handicap when speed and flexibility are a must. Timeseries is the fastest growing database model, and for a reason. Stocks, sensors from factories, autonomous vehicles, airplanes… Petabytes of data are generated every day, and most of it can be represented as numerical timeseries. The relational model isn’t optimized or even designed for timeseries. Foreign keys are a terrible match with timeseries data as they have significant performance implications. Additionally, typical timeseries needs such as trimming, resampling or alignment (“asof”) are cumbersome and inefficient to implement in a relational database. 3
4. The QuasarDB Technology Complete trend, starting with January 2013 Time Series DBMS Relational DBMS 450 400 350 300 250 200 150 100 50 2013 2014 2015 2016 2017 2018 2019 Popularity Changes Source : db-engines.com Finally, timeseries are typically tagged with contextual information such as origin, device type, data type. A rigid schema makes handling this metadata cumbersome and inefficient. Timeseries databases enable faster querying speed, optimized disk storage, and convenience. VOLUME – Because of its cumulative nature, timeseries tend to have a larger data set, resulting in data management problems typically found when working on data warehouses. This creates pain points at data insertion, disk usage, and scanning. APPEND MOSTLY – The data is usually appended to existing timeseries. A database engine aware of that can take smart decisions to optimize ingestion. LOOSE RELATIONS – Although timeseries can be represented as tables, the association between tables is loose when not inexistent. Additionally, metadata associated with the timeseries can be cumbersome to represent in a purely relational model. SPECIFIC OPERATIONS – Timeseries data have some specific operations such as resampling, trimming, windowed joins that are hard to express in SQL and require an optimized engine to perform fast enough. INTRINSICALLY COLUMN ORIENTED – Timeseries are column oriented: columns are significantly larger than rows and data tends to be aggregated within the same column. New data is often highly correlated with previous data, enabling clever indexing and compression techniques that generic databases cannot leverage. Specificities of Timeseries data management What makes timeseries so special? 4
5. Time to insert 1GB in milliseconds Lower is better 435,75 5 123,964 25,851 T InfluxDB QuasarDB imescaleDB 5 Should I use a timeseries database? Today, the power of computers gives us a lot of headroom, even with sub-optimal database engines. However explosive disk usage, slow query time, the lack of convenience, and total cost of ownership are many reasons that can make you consider a timeseries database early on in your project. When reaching a threshold in volume or complexity, a system doesn’t go slower: it stops functioning and your business is in jeopardy overnight. Our mission is to enable companies to gain instant insights, regardless of the data volume. store everything forever, query it in real time. Our vision QuasarDB was born in the financial market in 2008, as it became obvious that relational databases were unfit for the challenges of big numerical data. Existing alternatives just didn't deliver in terms of performance, convenience, or reliability. We thus decided to build our engine using best practices and innovative technology to achieve the best possible performance without compromising convenience or reliability. 2. QuasarDB’s principles The QuasarDB Technology
6. The principles The QuasarDB Technology Ingest all the dats you need at the speed you require. QuasarDB delivers the fastest timeseries ingestion speed on the market through its combination of ultra-optimized protocol, unique compression algorithms, and built-in clustering. QuasarDB is designed to ingest terabytes per day and manage petabytes. Out of the box, QuasarDB comes with the tooling to ingest text files at maximum efficiency without requiring you to write programs or scripts. Need to write your program? The API takes care of all the little details so you can focus on the essential. The model adapts to your need, not the other way around. Timeseries are stored into tables. Contrary to a typical relational database, tables have very low overhead, and QuasarDB supports a very large number of tables. Because tables can be tagged with arbitrary strings, and because these tags can be used in queries, QuasarDB combines the flexibility typically found in NoSQL database with the rigor of relational databases.The API takes care of all the little details so you can focus on the essential. QuasarDB was designed from the ground up with the following seven principles: SPEED – Machines should wait on humans, not the other way around. Use the best algorithms. Implement them efficiently. Don’t use scalability as an excuse for poor nominal performance. SCALABILITY – However fast the engine is, the business will outgrow a single core, a single socket, and a single machine. The engine must scale up, then scale out, without any headache. STORAGE EFFICIENCY – Data grows forever. Leverage compression and smart encoding as much as possible, enable the user to store as much as they want. SAFETY – Preserve the data. Don’t surprise the user with behavior such as “eventual consistency.” Be consistent. Be transactional. It’s easier to reason with. SECURITY – Make unauthorized access extremely hard. Deploy in-depth security. Empower the administrators to control and audit access. CONVENIENCE AND FAMILIARITY – Don’t push the hard problems back to the users (consistency, integrity, aggregations, etc.). Give users access to the data through familiar interfaces. UNIVERSALITY – Data is everywhere. Run everywhere. In the cloud. In the datacenter. On the desktop. On the edge (e.g., support Intel and ARM architectures). 6 Say farewell to artificial data silos: QuasarDB makes all your data available. 3. What does QuasarDB give?
7. T Technology Eventual Consistency Besides, QuasarDB gives you access to a powerful, transactional, distributed key-value store to accommodate all the use cases where the relational model is too cumbersome. Distributed storage: concepts, algorithms, and implementations Powerful and instant insights on all your data Query your data through SQL or perform powerful analytics with Python Pandas or R. Stream your results through the BI tool of your choice with our ODBC connector. Since access to the data is uniform regardless of its age, insights are more accurate and unconstrained. The power of QuasarDB engine ensures you spend more time thinking and less time waiting. Safely and securely manage your data QuasarDB gives you data integrity guarantees as any database should. Its distributed transactional engine enables you to perform complex queries over the cluster with the guarantee you’ll have a consistent view. Writes to disks are safe and consistent, and the API gives you clear feedback on the success of your operations. Security features such as fine-grained access control, cryptographically strong authentication, and traffic encryption enable you to use QuasarDB in environments where data confidentiality is a must. Unlimited and efficient storage With its patent-pending Delta4C compression algorithm, QuasarDB delivers efficient disk usage without impacting performance. Disk usage can be typically ten times less than with a relational database. Highly compressed storage means reduced storage and the possibility to have a higher tier of storage (for example, move from SDD to NVMe). Because it’s scale-out, QuasarDB can grow with your data need and automatically re-shards and rebalances tables. Contrary to other in-memory databases, QuasarDB does not require you to explicitly load the data from the disk before performing a query. It transparently paginates data in and out. Eventual consistency is a consistency model used in distributed computing to achieve high availability that informally guarantees, that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. (Wikipedia) One could say that eventual consistency outsources the challenges of data correctness to the users. In some use cases, this is an acceptable tradeoff, as data accuracy can be relaxed (for example, the order of chat messages sent to a user). In most cases however, this greatly increase the cognitive load on users that need to compensate in their code the potential inaccuracies in results. That is why QuasarDB gives consistency guarantees for every update: to make reasoning about the data easier. 7 Systems using Eventual Consistency result in decreased system load and increased system avaibility but result in increased cognitive com- plexity for users and developers. Yaniv Pessach The QuasarDB Technology
8. The QuasarDB Technology Low total cost of ownership The combination of efficient storage, low CPU usage, integration, and ease of use results in tremendous savings for businesses. Deploying QuasarDB is cheaper and faster than typical Big Data solutions: › Data analysts use known tools such as Python and SQL to interface with their data: no arcane, proprietary language to learn and skills are transferable. › QuasarDB integrates ETL, distribution, storage, analytics, orchestration, and connectivity in a single package radically simplifying deployment and administration, shortening projects time. › Efficient storage means significantly reduced storage costs. › Speed means lower CPU usage, which directly translates to lower hosting costs. Max Min Timestamp 14:00 100 110 15:00 110 120 16:00 105 120 QuasarDB is column oriented in memory and in disk. When storing tables there are two ways or storing the data: per row or per column. When reading data in memory, processors prefetch the data adjacent to it. Thus, reading continuous data is faster than reading scattered data. Row oriented organization favors operations that work on rows, whereas column-oriented favors operations that work on columns. Let us consider a table with min and max values per hour. Computing the minimum value for all hours is a column oriented, whereas computing the spread for a given hour is row oriented. Timeseries columns are significantly larger than the rows, and, most operations are column oriented, making timeseries benefit greatly from column oriented storage. Finally, organizing data as columns enables compression to be more efficient as patterns are more likely to appear within the same column than within the same row. 8 4. Technology deep dive How does QuasarDB deliver these benefits? What sets it apart from the competition? The answer is found in the overwhelming amount innovation packaged into the software. Implementation QuasarDB is written in modern C++ 17 with the most critical parts handwritten in assembly. This delivers a portable binary unconstrainted by a virtual machine or complex dependencies. QuasarDB is available for Intel-compatible and ARM architectures, both 32-bit and 64-bit. Zero-overhead architecture As QuasarDB combines in a single binary the orchestrator, the query engine, the aggregation engine, and the persistence. Thus, it can use a zero-copy architecture that prevents the efficiency loss typically found in big data systems that combines different software products. Data isn’t moved around: it’s processed in place. For example, the memory buffer from the network card can be written directly to disk after sanity checks, removing overheads caused by allocations and copies. What does column- oriented mean ?
9. 42,50 2 2,800 91 InfluxDB QuasarDB TimescaleDB Time to query 1GB in milliseconds Lower is better 9 Core engine At its heart, QuasarDB is a multithreaded, column-oriented timeseries database. It combines the best of two worlds: the speed of in-memory databases with the capacity of long-term storage through its hybrid engine. The aggregation engine leverages the column-oriented nature of data for ultra-fast aggregations and uses Single Instruction Multiple Data (SIMD) instructions whenever available. Aggregations are made on-demand and leverage indexes when possible. Clustering QuasarDB transparently scales-out on an unlimited number of machines. Using a clustering technology based on MIT’s Chord, which is based on distributed hash tables (DHT). Timeseries rows are sharded across the cluster based on a duration block configurable for each table. For example, if an user choses a 1 day shard for a timeseries, the timeseries data will be divided in chunks of one day, and chunks will be distributed across the cluster for scalable storage and aggregation capabilities. Timeseries sharding is user-configurable per timeseries. Data load is automatically rebalanced (resharded) as the cluster grows, without a need to manually re-shard the tables. This resharding phase is transparent and automatic, delivering true, elastic, timeseries storage. The clustering is designed to deliver instant lookup with minimal network chatter. When a client process requests a query, it will split it into atomic requests and send those requests to the appropriate nodes. If the topology of the cluster is outdated (node loss, cluster growth), the client will detect that and remap the nodes. Asynchronous cluster to cluster replication is supported. The QuasarDB Technology
10. The QuasarDB Technology Concurrency Every server has a configurable number of partitions that will process requests concurrently. It uses lock-free and wait-free memory structures to reduce frictions between each partition. Partitions run within the same process to reduce overall memory usage and deliver better performance. A query is split by the client into smaller atomic requests, depending on the cluster topology at the time the query runs. This is done automatically and optimally for the user’s comfort. Atomic requests are then spread out on the cluster (map). The client then collects and aggregates the result to present the answer to the user (reduce). The splitting is done to offer the best possible compromise between resource management and performance. Queries are automatically parsed and deconstructed in smaller blocks. Aggregations automatically benefit from the map/reduce strategy without any user intervention. For example, if an user asks for the average value of a column, the average computation is distributed across all necessary chunks of the timeseries. 10 Built-in, elastic, clustering. Cluster-wide replication is available for maximum availability. Clustering Storage Network QuasarDB Daemon The daemon leverages zero copy on the network and disk layer. It can even bypass the operating system for maximum performance. Each node manages its storage independently.
11. Partitioning 11 Efficient networking On the wire, QuasarDB has a zero-copy binary protocol for optimal transitions, and its network engine is multicore aware in order to reduce inefficiencies due to context switching. QuasarDB runs on any network card and is optimized for Mellanox and Solarflare network cards. Smart indexing QuasarDB features a zero-overhead - in terms of memory and storage usage - index for timeseries data. It delivers constant-time lookup for timerrelated queries across the cluster. Sparse indexes enable the engine to speed-up numerical queries, whether it’s a filter (WHERE clause) or an aggregation. These indexes deliver the best compromise between disk usage and queries acceleration in using a stochastic approach. QuasarDB automatically creates those indexes, removing a lot of work on the plates of DBAs. Transaction QuasarDB supports distributed transaction using Multi Version Concurrency Control (MVCC). Unless otherwise specified, operations are read-committed and instantly consistent: writes are available everywhere after the operation is confirmed. When a query or an API call returns, the result is consistent for when the query was made, making it easier to reason about updates. The client does not have to “guess” the actual value of the data. Transactions are managed using high precision timestamps. When it needs to decide between consistency and availability, QuasarDB will choose consistency. Nevertheless, it has built-in, configurable, retries mechanisms that will shield the user from short service interruptions.. Storage Query Remote node Lock-free in-memory data structure The client receives the result for each atomic requests and aggregate them together to form the final answer Requests are sent by the client to nodes holding the data necessary to execute the atomic request The client splits a query into atomic requests Query-aware pagination Dispatcher Each request runs in an independent partition 0 Partition Result Client Client Partition Result Request Request Partition Request Answer Request The QuasarDB Technology
12. The QuasarDB Technology C1 Instead of using the default pagination mechanism of the operating system, QuasarDB uses an eviction strategy that is aware of the queries made by the user, enabling smart pre-fetching and eviction, delivering significantly better performance and lower memory usage. As needed, QuasarDB will read, decompress, and load into memories column as queries come in. When memory usage peaks, it will evict data using a scan-resistant strategy. Storage 12 RAM C2 Storage New points Compressed C1 and C2 Compress Acknowledgement after persistence Page-in C2 RAM C1 Storage column 1 and 2 not in memory decompress Lookup C1 and C2 Compressed C1 and C2 Query on C1 and C2 Load C1 and C2
13. Eviction RAM C2 C1 Page-out When the memory reaches the configured maximum usage the least-recently used columns are removed from memory Storage QuasarDB writes to disk in a way that minimizes data loss, corruptions, and inconsistencies in case of failures. It computes a checksum of every block to detect data corruption and does not delay writes. With Levyx Helium, QuasarDB can mount the device directly, bypassing the filesystem and the operating system completely for maximum performance and reliability. Compression QuasarDB compresses numerical data on disk using its custom Delta4C (patent pending) compression algorithm that has been trained through machine learning on petabytes of data to deliver optimal ratios at very high speed. Compared to LZ4, the fastest dictionary-based compression to date, Delta4C is up to 12 times faster and 60% more efficient. For data that cannot be compressed with Delta4C, QuasarDB uses a modified LZ4 for timeseries data that delivers the same compression at up to twice the speed. 13 The QuasarDB Technology
14. LZ4 1 2 12 Delta4C QDB LZ4 LZ4 18 19 30 Delta4C QDB LZ4 Average speed factor Ratio on integer timeseries Queries QuasarDB can be operated through a comprehensive and simple to use API available in all popular languages. It also supports SQL queries with timeseries extensions. Instead of inventing a brand-new language and jailing users in a proprietary product, QuasarDB makes sure data scientists are quickly up to speed. The timeseries extensions simplify the writing of complex time-related queries while making them easy to understand, even for someone unfamiliar with QuasarDB. The query language is aware of the Gregorian calendar and makes creating time slices a breeze. SELECT sum(volume) FROM my_stocks IN RANGE (today, -1month) GROUP BY day 14 The QuasarDB Technology
15. 15 Authentication and rights management User authentication is cryptographically strong and based on opaque security tokens, preventing security holes associated with passwords. Administrators can regenerate tokens and disable users at will, should an account be compromised. Users have default privileges, and the database supports fine-grained access control to handle even the most advanced security requirements. Data entitlement can be managed with table-based filters, for example, to prevent users who don’t have the proper privileges to see the most recent data. Audit trial Every update to the database is stored into an audit trail, stored as a timeseries. This timeseries can be protected through access control to prevent external modifications. The audit trail contains the nature of the operation, the concerned timeseries, the modified range, and the user at the origin of the modification. Backup, replication and migration QuasarDB supports intra-cluster replication through a configuration setting. This internal replication delivers continuous service should a node fail. Cluster to cluster replication is possible through an external tool developed by the QuasarDB team. This tool leverages the audit trail and thus allows one time replication, migration, and continuous updates. Backup can be performed at the file level, or, through a backup tool that also leverages the audit trail. Full and incremental backups are thus supported. Security QuasarDB supports traffic encryption based on AES-256 GCM. The key is 256-bit large and the MAC 128-bit large. Every packet is authenticated to detect accidental or intentional modifications. Session keys are securely exchanged using X25519. Each session key is used only once, which permits Perfect Forward Secrecy (PFS). This means that should an attacker manage to compromise a user certificate; she will not be able to decipher the messages from the past, only future messages. QuasarDB security isn’t limited to user’s management and authentication. It is also deeply integrated into the protocol. The server has several mechanisms to protect from Denial of Service (DoS) whether it’s through maliciously crafted packets or valid, but excessive user requests. For example, there is a server-side parameter that limits how large user packets may be, ensuring the server cannot be in a situation where it goes OOM because of user requests. QuasarDB’s cryptographic primitives are based on NaCl written by Daniel J. Bernstein, Tanja Lange, and Peter Schwabe. The QuasarDB Technology
16. Learn more at www.quasardb.net France 24 Rue Feydeau, F-75002 Paris United States 222 Broadway – 19th floor, New York, NY 10038 Contact us at contact@quasardb.net