PTSM - #2 - White Paper QuasarDB Technology

The QuasarDB Techology
From Big Data to Big Value
October 2019
Intended audiences
This document is intended for software architects, software developers, IT managers, and anyone
curious about the QuasarDB technology. In this white paper we will contextualize the QuasarDB
technology and then dig into the details of how it uniquely addresses the challenges of modern
data management.

View Slide

2
The QuasarDB Technology
Table of contents
Part 3: What does QuasarDB give me?
06
06
07
07
07
08
Ingest all the data you need at the speed you require
The model adapts to your need, not the other way around
Powerful and instant insights on all your data
Safely and securely manage your data
Unlimited and efficient storage
Low Total Cost of Ownership
Part 1: The case for timeseries databses
03
04
04
05
Relational databases
Numerical data and timeseries
Specificities of timeseries data management
Should I use a timeseries database?
Part 2: QuasarDB’s principles
Our vision
The principles
05
06
Part 4: Technology deep dive
Implementation
Zero-overhead architecture
Core engine
Clustering
Concurrency
Efficient networking
Smart indexing
Transaction
Storage
Compression
Queries
Authentication and rights management
Audit trail
Security
08
08
09
09
10
11
11
11
12
13
14
15
15
15

View Slide

Relational database
Numerical data and timeseries
1. The case for timeseries database
Annual size of the global Datasphere
Source: Data Age 2025, sponsored by Seagate with data from IDC Global Datasphere, Nov 2018
Zetabytes
175 ZB
2010
0
20
40
60
80
100
120
140
160
180
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025
Relational databases, and their companion language, SQL, are the safe, default choice when
it comes to data management. Their flexibility and the solidity of the relational model
enables them to cater to the typical needs of a business.
It is no surprise that SQL became a standard for data querying and that so many businesses
in the world depend on relational databases on a day to day basis.
However, the constraints of the relational model become a handicap when speed and
flexibility are a must.
Timeseries is the fastest growing database model, and for a reason. Stocks, sensors from
factories, autonomous vehicles, airplanes… Petabytes of data are generated every day, and
most of it can be represented as numerical timeseries.
The relational model isn’t optimized or even designed for timeseries. Foreign keys are a
terrible match with timeseries data as they have significant performance implications.
Additionally, typical timeseries needs such as trimming, resampling or alignment (“asof”)
are cumbersome and inefficient to implement in a relational database.
3

View Slide

Complete trend, starting with January 2013
Time Series
DBMS
Relational
DBMS
450
400
350
300
250
200
150
100
50
2013 2014 2015 2016 2017 2018 2019
Popularity Changes
Source : db-engines.com
Finally, timeseries are typically tagged with contextual information such as origin, device type,
data type. A rigid schema makes handling this metadata cumbersome and inefficient. Timeseries
databases enable faster querying speed, optimized disk storage, and convenience.
VOLUME – Because of its cumulative nature, timeseries tend to have a larger data set, resulting
in data management problems typically found when working on data warehouses. This creates
pain points at data insertion, disk usage, and scanning.
APPEND MOSTLY – The data is usually appended to existing timeseries. A database engine
aware of that can take smart decisions to optimize ingestion.
LOOSE RELATIONS – Although timeseries can be represented as tables, the association between
tables is loose when not inexistent. Additionally, metadata associated with the timeseries can be
cumbersome to represent in a purely relational model.
SPECIFIC OPERATIONS – Timeseries data have some specific operations such as resampling,
trimming, windowed joins that are hard to express in SQL and require an optimized engine to
perform fast enough.
INTRINSICALLY COLUMN ORIENTED – Timeseries are column oriented: columns are
significantly larger than rows and data tends to be aggregated within the same column. New
data is often highly correlated with previous data, enabling clever indexing and compression
techniques that generic databases cannot leverage.
Speciﬁcities of Timeseries data management
What makes timeseries so special?
4

View Slide

Time to insert 1GB in milliseconds
Lower is better
435,75
5
123,964
25,851
T
InfluxDB QuasarDB
imescaleDB
5
Should I use a timeseries database?
Today, the power of computers gives us a lot
of headroom, even with sub-optimal
database engines.
However explosive disk usage, slow query
time, the lack of convenience, and total cost
of ownership are many reasons that can
make you consider a timeseries database
early on in your project.
When reaching a threshold in volume or
complexity, a system doesn’t go slower: it
stops functioning and your business is in
jeopardy overnight.
Our mission is to
enable companies to
gain instant insights,
regardless of the
data volume.
store everything
forever, query it in
real time.
Our vision
QuasarDB was born in the financial market in 2008, as it became obvious that relational
databases were unfit for the challenges of big numerical data. Existing alternatives just
didn't deliver in terms of performance, convenience, or reliability.
We thus decided to build our engine using best practices and innovative technology to
achieve the best possible performance without compromising convenience or reliability.
2. QuasarDB’s principles

View Slide

The principles
Ingest all the dats you need at the speed you require.
QuasarDB delivers the fastest timeseries ingestion speed on the market through its
combination of ultra-optimized protocol, unique compression algorithms, and built-in
clustering.
QuasarDB is designed to ingest terabytes per day and manage petabytes.
Out of the box, QuasarDB comes with the tooling to ingest text files at maximum efficiency
without requiring you to write programs or scripts. Need to write your program?
The API takes care of all the little details so you can focus on the essential.
The model adapts to your need, not the other way around.
Timeseries are stored into tables. Contrary to a typical relational
database, tables have very low overhead, and QuasarDB supports
a very large number of tables. Because tables can be tagged with
arbitrary strings, and because these tags can be used in queries,
QuasarDB combines the flexibility typically found in NoSQL
database with the rigor of relational databases.The API takes care
of all the little details so you can focus on the essential.
QuasarDB was designed from the ground up with the following seven principles:
SPEED – Machines should wait on humans, not the other way around. Use the best algorithms.
Implement them efficiently. Don’t use scalability as an excuse for poor nominal performance.
SCALABILITY – However fast the engine is, the business will outgrow a single core, a single
socket, and a single machine. The engine must scale up, then scale out, without any headache.
STORAGE EFFICIENCY – Data grows forever. Leverage compression and smart encoding as
much as possible, enable the user to store as much as they want.
SAFETY – Preserve the data. Don’t surprise the user with behavior such as “eventual
consistency.” Be consistent. Be transactional. It’s easier to reason with.
SECURITY – Make unauthorized access extremely hard. Deploy in-depth security. Empower the
administrators to control and audit access.
CONVENIENCE AND FAMILIARITY – Don’t push the hard problems back to the users
(consistency, integrity, aggregations, etc.). Give users access to the data through familiar
interfaces.
UNIVERSALITY – Data is everywhere. Run everywhere. In the cloud. In the datacenter. On the
desktop. On the edge (e.g., support Intel and ARM architectures).
6
Say farewell to
artificial data
silos: QuasarDB
makes all your
data available.
3. What does QuasarDB give?

View Slide

T Technology
Eventual Consistency
Besides, QuasarDB gives you access to a powerful, transactional, distributed key-value store to
accommodate all the use cases where the relational model is too cumbersome.
Distributed storage: concepts,
algorithms, and implementations
Powerful and instant insights on all your data
Query your data through SQL or perform powerful
analytics with Python Pandas or R. Stream your results
through the BI tool of your choice with our ODBC
connector. Since access to the data is uniform
regardless of its age, insights are more accurate and
unconstrained. The power of QuasarDB engine
ensures you spend more time thinking and less time
waiting.
Safely and securely manage your data
QuasarDB gives you data integrity guarantees as any
database should. Its distributed transactional engine
enables you to perform complex queries over the
cluster with the guarantee you’ll have a consistent
view.
Writes to disks are safe and consistent, and the API
gives you clear feedback on the success of your
operations. Security features such as fine-grained
access control, cryptographically strong
authentication, and traffic encryption enable you to
use QuasarDB in environments where data
confidentiality is a must.
Unlimited and efficient storage
With its patent-pending Delta4C compression
algorithm, QuasarDB delivers efficient disk usage
without impacting performance. Disk usage can be
typically ten times less than with a relational database.
Highly compressed storage means reduced storage
and the possibility to have a higher tier of storage (for
example, move from SDD to NVMe). Because it’s
scale-out, QuasarDB can grow with your data need
and automatically re-shards and rebalances tables.
Contrary to other in-memory databases, QuasarDB
does not require you to explicitly load the data from
the disk before performing a query. It transparently
paginates data in and out.
Eventual consistency is a
consistency model used in
distributed computing to achieve
high availability that informally
guarantees, that, if no new updates
are made to a given data item,
eventually all accesses to that item
will return the last updated value.
(Wikipedia) One could say that
eventual consistency outsources
the challenges of data correctness
to the users.
In some use cases, this is an
acceptable tradeoff, as data
accuracy can be relaxed (for
example, the order of chat
messages sent to a user).
In most cases however, this greatly
increase the cognitive load on
users that need to compensate in
their code the potential
inaccuracies in results. That is why
QuasarDB gives consistency
guarantees for every update: to
make reasoning about the data
easier.
7
Systems using Eventual
Consistency result in
decreased system load
and increased system
avaibility but result in
increased cognitive com-
plexity for users and
developers.
Yaniv Pessach

View Slide

Low total cost of ownership
The combination of efficient storage, low CPU usage,
integration, and ease of use results in tremendous
savings for businesses. Deploying QuasarDB is cheaper
and faster than typical Big Data solutions:
› Data analysts use known tools such as Python
and SQL to interface with their data: no arcane,
proprietary language to learn and skills are
transferable.
› QuasarDB integrates ETL, distribution, storage,
analytics, orchestration, and connectivity in a
single package radically simplifying deployment
and administration, shortening projects time.
› Efficient storage means significantly reduced
storage costs.
› Speed means lower CPU usage, which directly
translates to lower hosting costs.
Max
Min
Timestamp
14:00 100 110
15:00 110 120
16:00 105 120
QuasarDB is column oriented in
memory and in disk.
When storing tables there are two
ways or storing the data: per row
or per column. When reading data
in memory, processors prefetch the
data adjacent to it. Thus,
reading continuous data is faster
than reading scattered data.
Row oriented organization favors
operations that work on rows,
whereas column-oriented favors
operations that work on columns.
Let us consider a table with min
and max values per hour.
Computing the minimum value for
all hours is a column oriented,
whereas computing the spread for
a given hour is row oriented.
Timeseries columns are
significantly larger than the rows,
and, most operations
are column oriented, making
timeseries benefit greatly from
column oriented storage.
Finally, organizing data as columns
enables compression to be more
efficient as patterns are more likely
to appear within the same column
than within the same row.
8
4. Technology deep dive
How does QuasarDB deliver these benefits? What sets
it apart from the competition? The answer is found in
the overwhelming amount innovation packaged into
the software.
Implementation
QuasarDB is written in modern C++ 17 with the most
critical parts handwritten in assembly. This delivers a
portable binary unconstrainted by a virtual machine or
complex dependencies. QuasarDB is available for
Intel-compatible and ARM architectures, both 32-bit
and 64-bit.
Zero-overhead architecture
As QuasarDB combines in a single binary the
orchestrator, the query engine, the aggregation
engine, and the persistence. Thus, it can use a
zero-copy architecture that prevents the efficiency
loss typically found in big data systems that combines
different software products. Data isn’t moved around:
it’s processed in place. For example, the memory
buffer from the network card can be written directly to
disk after sanity checks, removing overheads caused
by allocations and copies.
What does column-
oriented mean ?

View Slide

42,50
2
2,800
91
InfluxDB QuasarDB
TimescaleDB
Time to query 1GB in milliseconds
Lower is better
9
Core engine
At its heart, QuasarDB is a multithreaded, column-oriented timeseries database. It combines the
best of two worlds: the speed of in-memory databases with the capacity of long-term storage
through its hybrid engine.
The aggregation engine leverages the column-oriented nature of data for ultra-fast aggregations
and uses Single Instruction Multiple Data (SIMD) instructions whenever available. Aggregations
are made on-demand and leverage indexes when possible.
Clustering
QuasarDB transparently scales-out on an unlimited number of machines. Using a clustering
technology based on MIT’s Chord, which is based on distributed hash tables (DHT). Timeseries
rows are sharded across the cluster based on a duration block configurable for each table. For
example, if an user choses a 1 day shard for a timeseries, the timeseries data will be divided in
chunks of one day, and chunks will be distributed across the cluster for scalable storage and
aggregation capabilities. Timeseries sharding is user-configurable per timeseries. Data load is
automatically rebalanced (resharded) as the cluster grows, without a need to manually re-shard
the tables. This resharding phase is transparent and automatic, delivering true, elastic, timeseries
storage. The clustering is designed to deliver instant lookup with minimal network chatter.
When a client process requests a query, it will split it into atomic requests and send those
requests to the appropriate nodes. If the topology of the cluster is outdated (node loss, cluster
growth), the client will detect that and remap the nodes. Asynchronous cluster to cluster
replication is supported.

View Slide

Concurrency
Every server has a configurable number of partitions that will process requests concurrently.
It uses lock-free and wait-free memory structures to reduce frictions between each partition.
Partitions run within the same process to reduce overall memory usage and deliver better
performance.
A query is split by the client into smaller atomic requests, depending on the cluster topology at
the time the query runs. This is done automatically and optimally for the user’s comfort.
Atomic requests are then spread out on the cluster (map). The client then collects and
aggregates the result to present the answer to the user (reduce).
The splitting is done to offer the best possible compromise between resource management and
performance.
Queries are automatically parsed and deconstructed in smaller blocks. Aggregations
automatically benefit from the map/reduce strategy without any user intervention. For example,
if an user asks for the average value of a column, the average computation is distributed across
all necessary chunks of the timeseries.
10
Built-in, elastic, clustering.
Cluster-wide replication is available for maximum availability.
Clustering
Storage
Network
QuasarDB Daemon
The daemon leverages zero
copy on the network and disk
layer. It can even bypass the
operating system for
maximum performance. Each
node manages its storage
independently.

View Slide

Partitioning
11
Efficient networking
On the wire, QuasarDB has a zero-copy binary
protocol for optimal transitions, and its
network engine is multicore aware in order to
reduce inefficiencies due to context switching.
QuasarDB runs on any network card and is
optimized for Mellanox and Solarflare network
cards.
Smart indexing
QuasarDB features a zero-overhead - in terms
of memory and storage usage - index for
timeseries data. It delivers constant-time lookup
for timerrelated queries across the cluster.
Sparse indexes enable the engine to speed-up
numerical queries, whether it’s a filter (WHERE
clause) or an aggregation. These indexes deliver
the best compromise between disk usage and
queries acceleration in using a stochastic
approach. QuasarDB automatically creates
those indexes, removing a lot of work on the
plates of DBAs.
Transaction
QuasarDB supports distributed transaction using Multi Version Concurrency Control (MVCC).
Unless otherwise specified, operations are read-committed and instantly consistent: writes are
available everywhere after the operation is confirmed.
When a query or an API call returns, the result is consistent for when the query was made, making
it easier to reason about updates. The client does not have to “guess” the actual value of the
data. Transactions are managed using high precision timestamps.
When it needs to decide between consistency and availability, QuasarDB will choose consistency.
Nevertheless, it has built-in, configurable, retries mechanisms that will shield the user from short
service interruptions..
Storage
Query
Remote node
Lock-free in-memory
data structure
The client receives the result
for each atomic requests
and aggregate them together
to form the ﬁnal answer
Requests are sent by the client
to nodes holding the data necessary
to execute the atomic request
The client splits a query
into atomic requests
Query-aware pagination
Dispatcher
Each request runs
in an independent
partition
0
Partition Result
Client
Client
Partition Result
Request
Request
Partition
Request Answer
Request

View Slide

C1
Instead of using the default pagination mechanism of the operating system, QuasarDB uses an
eviction strategy that is aware of the queries made by the user, enabling smart pre-fetching and
eviction, delivering significantly better performance and lower memory usage.
As needed, QuasarDB will read, decompress, and load into memories column as queries come in.
When memory usage peaks, it will evict data using a scan-resistant strategy.
Storage
12
RAM
C2
Storage
New points
Compressed
C1 and C2
Compress
Acknowledgement
after persistence
Page-in
C2
RAM
C1
Storage
column 1 and 2 not in memory decompress
Lookup C1 and C2
Compressed
C1 and C2
Query on C1 and C2
Load C1 and C2

View Slide

Eviction
RAM
C2
C1
Page-out
When the memory reaches the conﬁgured maximum usage
the least-recently used columns are removed from memory
Storage
QuasarDB writes to disk in a way that minimizes data loss, corruptions, and inconsistencies in
case of failures. It computes a checksum of every block to detect data corruption and does not
delay writes.
With Levyx Helium, QuasarDB can mount the device directly, bypassing the filesystem and the
operating system completely for maximum performance and reliability.
Compression
QuasarDB compresses numerical data on disk using its custom Delta4C (patent pending)
compression algorithm that has been trained through machine learning on petabytes of data to
deliver optimal ratios at very high speed.
Compared to LZ4, the fastest dictionary-based compression to date, Delta4C is up to 12 times
faster and 60% more efficient.
For data that cannot be compressed with Delta4C, QuasarDB uses a modified LZ4 for timeseries
data that delivers the same compression at up to twice the speed.
13

View Slide

LZ4
1
2
12
Delta4C
QDB LZ4 LZ4
18 19
30
Delta4C
QDB LZ4
Average speed factor Ratio on integer timeseries
Queries
QuasarDB can be operated through a comprehensive and simple to use API available in all
popular languages.
It also supports SQL queries with timeseries extensions. Instead of inventing a brand-new
language and jailing users in a proprietary product, QuasarDB makes sure data scientists are
quickly up to speed.
The timeseries extensions simplify the writing of complex time-related queries while making them
easy to understand, even for someone unfamiliar with QuasarDB. The query language is aware of
the Gregorian calendar and makes creating time slices a breeze.
SELECT sum(volume) FROM my_stocks IN RANGE (today, -1month) GROUP BY day
14

View Slide

15
Authentication and rights management
User authentication is cryptographically strong and based on opaque security tokens, preventing
security holes associated with passwords.
Administrators can regenerate tokens and disable users at will, should an account be
compromised.
Users have default privileges, and the database supports fine-grained access control to handle
even the most advanced security requirements.
Data entitlement can be managed with table-based filters, for example, to prevent users who
don’t have the proper privileges to see the most recent data.
Audit trial
Every update to the database is stored into an audit trail, stored as a timeseries. This timeseries
can be protected through access control to prevent external modifications.
The audit trail contains the nature of the operation, the concerned timeseries, the modified
range, and the user at the origin of the modification.
Backup, replication and migration
QuasarDB supports intra-cluster replication through a configuration setting. This internal
replication delivers continuous service should a node fail.
Cluster to cluster replication is possible through an external tool developed by the QuasarDB
team. This tool leverages the audit trail and thus allows one time replication, migration, and
continuous updates.
Backup can be performed at the file level, or, through a backup tool that also leverages the audit
trail. Full and incremental backups are thus supported.
Security
QuasarDB supports traffic encryption based on AES-256 GCM. The key is 256-bit large and the
MAC 128-bit large. Every packet is authenticated to detect accidental or intentional
modifications.
Session keys are securely exchanged using X25519. Each session key is used only once, which
permits Perfect Forward Secrecy (PFS).
This means that should an attacker manage to compromise a user certificate; she will not be able
to decipher the messages from the past, only future messages.
QuasarDB security isn’t limited to user’s management and authentication. It is also deeply
integrated into the protocol.
The server has several mechanisms to protect from Denial of Service (DoS) whether it’s through
maliciously crafted packets or valid, but excessive user requests. For example, there is a
server-side parameter that limits how large user packets may be, ensuring the server cannot be
in a situation where it goes OOM because of user requests.
QuasarDB’s cryptographic primitives are based on NaCl written by Daniel J. Bernstein, Tanja
Lange, and Peter Schwabe.

View Slide

Learn more at www.quasardb.net
France
24 Rue Feydeau,
F-75002 Paris
United States
222 Broadway – 19th floor,
New York, NY 10038
Contact us at [email protected]

View Slide

PTSM - #2 - White Paper QuasarDB Technology

PTSM - #2 - White Paper QuasarDB Technology

TimeSeriesFr

More Decks by TimeSeriesFr

Other Decks in Technology

Featured

Transcript