Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PTSM - #2 - White Paper QuasarDB Technology

TimeSeriesFr
November 05, 2019

PTSM - #2 - White Paper QuasarDB Technology

TimeSeriesFr

November 05, 2019
Tweet

More Decks by TimeSeriesFr

Other Decks in Technology

Transcript

  1. The QuasarDB Techology
    From Big Data to Big Value
    October 2019
    Intended audiences
    This document is intended for software architects, software developers, IT managers, and anyone
    curious about the QuasarDB technology. In this white paper we will contextualize the QuasarDB
    technology and then dig into the details of how it uniquely addresses the challenges of modern
    data management.

    View full-size slide

  2. 2
    The QuasarDB Technology
    Table of contents
    Part 3: What does QuasarDB give me?
    06
    06
    07
    07
    07
    08
    Ingest all the data you need at the speed you require
    The model adapts to your need, not the other way around
    Powerful and instant insights on all your data
    Safely and securely manage your data
    Unlimited and efficient storage
    Low Total Cost of Ownership
    Part 1: The case for timeseries databses
    03
    04
    04
    05
    Relational databases
    Numerical data and timeseries
    Specificities of timeseries data management
    Should I use a timeseries database?
    Part 2: QuasarDB’s principles
    Our vision
    The principles
    05
    06
    Part 4: Technology deep dive
    Implementation
    Zero-overhead architecture
    Core engine
    Clustering
    Concurrency
    Efficient networking
    Smart indexing
    Transaction
    Storage
    Compression
    Queries
    Authentication and rights management
    Audit trail
    Security
    08
    08
    09
    09
    10
    11
    11
    11
    12
    13
    14
    15
    15
    15

    View full-size slide

  3. The QuasarDB Technology
    Relational database
    Numerical data and timeseries
    1. The case for timeseries database
    Annual size of the global Datasphere
    Source: Data Age 2025, sponsored by Seagate with data from IDC Global Datasphere, Nov 2018
    Zetabytes
    175 ZB
    2010
    0
    20
    40
    60
    80
    100
    120
    140
    160
    180
    2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025
    Relational databases, and their companion language, SQL, are the safe, default choice when
    it comes to data management. Their flexibility and the solidity of the relational model
    enables them to cater to the typical needs of a business.
    It is no surprise that SQL became a standard for data querying and that so many businesses
    in the world depend on relational databases on a day to day basis.
    However, the constraints of the relational model become a handicap when speed and
    flexibility are a must.
    Timeseries is the fastest growing database model, and for a reason. Stocks, sensors from
    factories, autonomous vehicles, airplanes… Petabytes of data are generated every day, and
    most of it can be represented as numerical timeseries.
    The relational model isn’t optimized or even designed for timeseries. Foreign keys are a
    terrible match with timeseries data as they have significant performance implications.
    Additionally, typical timeseries needs such as trimming, resampling or alignment (“asof”)
    are cumbersome and inefficient to implement in a relational database.
    3

    View full-size slide

  4. The QuasarDB Technology
    Complete trend, starting with January 2013
    Time Series
    DBMS
    Relational
    DBMS
    450
    400
    350
    300
    250
    200
    150
    100
    50
    2013 2014 2015 2016 2017 2018 2019
    Popularity Changes
    Source : db-engines.com
    Finally, timeseries are typically tagged with contextual information such as origin, device type,
    data type. A rigid schema makes handling this metadata cumbersome and inefficient. Timeseries
    databases enable faster querying speed, optimized disk storage, and convenience.
    VOLUME – Because of its cumulative nature, timeseries tend to have a larger data set, resulting
    in data management problems typically found when working on data warehouses. This creates
    pain points at data insertion, disk usage, and scanning.
    APPEND MOSTLY – The data is usually appended to existing timeseries. A database engine
    aware of that can take smart decisions to optimize ingestion.
    LOOSE RELATIONS – Although timeseries can be represented as tables, the association between
    tables is loose when not inexistent. Additionally, metadata associated with the timeseries can be
    cumbersome to represent in a purely relational model.
    SPECIFIC OPERATIONS – Timeseries data have some specific operations such as resampling,
    trimming, windowed joins that are hard to express in SQL and require an optimized engine to
    perform fast enough.
    INTRINSICALLY COLUMN ORIENTED – Timeseries are column oriented: columns are
    significantly larger than rows and data tends to be aggregated within the same column. New
    data is often highly correlated with previous data, enabling clever indexing and compression
    techniques that generic databases cannot leverage.
    Specificities of Timeseries data management
    What makes timeseries so special?
    4

    View full-size slide

  5. Time to insert 1GB in milliseconds
    Lower is better
    435,75
    5
    123,964
    25,851
    T
    InfluxDB QuasarDB
    imescaleDB
    5
    Should I use a timeseries database?
    Today, the power of computers gives us a lot
    of headroom, even with sub-optimal
    database engines.
    However explosive disk usage, slow query
    time, the lack of convenience, and total cost
    of ownership are many reasons that can
    make you consider a timeseries database
    early on in your project.
    When reaching a threshold in volume or
    complexity, a system doesn’t go slower: it
    stops functioning and your business is in
    jeopardy overnight.
    Our mission is to
    enable companies to
    gain instant insights,
    regardless of the
    data volume.
    store everything
    forever, query it in
    real time.
    Our vision
    QuasarDB was born in the financial market in 2008, as it became obvious that relational
    databases were unfit for the challenges of big numerical data. Existing alternatives just
    didn't deliver in terms of performance, convenience, or reliability.
    We thus decided to build our engine using best practices and innovative technology to
    achieve the best possible performance without compromising convenience or reliability.
    2. QuasarDB’s principles
    The QuasarDB Technology

    View full-size slide

  6. The principles
    The QuasarDB Technology
    Ingest all the dats you need at the speed you require.
    QuasarDB delivers the fastest timeseries ingestion speed on the market through its
    combination of ultra-optimized protocol, unique compression algorithms, and built-in
    clustering.
    QuasarDB is designed to ingest terabytes per day and manage petabytes.
    Out of the box, QuasarDB comes with the tooling to ingest text files at maximum efficiency
    without requiring you to write programs or scripts. Need to write your program?
    The API takes care of all the little details so you can focus on the essential.
    The model adapts to your need, not the other way around.
    Timeseries are stored into tables. Contrary to a typical relational
    database, tables have very low overhead, and QuasarDB supports
    a very large number of tables. Because tables can be tagged with
    arbitrary strings, and because these tags can be used in queries,
    QuasarDB combines the flexibility typically found in NoSQL
    database with the rigor of relational databases.The API takes care
    of all the little details so you can focus on the essential.
    QuasarDB was designed from the ground up with the following seven principles:
    SPEED – Machines should wait on humans, not the other way around. Use the best algorithms.
    Implement them efficiently. Don’t use scalability as an excuse for poor nominal performance.
    SCALABILITY – However fast the engine is, the business will outgrow a single core, a single
    socket, and a single machine. The engine must scale up, then scale out, without any headache.
    STORAGE EFFICIENCY – Data grows forever. Leverage compression and smart encoding as
    much as possible, enable the user to store as much as they want.
    SAFETY – Preserve the data. Don’t surprise the user with behavior such as “eventual
    consistency.” Be consistent. Be transactional. It’s easier to reason with.
    SECURITY – Make unauthorized access extremely hard. Deploy in-depth security. Empower the
    administrators to control and audit access.
    CONVENIENCE AND FAMILIARITY – Don’t push the hard problems back to the users
    (consistency, integrity, aggregations, etc.). Give users access to the data through familiar
    interfaces.
    UNIVERSALITY – Data is everywhere. Run everywhere. In the cloud. In the datacenter. On the
    desktop. On the edge (e.g., support Intel and ARM architectures).
    6
    Say farewell to
    artificial data
    silos: QuasarDB
    makes all your
    data available.
    3. What does QuasarDB give?

    View full-size slide

  7. T Technology
    Eventual Consistency
    Besides, QuasarDB gives you access to a powerful, transactional, distributed key-value store to
    accommodate all the use cases where the relational model is too cumbersome.
    Distributed storage: concepts,
    algorithms, and implementations
    Powerful and instant insights on all your data
    Query your data through SQL or perform powerful
    analytics with Python Pandas or R. Stream your results
    through the BI tool of your choice with our ODBC
    connector. Since access to the data is uniform
    regardless of its age, insights are more accurate and
    unconstrained. The power of QuasarDB engine
    ensures you spend more time thinking and less time
    waiting.
    Safely and securely manage your data
    QuasarDB gives you data integrity guarantees as any
    database should. Its distributed transactional engine
    enables you to perform complex queries over the
    cluster with the guarantee you’ll have a consistent
    view.
    Writes to disks are safe and consistent, and the API
    gives you clear feedback on the success of your
    operations. Security features such as fine-grained
    access control, cryptographically strong
    authentication, and traffic encryption enable you to
    use QuasarDB in environments where data
    confidentiality is a must.
    Unlimited and efficient storage
    With its patent-pending Delta4C compression
    algorithm, QuasarDB delivers efficient disk usage
    without impacting performance. Disk usage can be
    typically ten times less than with a relational database.
    Highly compressed storage means reduced storage
    and the possibility to have a higher tier of storage (for
    example, move from SDD to NVMe). Because it’s
    scale-out, QuasarDB can grow with your data need
    and automatically re-shards and rebalances tables.
    Contrary to other in-memory databases, QuasarDB
    does not require you to explicitly load the data from
    the disk before performing a query. It transparently
    paginates data in and out.
    Eventual consistency is a
    consistency model used in
    distributed computing to achieve
    high availability that informally
    guarantees, that, if no new updates
    are made to a given data item,
    eventually all accesses to that item
    will return the last updated value.
    (Wikipedia) One could say that
    eventual consistency outsources
    the challenges of data correctness
    to the users.
    In some use cases, this is an
    acceptable tradeoff, as data
    accuracy can be relaxed (for
    example, the order of chat
    messages sent to a user).
    In most cases however, this greatly
    increase the cognitive load on
    users that need to compensate in
    their code the potential
    inaccuracies in results. That is why
    QuasarDB gives consistency
    guarantees for every update: to
    make reasoning about the data
    easier.
    7
    Systems using Eventual
    Consistency result in
    decreased system load
    and increased system
    avaibility but result in
    increased cognitive com-
    plexity for users and
    developers.
    Yaniv Pessach
    The QuasarDB Technology

    View full-size slide

  8. The QuasarDB Technology
    Low total cost of ownership
    The combination of efficient storage, low CPU usage,
    integration, and ease of use results in tremendous
    savings for businesses. Deploying QuasarDB is cheaper
    and faster than typical Big Data solutions:
    › Data analysts use known tools such as Python
    and SQL to interface with their data: no arcane,
    proprietary language to learn and skills are
    transferable.
    › QuasarDB integrates ETL, distribution, storage,
    analytics, orchestration, and connectivity in a
    single package radically simplifying deployment
    and administration, shortening projects time.
    › Efficient storage means significantly reduced
    storage costs.
    › Speed means lower CPU usage, which directly
    translates to lower hosting costs.
    Max
    Min
    Timestamp
    14:00 100 110
    15:00 110 120
    16:00 105 120
    QuasarDB is column oriented in
    memory and in disk.
    When storing tables there are two
    ways or storing the data: per row
    or per column. When reading data
    in memory, processors prefetch the
    data adjacent to it. Thus,
    reading continuous data is faster
    than reading scattered data.
    Row oriented organization favors
    operations that work on rows,
    whereas column-oriented favors
    operations that work on columns.
    Let us consider a table with min
    and max values per hour.
    Computing the minimum value for
    all hours is a column oriented,
    whereas computing the spread for
    a given hour is row oriented.
    Timeseries columns are
    significantly larger than the rows,
    and, most operations
    are column oriented, making
    timeseries benefit greatly from
    column oriented storage.
    Finally, organizing data as columns
    enables compression to be more
    efficient as patterns are more likely
    to appear within the same column
    than within the same row.
    8
    4. Technology deep dive
    How does QuasarDB deliver these benefits? What sets
    it apart from the competition? The answer is found in
    the overwhelming amount innovation packaged into
    the software.
    Implementation
    QuasarDB is written in modern C++ 17 with the most
    critical parts handwritten in assembly. This delivers a
    portable binary unconstrainted by a virtual machine or
    complex dependencies. QuasarDB is available for
    Intel-compatible and ARM architectures, both 32-bit
    and 64-bit.
    Zero-overhead architecture
    As QuasarDB combines in a single binary the
    orchestrator, the query engine, the aggregation
    engine, and the persistence. Thus, it can use a
    zero-copy architecture that prevents the efficiency
    loss typically found in big data systems that combines
    different software products. Data isn’t moved around:
    it’s processed in place. For example, the memory
    buffer from the network card can be written directly to
    disk after sanity checks, removing overheads caused
    by allocations and copies.
    What does column-
    oriented mean ?

    View full-size slide

  9. 42,50
    2
    2,800
    91
    InfluxDB QuasarDB
    TimescaleDB
    Time to query 1GB in milliseconds
    Lower is better
    9
    Core engine
    At its heart, QuasarDB is a multithreaded, column-oriented timeseries database. It combines the
    best of two worlds: the speed of in-memory databases with the capacity of long-term storage
    through its hybrid engine.
    The aggregation engine leverages the column-oriented nature of data for ultra-fast aggregations
    and uses Single Instruction Multiple Data (SIMD) instructions whenever available. Aggregations
    are made on-demand and leverage indexes when possible.
    Clustering
    QuasarDB transparently scales-out on an unlimited number of machines. Using a clustering
    technology based on MIT’s Chord, which is based on distributed hash tables (DHT). Timeseries
    rows are sharded across the cluster based on a duration block configurable for each table. For
    example, if an user choses a 1 day shard for a timeseries, the timeseries data will be divided in
    chunks of one day, and chunks will be distributed across the cluster for scalable storage and
    aggregation capabilities. Timeseries sharding is user-configurable per timeseries. Data load is
    automatically rebalanced (resharded) as the cluster grows, without a need to manually re-shard
    the tables. This resharding phase is transparent and automatic, delivering true, elastic, timeseries
    storage. The clustering is designed to deliver instant lookup with minimal network chatter.
    When a client process requests a query, it will split it into atomic requests and send those
    requests to the appropriate nodes. If the topology of the cluster is outdated (node loss, cluster
    growth), the client will detect that and remap the nodes. Asynchronous cluster to cluster
    replication is supported.
    The QuasarDB Technology

    View full-size slide

  10. The QuasarDB Technology
    Concurrency
    Every server has a configurable number of partitions that will process requests concurrently.
    It uses lock-free and wait-free memory structures to reduce frictions between each partition.
    Partitions run within the same process to reduce overall memory usage and deliver better
    performance.
    A query is split by the client into smaller atomic requests, depending on the cluster topology at
    the time the query runs. This is done automatically and optimally for the user’s comfort.
    Atomic requests are then spread out on the cluster (map). The client then collects and
    aggregates the result to present the answer to the user (reduce).
    The splitting is done to offer the best possible compromise between resource management and
    performance.
    Queries are automatically parsed and deconstructed in smaller blocks. Aggregations
    automatically benefit from the map/reduce strategy without any user intervention. For example,
    if an user asks for the average value of a column, the average computation is distributed across
    all necessary chunks of the timeseries.
    10
    Built-in, elastic, clustering.
    Cluster-wide replication is available for maximum availability.
    Clustering
    Storage
    Network
    QuasarDB Daemon
    The daemon leverages zero
    copy on the network and disk
    layer. It can even bypass the
    operating system for
    maximum performance. Each
    node manages its storage
    independently.

    View full-size slide

  11. Partitioning
    11
    Efficient networking
    On the wire, QuasarDB has a zero-copy binary
    protocol for optimal transitions, and its
    network engine is multicore aware in order to
    reduce inefficiencies due to context switching.
    QuasarDB runs on any network card and is
    optimized for Mellanox and Solarflare network
    cards.
    Smart indexing
    QuasarDB features a zero-overhead - in terms
    of memory and storage usage - index for
    timeseries data. It delivers constant-time lookup
    for timerrelated queries across the cluster.
    Sparse indexes enable the engine to speed-up
    numerical queries, whether it’s a filter (WHERE
    clause) or an aggregation. These indexes deliver
    the best compromise between disk usage and
    queries acceleration in using a stochastic
    approach. QuasarDB automatically creates
    those indexes, removing a lot of work on the
    plates of DBAs.
    Transaction
    QuasarDB supports distributed transaction using Multi Version Concurrency Control (MVCC).
    Unless otherwise specified, operations are read-committed and instantly consistent: writes are
    available everywhere after the operation is confirmed.
    When a query or an API call returns, the result is consistent for when the query was made, making
    it easier to reason about updates. The client does not have to “guess” the actual value of the
    data. Transactions are managed using high precision timestamps.
    When it needs to decide between consistency and availability, QuasarDB will choose consistency.
    Nevertheless, it has built-in, configurable, retries mechanisms that will shield the user from short
    service interruptions..
    Storage
    Query
    Remote node
    Lock-free in-memory
    data structure
    The client receives the result
    for each atomic requests
    and aggregate them together
    to form the final answer
    Requests are sent by the client
    to nodes holding the data necessary
    to execute the atomic request
    The client splits a query
    into atomic requests
    Query-aware pagination
    Dispatcher
    Each request runs
    in an independent
    partition
    0
    Partition Result
    Client
    Client
    Partition Result
    Request
    Request
    Partition
    Request Answer
    Request
    The QuasarDB Technology

    View full-size slide

  12. The QuasarDB Technology
    C1
    Instead of using the default pagination mechanism of the operating system, QuasarDB uses an
    eviction strategy that is aware of the queries made by the user, enabling smart pre-fetching and
    eviction, delivering significantly better performance and lower memory usage.
    As needed, QuasarDB will read, decompress, and load into memories column as queries come in.
    When memory usage peaks, it will evict data using a scan-resistant strategy.
    Storage
    12
    RAM
    C2
    Storage
    New points
    Compressed
    C1 and C2
    Compress
    Acknowledgement
    after persistence
    Page-in
    C2
    RAM
    C1
    Storage
    column 1 and 2 not in memory decompress
    Lookup C1 and C2
    Compressed
    C1 and C2
    Query on C1 and C2
    Load C1 and C2

    View full-size slide

  13. Eviction
    RAM
    C2
    C1
    Page-out
    When the memory reaches the configured maximum usage
    the least-recently used columns are removed from memory
    Storage
    QuasarDB writes to disk in a way that minimizes data loss, corruptions, and inconsistencies in
    case of failures. It computes a checksum of every block to detect data corruption and does not
    delay writes.
    With Levyx Helium, QuasarDB can mount the device directly, bypassing the filesystem and the
    operating system completely for maximum performance and reliability.
    Compression
    QuasarDB compresses numerical data on disk using its custom Delta4C (patent pending)
    compression algorithm that has been trained through machine learning on petabytes of data to
    deliver optimal ratios at very high speed.
    Compared to LZ4, the fastest dictionary-based compression to date, Delta4C is up to 12 times
    faster and 60% more efficient.
    For data that cannot be compressed with Delta4C, QuasarDB uses a modified LZ4 for timeseries
    data that delivers the same compression at up to twice the speed.
    13
    The QuasarDB Technology

    View full-size slide

  14. LZ4
    1
    2
    12
    Delta4C
    QDB LZ4 LZ4
    18 19
    30
    Delta4C
    QDB LZ4
    Average speed factor Ratio on integer timeseries
    Queries
    QuasarDB can be operated through a comprehensive and simple to use API available in all
    popular languages.
    It also supports SQL queries with timeseries extensions. Instead of inventing a brand-new
    language and jailing users in a proprietary product, QuasarDB makes sure data scientists are
    quickly up to speed.
    The timeseries extensions simplify the writing of complex time-related queries while making them
    easy to understand, even for someone unfamiliar with QuasarDB. The query language is aware of
    the Gregorian calendar and makes creating time slices a breeze.
    SELECT sum(volume) FROM my_stocks IN RANGE (today, -1month) GROUP BY day
    14
    The QuasarDB Technology

    View full-size slide

  15. 15
    Authentication and rights management
    User authentication is cryptographically strong and based on opaque security tokens, preventing
    security holes associated with passwords.
    Administrators can regenerate tokens and disable users at will, should an account be
    compromised.
    Users have default privileges, and the database supports fine-grained access control to handle
    even the most advanced security requirements.
    Data entitlement can be managed with table-based filters, for example, to prevent users who
    don’t have the proper privileges to see the most recent data.
    Audit trial
    Every update to the database is stored into an audit trail, stored as a timeseries. This timeseries
    can be protected through access control to prevent external modifications.
    The audit trail contains the nature of the operation, the concerned timeseries, the modified
    range, and the user at the origin of the modification.
    Backup, replication and migration
    QuasarDB supports intra-cluster replication through a configuration setting. This internal
    replication delivers continuous service should a node fail.
    Cluster to cluster replication is possible through an external tool developed by the QuasarDB
    team. This tool leverages the audit trail and thus allows one time replication, migration, and
    continuous updates.
    Backup can be performed at the file level, or, through a backup tool that also leverages the audit
    trail. Full and incremental backups are thus supported.
    Security
    QuasarDB supports traffic encryption based on AES-256 GCM. The key is 256-bit large and the
    MAC 128-bit large. Every packet is authenticated to detect accidental or intentional
    modifications.
    Session keys are securely exchanged using X25519. Each session key is used only once, which
    permits Perfect Forward Secrecy (PFS).
    This means that should an attacker manage to compromise a user certificate; she will not be able
    to decipher the messages from the past, only future messages.
    QuasarDB security isn’t limited to user’s management and authentication. It is also deeply
    integrated into the protocol.
    The server has several mechanisms to protect from Denial of Service (DoS) whether it’s through
    maliciously crafted packets or valid, but excessive user requests. For example, there is a
    server-side parameter that limits how large user packets may be, ensuring the server cannot be
    in a situation where it goes OOM because of user requests.
    QuasarDB’s cryptographic primitives are based on NaCl written by Daniel J. Bernstein, Tanja
    Lange, and Peter Schwabe.
    The QuasarDB Technology

    View full-size slide

  16. Learn more at www.quasardb.net
    France
    24 Rue Feydeau,
    F-75002 Paris
    United States
    222 Broadway – 19th floor,
    New York, NY 10038
    Contact us at [email protected]

    View full-size slide