In-memory database

A slide about Aerospike and Apache Ingnite

Transcript

2. Problems - Multiple Redis instances on 1 server - Each instance contains some types of data - 200000 users active /s - About 5 milions records in memory - Master-Slave structures * Redis 1 instances → 1 core → Bottleneck read/write * No cluster * Scale out ???
4. Requirements • Horizontal Scale • Fast read for real time access • Use full core of CPU • High availability
5. Replacements • Aerospike • Redis cluster • Apache ignite
6. Aerospike • Written in C • Opensource • Community Edition (free) and Enterprise Edititon • 1177 stars on Github
7. Aerospike • Sharding supported • ACID (Atomicity, Consistency, Isolation, Durability) • Data can be saved to disk or RAM or hybrid storage (indexes in RAM, data in disk) • Self managing • Optimized for SSD • Reduce money (Less server, SSD is more cheaper than RAM) • Proved by AppNexus, Inmobi,...
9. Aerospike • Multicast • No master/slave node. Each node contains some different primary shards and some different replica shards
10. Storage • Pure DRAM without persistence • DRAM with persistence • Flash storage (SSDs)
11. Namespace – 1 database = many namespaces – How data is stored: RAM or DISK – How many replicas – When record expire
12. Sets • Group of records • 1 namespace = many sets
13. Record • Key • Meta-data (generation, time to live) • Bins (1 or more bins) → as rows and their value in DBMS
14. Apache Ignite Many features • Written in java • In-memory database • Data grid • Accelator for Hadoop and Spark ….
17. Shared RDDs among Spark jobs
18. Indexed SQL Query • SparkSQL does not index RDD • IgniteSQL can index its cached data by field or using lucene index for text. Examples: val cacheRdd = igniteContext.fromCache("partitioned") cacheRdd.savePairs(sparkContext.parallelize(1 to 10000, 10).map(i => (i, i))) val cacheRdd = igniteContext.fromCache("partitioned") val result = cacheRdd.sql( "select _val from Integer where val > ? and val < ?",10,100)
19. public class Person implements Serializable { /** Person ID (indexed). */ @QuerySqlField(index = true) private long id; /** First name (not-indexed). */ @QuerySqlField private String name; /** Resume text (create LUCENE-based TEXT index for this field). */ @QueryTextField private String resume; /** Salary (indexed). */ @QuerySqlField(index = true) private double salary; ….. }