Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IMDB Showdown - OrigoDB, Redis and Hekaton

IMDB Showdown - OrigoDB, Redis and Hekaton

From my presentation at NDC Oslo 2015

Robert Friberg

June 17, 2015
Tweet

More Decks by Robert Friberg

Other Decks in Technology

Transcript

  1. Speed of light vs spinning metal What Time Scale L1

    Cache 0.5 ns 0.008 2 m L2 Cache 7 ns 0.23 RAM 60 ns 1 240 m 1 second 1K over Gbit network 10 µs 167 2.5 minutes 4K read SSD 150 µs 2500 Rotating disk seek 10 ms 167000 40000 km 46 hours
  2. In-memory stores • VoltDB • MemSQL • Hazelcast • Aerospike

    • Memcached • Oracle Coherence • Redis • Oracle Times Ten • SQL Server In-memory OLTP • SAP Hana • OrigoDB Key/value New SQL Hybrid ?
  3. B-trees and Transactions LOG DATA 64KB blocks w 8x8KB pages

    Logical BTREE of 8kb data pages In the buffer pool (cache) Buffer Manager Transactions append inserted, deleted, original and modified pages to the LOG CHECKPOINT
  4. SQL Server In-memory OLTP SP Logging Latches Locks Buffer Manager

    I/O Native compiled SPs Minimal Logging and checkpointing Lock-free data structures Multi-version Currency control In Memory and Memory optimized data structures
  5. Sql Server In-memory OLTP • Heka = Greek for 100

    • Transparent for application • Integrates with Sql Server but with limitations • Enterprise license • 5-30x performance gain
  6. • Hybrid Key/Value store • Value -> String | List

    | Set | Hash | SortedSet • Predefined Commands -> SET | GET | HSET | ... • Persistence: Snapshots + AOF • Highly optimized C, fast algorithms
  7. Twitter using redis INCR next_user_id //returns 1000 HMSET user:1000 name

    bart password ¤¤¤hash¤¤¤ HSET users bart 1000 ZADD followers:1000 1401267618 1234 ZADD following:1234 1401267618 1000 INCR next_post_id => 10343 HMSET post:10343 user 1000 time $time body ’Ay Caramba’ RPUSH posts:1000 10343
  8. One simple idea... Keep state in memory Persist operations, not

    system state s0 s1 s2 op1 op2 Sn = apply(opn , Sn-1 )
  9. ... with many names • System prevalance – Prevalyer, java

    • MongoDB op log • Redis AOF • Memory Image – Martin Fowler • VoltDB – logical logging • Akka persistence – logging per actor • Event Sourcing
  10. OrigoDB Kernel Engine Model Storage App Code Server Command Query

    File Sql Event Store Custom Consistency Isolation concurrency Sends commands and queries Journaling Snapshots BinaryFormatter ProtoBuf JSON tcp JSON/http In-process calls Domain specific object-graph Domain specific operations Replication Ad-hoc queries Web ui Console or win svc
  11. Complete history of events • Point in time • Debugging

    • Restore • Queries • Audit trail • New interpretations
  12. Example – the model [Serializable] public class CommerceModel : Model

    { internal SortedDictionary<Guid, Customer> Customers { get; set; } internal SortedDictionary<Guid, Order> Orders { get; set; } internal SortedDictionary<Guid, Product> Products { get; set; } public CommerceModel() { Customers = new SortedDictionary<Guid, Customer>(); Orders = new SortedDictionary<Guid, Order>(); Products = new SortedDictionary<Guid, Product>(); } }
  13. Command [Serializable] public class AddCustomer : Command<CommerceModel> { public readonly

    Guid Id; public readonly string Name; public AddCustomer(Guid id, String name) { Id = id; Name = name; } public override void Execute(CommerceModel model) { if (model.Customers.ContainsKey(Id)) Abort("Duplicate customer id"); var customer = new Customer {Id = Id, Name = Name}; model.Customers.Add(Id, customer); } }
  14. Query [Serializable] public class CustomerById : Query<CommerceModel, CustomerView> { public

    readonly Guid Id; public CustomerById(Guid id) { Id = id; } public override CustomerView Execute(CommerceModel model) { if (!model.Customers.ContainsKey(Id)) throw new Exception("no such customer"); return new CustomerView(model.Customers[Id]); } }
  15. Start your engines! static void Main(string[] args) { var engine

    = Engine.For<CommerceModel>(); Guid id = Guid.NewGuid(); var customerCommand = new AddCustomer(id, "Homer"); engine.Execute(customerCommand); var customerView = engine.Execute(new CustomerById(id)); Console.WriteLine(customerView.Name); Console.WriteLine("{0} orders", customerView.OrderIds.Count); Console.ReadLine(); }
  16. 100% ACID Out of the Box? SQL Server Redis OrigoDB

    Atomicity NO NO YES Consistency NO NO YES Isolation NO YES YES Durability YES YES YES
  17. Comparison Matrix SQL REDIS ORIGO License/Cost $$ OSS/Free MIT/Free (+$)

    Language TSQL Commands + Lua C#/LINQ OLTP YES YES YES OLAP (indexing) YES NO YES In-process NO NO YES Througput/latency 3 1 2 Modeling Relational Fixed Multi Maturity 1 2 3 Size 250GB/DB Available RAM Available RAM