IMDB Showdown - OrigoDB, Redis and Hekaton

From my presentation at NDC Oslo 2015

1. IMDB Showdown @robertfriberg robert@devrexlabs.com origodb.com
2. Speed of light vs spinning metal What Time Scale L1 Cache 0.5 ns 0.008 2 m L2 Cache 7 ns 0.23 RAM 60 ns 1 240 m 1 second 1K over Gbit network 10 µs 167 2.5 minutes 4K read SSD 150 µs 2500 Rotating disk seek 10 ms 167000 40000 km 46 hours
3. Price/GB vs GB/Server 1980- 2015 8 USD 6.480.000 USD 0.001 GB 2000 GB
4. In-memory stores • VoltDB • MemSQL • Hazelcast • Aerospike • Memcached • Oracle Coherence • Redis • Oracle Times Ten • SQL Server In-memory OLTP • SAP Hana • OrigoDB Key/value New SQL Hybrid ?
5. 1. SQL Server In-memory OLTP
6. B-trees and Transactions LOG DATA 64KB blocks w 8x8KB pages Logical BTREE of 8kb data pages In the buffer pool (cache) Buffer Manager Transactions append inserted, deleted, original and modified pages to the LOG CHECKPOINT
7. SQL Server In-memory OLTP SP Logging Latches Locks Buffer Manager I/O Native compiled SPs Minimal Logging and checkpointing Lock-free data structures Multi-version Currency control In Memory and Memory optimized data structures
8. Sql Server In-memory OLTP • Heka = Greek for 100 • Transparent for application • Integrates with Sql Server but with limitations • Enterprise license • 5-30x performance gain
9. Demo!
10. 2. Redis REmote DIctionary Server
11. • Hybrid Key/Value store • Value -> String | List | Set | Hash | SortedSet • Predefined Commands -> SET | GET | HSET | ... • Persistence: Snapshots + AOF • Highly optimized C, fast algorithms
12. Twitter using redis INCR next_user_id //returns 1000 HMSET user:1000 name bart password ¤¤¤hash¤¤¤ HSET users bart 1000 ZADD followers:1000 1401267618 1234 ZADD following:1234 1401267618 1000 INCR next_post_id => 10343 HMSET post:10343 user 1000 time $time body ’Ay Caramba’ RPUSH posts:1000 10343
13. 3. OrigoDB Build faster systems faster
14. What’s the problem? Service Layer Domain Layer Data Access Layer Relational Model Views/SP’s Cache
15. One simple idea... Keep state in memory Persist operations, not system state s0 s1 s2 op1 op2 Sn = apply(opn , Sn-1 )
16. ... with many names • System prevalance – Prevalyer, java • MongoDB op log • Redis AOF • Memory Image – Martin Fowler • VoltDB – logical logging • Akka persistence – logging per actor • Event Sourcing
17. OrigoDB Kernel Engine Model Storage App Code Server Command Query File Sql Event Store Custom Consistency Isolation concurrency Sends commands and queries Journaling Snapshots BinaryFormatter ProtoBuf JSON tcp JSON/http In-process calls Domain specific object-graph Domain specific operations Replication Ad-hoc queries Web ui Console or win svc
18. Complete history of events • Point in time • Debugging • Restore • Queries • Audit trail • New interpretations
19. Example – the model [Serializable] public class CommerceModel : Model { internal SortedDictionary<Guid, Customer> Customers { get; set; } internal SortedDictionary<Guid, Order> Orders { get; set; } internal SortedDictionary<Guid, Product> Products { get; set; } public CommerceModel() { Customers = new SortedDictionary<Guid, Customer>(); Orders = new SortedDictionary<Guid, Order>(); Products = new SortedDictionary<Guid, Product>(); } }
20. Command [Serializable] public class AddCustomer : Command<CommerceModel> { public readonly Guid Id; public readonly string Name; public AddCustomer(Guid id, String name) { Id = id; Name = name; } public override void Execute(CommerceModel model) { if (model.Customers.ContainsKey(Id)) Abort("Duplicate customer id"); var customer = new Customer {Id = Id, Name = Name}; model.Customers.Add(Id, customer); } }
21. Query [Serializable] public class CustomerById : Query<CommerceModel, CustomerView> { public readonly Guid Id; public CustomerById(Guid id) { Id = id; } public override CustomerView Execute(CommerceModel model) { if (!model.Customers.ContainsKey(Id)) throw new Exception("no such customer"); return new CustomerView(model.Customers[Id]); } }
22. Start your engines! static void Main(string[] args) { var engine = Engine.For<CommerceModel>(); Guid id = Guid.NewGuid(); var customerCommand = new AddCustomer(id, "Homer"); engine.Execute(customerCommand); var customerView = engine.Execute(new CustomerById(id)); Console.WriteLine(customerView.Name); Console.WriteLine("{0} orders", customerView.OrderIds.Count); Console.ReadLine(); }
23. Demo! Geekstream
24. 4. Conclusion The old, the new and the ugly
25. 100% ACID Out of the Box? SQL Server Redis OrigoDB Atomicity NO NO YES Consistency NO NO YES Isolation NO YES YES Durability YES YES YES
26. Comparison Matrix SQL REDIS ORIGO License/Cost $$ OSS/Free MIT/Free (+$) Language TSQL Commands + Lua C#/LINQ OLTP YES YES YES OLAP (indexing) YES NO YES In-process NO NO YES Througput/latency 3 1 2 Modeling Relational Fixed Multi Maturity 1 2 3 Size 250GB/DB Available RAM Available RAM
27. Thank you! • Try Origo! • Contribute, it’s open source • http://origodb.com • @robertfriberg, robert@devrexlabs.com