Kestrel talk

Transcript

1. Kestrel Queue
2. https://s17t.net/ @s17tnet
3. Kestrel • Apache ActiveMQ • Apache Zookeeper • Linkedin Kafka • ...
6. Kestrel • Twitter • Active (last version Oct 2012) • Small (~6k Scala lines) • Easy to deploy (happy sysadmin) • Simple to integrate
7. Kestrel • Memory + Journal File (multiple files) • Fine tuning: • Max queues/in memory/journal size (in bytes or num items) • Max size on single item • Items/queues expiration
8. Nice features /1 • Live configuration, no restart needed • Fanout queues • set parent+child <item> – <item> will be written in 'parent' and 'child' queue – 'parent' and 'child' will be indipendent queues but • set parent <item2> – <item2> will be in both parent and child
9. Nice features /2 • 'peek' commad (a get without pop) • Reliable reads (with memcache) • get dirty_jobs/close/open • not so clean (see docs) • Stats!
10. data persistence with kestrel
11. “I dare you” • NO PERSISTENCE, NO DB • NO TRANSACTIONAL (only read)
13. (infinity) horizontal scalability
14. Kestrel $wget \ http://robey.github.com/kestrel/download/kestrel-2.3.4.zip … $java -jar kestrel_2.9.1-2.3.4.jar -f \ config/development.scala
15. A little detour... Memcached (from LiveJournal guy) “Free & open source, high-performance, distributed memory object caching system”
16. A little detour... Memcached (from LiveJournal guys) “Free & open source, high-performance, distributed memory object caching system” k1 ↔ v1 k2 ↔ v2 k3 ↔ v3 k4 ↔ v4 GET / SET
17. Kestrel SET (k1,v1) →{v1} SET (k2,v2) → {v2} SET (k2,v3) → {v3} … GET (k1) ← {v1} ….
18. Kestrel k1 ↔{v1, v2, v3, …, vn} K1 SET (k1, v1) → {v1} SET (k1, v2) → {v1, v2} SET (k1, v3) → {v1, v2, v3} GET k1 ← {v1} GET k1 ← {v2} FIFO
19. talk to kestrel memcache protocol (most commands) text protocol (limited) thrift (by apache)
20. talk to kestrel memcache protocol (most commands) text protocol (limited) thrift (by apache) + serialization (xml, json, object output, protocol buffer, <your exotic format here>)
23. Kestrel • Elements on single kestrel are strictly ordered but 'cluster' are loosely ordered • Round trip each GET/SET • Fanout queues are expansive q.size = #fanout * items • Disk space is not infinite • No persistence • No transactions
24. Will it blend ?
25. Will it blend ? • Vm with 4GB RAM, 2 CPU 4Ghz, local disk STAT total_items 552 774 274 (and couting)
28. code • https://github.com/nathanmarz/storm/wiki/Kestrel-and-Storm • http://code.google.com/p/spymemcached/
29. links • http://robey.github.com/kestrel/ • http://robey.github.com/kestrel/kestrel-tech-talk-jun2011.pdf (kestrel 2.0, Jun 2011) • https://github.com/robey/kestrel/blob/master/docs/guide.md • https://github.com/robey/kestrel/blob/master