Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Marconi from MongoDB's POV

C05edcc8a57f64b4e040d94ad89cee57?s=47 flaper87
October 26, 2013

Marconi from MongoDB's POV

This talk explains some of the issues that have been faced during Marconi's development and how they've been overcome.



October 26, 2013

More Decks by flaper87

Other Decks in Programming


  1. @flaper87 Marconi from MongoDB's POV

  2. @flaper87 Steps out of hell: Marconi, 2 mins intro Stamped

    with MongoDB Welcome to indexes hell Fight race conditions Other ideas Takeaways
  3. Who's this dude? @flaper87 (a.k.a, Flavio Percoco): Very proud Red

    Hatter, who's also part of the MongoDB Master team and has contributed to Open Source since... forever. One of those who thinks about programming when he's not programming. member of
  4. @flaper87 Marconi, 2 mins intro Queues as a Service

  5. @flaper87 Marconi, 2 mins intro Notifications as a Service Queues

    as a Service
  6. @flaper87 Marconi, 2 mins intro All that, using lego bricks

  7. @flaper87 M ongoDB

  8. @flaper87 Name Field ------------------- scope -> p_q expires -> e

    ttl -> t uuid -> u claim -> c marker -> k Message Schema
  9. @flaper87 Name Field ------------------- id -> id expires -> e

    Claim Schema
  10. @flaper87 Compound Indexes

  11. @flaper87 {Eq, Sort, Range} Index order matters

  12. @flaper87 [start, End) Pure evilness →

  13. @flaper87 #Hint: Use hints ;)

  14. @flaper87 Avoid counts!

  15. @flaper87 Aggregate

  16. @flaper87 Race Fighters No `limit` for updates...

  17. @flaper87 Race Fighters FIFO … when ObjectID is not enough

  18. @flaper87 Race Fighters Use TTL Collections tweak expiration field

  19. @flaper87 Race Fighters Handling connection errors...

  20. @flaper87 Race Fighters Parsing Errors

  21. @flaper87 scale Multiple databases – q+p hash distribution

  22. @flaper87 scale Marconi Partitions – Multiple MongoDB nodes

  23. @flaper87 hacks # NOTE(flaper87): Dirty hack! # This sets the

    expiration time to # `expires` on messages that would # expire before claim. Transaction-less
  24. @flaper87 Ideas 1 collection per project • Smaller indexes •

    Fewer lookups
  25. @flaper87 Takeaways Compound indexes are good Range queries are evil

    Transaction-less trade-off Database lock can be avoided OIDs are awesome but not fully reliable
  26. @flaper87 Q&A Thanks we're hiring http:/ /openstack.redhat.com/ http:/ /www.openstack.org/