Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Marconi from MongoDB's POV

flaper87
October 26, 2013

Marconi from MongoDB's POV

This talk explains some of the issues that have been faced during Marconi's development and how they've been overcome.

flaper87

October 26, 2013
Tweet

More Decks by flaper87

Other Decks in Programming

Transcript

  1. @flaper87
    Marconi from MongoDB's POV

    View Slide

  2. @flaper87
    Steps out of hell:
    Marconi, 2 mins intro
    Stamped with MongoDB
    Welcome to indexes hell
    Fight race conditions
    Other ideas
    Takeaways

    View Slide

  3. Who's this
    dude?
    @flaper87 (a.k.a, Flavio Percoco):
    Very proud Red Hatter, who's also part of the
    MongoDB Master team and has contributed to Open
    Source since... forever.
    One of those who thinks about programming when
    he's not programming.
    member of

    View Slide

  4. @flaper87
    Marconi,
    2 mins intro
    Queues as a Service

    View Slide

  5. @flaper87
    Marconi,
    2 mins intro
    Notifications as a Service
    Queues as a Service

    View Slide

  6. @flaper87
    Marconi,
    2 mins intro
    All that, using lego bricks

    View Slide

  7. @flaper87
    M
    ongoDB

    View Slide

  8. @flaper87
    Name Field
    -------------------
    scope -> p_q
    expires -> e
    ttl -> t
    uuid -> u
    claim -> c
    marker -> k
    Message
    Schema

    View Slide

  9. @flaper87
    Name Field
    -------------------
    id -> id
    expires -> e
    Claim
    Schema

    View Slide

  10. @flaper87
    Compound Indexes

    View Slide

  11. @flaper87
    {Eq, Sort, Range}
    Index order matters

    View Slide

  12. @flaper87
    [start, End) Pure evilness

    View Slide

  13. @flaper87
    #Hint: Use hints ;)

    View Slide

  14. @flaper87
    Avoid counts!

    View Slide

  15. @flaper87
    Aggregate

    View Slide

  16. @flaper87
    Race
    Fighters
    No `limit` for updates...

    View Slide

  17. @flaper87
    Race
    Fighters
    FIFO
    … when ObjectID is not enough

    View Slide

  18. @flaper87
    Race
    Fighters
    Use TTL
    Collections
    tweak expiration field

    View Slide

  19. @flaper87
    Race
    Fighters
    Handling connection errors...

    View Slide

  20. @flaper87
    Race
    Fighters
    Parsing
    Errors

    View Slide

  21. @flaper87
    scale
    Multiple databases
    – q+p hash distribution

    View Slide

  22. @flaper87
    scale
    Marconi Partitions
    – Multiple MongoDB nodes

    View Slide

  23. @flaper87
    hacks
    # NOTE(flaper87): Dirty hack!
    # This sets the expiration time to
    # `expires` on messages that would
    # expire before claim.
    Transaction-less

    View Slide

  24. @flaper87
    Ideas
    1 collection per project
    ● Smaller indexes
    ● Fewer lookups

    View Slide

  25. @flaper87
    Takeaways
    Compound indexes are good
    Range queries are evil
    Transaction-less trade-off
    Database lock can be avoided
    OIDs are awesome but not fully reliable

    View Slide

  26. @flaper87
    Q&A
    Thanks
    we're
    hiring
    http:/
    /openstack.redhat.com/
    http:/
    /www.openstack.org/

    View Slide