Marconi from MongoDB's POV

This talk explains some of the issues that have been faced during Marconi's development and how they've been overcome.

Transcript

1. @flaper87
   Marconi from MongoDB's POV
   

   View Slide

2. @flaper87
   Steps out of hell:
   Marconi, 2 mins intro
   Stamped with MongoDB
   Welcome to indexes hell
   Fight race conditions
   Other ideas
   Takeaways
   

   View Slide

3. Who's this
   dude?
   @flaper87 (a.k.a, Flavio Percoco):
   Very proud Red Hatter, who's also part of the
   MongoDB Master team and has contributed to Open
   Source since... forever.
   One of those who thinks about programming when
   he's not programming.
   member of
   

   View Slide

4. @flaper87
   Marconi,
   2 mins intro
   Queues as a Service
   

   View Slide

5. @flaper87
   Marconi,
   2 mins intro
   Notifications as a Service
   Queues as a Service
   

   View Slide

6. @flaper87
   Marconi,
   2 mins intro
   All that, using lego bricks
   

   View Slide

7. @flaper87
   M
   ongoDB
   

   View Slide

8. @flaper87
   Name Field
   -------------------
   scope -> p_q
   expires -> e
   ttl -> t
   uuid -> u
   claim -> c
   marker -> k
   Message
   Schema
   

   View Slide

9. @flaper87
   Name Field
   -------------------
   id -> id
   expires -> e
   Claim
   Schema
   

   View Slide

10. @flaper87
    Compound Indexes
    

    View Slide

11. @flaper87
    {Eq, Sort, Range}
    Index order matters
    

    View Slide

12. @flaper87
    [start, End) Pure evilness
    →
    

    View Slide

13. @flaper87
    #Hint: Use hints ;)
    

    View Slide

14. @flaper87
    Avoid counts!
    

    View Slide

15. @flaper87
    Aggregate
    

    View Slide

16. @flaper87
    Race
    Fighters
    No `limit` for updates...
    

    View Slide

17. @flaper87
    Race
    Fighters
    FIFO
    … when ObjectID is not enough
    

    View Slide

18. @flaper87
    Race
    Fighters
    Use TTL
    Collections
    tweak expiration field
    

    View Slide

19. @flaper87
    Race
    Fighters
    Handling connection errors...
    

    View Slide

20. @flaper87
    Race
    Fighters
    Parsing
    Errors
    

    View Slide

21. @flaper87
    scale
    Multiple databases
    – q+p hash distribution
    

    View Slide

22. @flaper87
    scale
    Marconi Partitions
    – Multiple MongoDB nodes
    

    View Slide

23. @flaper87
    hacks
    # NOTE(flaper87): Dirty hack!
    # This sets the expiration time to
    # `expires` on messages that would
    # expire before claim.
    Transaction-less
    

    View Slide

24. @flaper87
    Ideas
    1 collection per project
    ● Smaller indexes
    ● Fewer lookups
    

    View Slide

25. @flaper87
    Takeaways
    Compound indexes are good
    Range queries are evil
    Transaction-less trade-off
    Database lock can be avoided
    OIDs are awesome but not fully reliable
    

    View Slide

26. @flaper87
    Q&A
    Thanks
    we're
    hiring
    http:/
    /openstack.redhat.com/
    http:/
    /www.openstack.org/
    

    View Slide