Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Spray Some NSQ On It

Spray Some NSQ On It

GopherCon 2014

Matt Reiferson

April 24, 2014
Tweet

More Decks by Matt Reiferson

Other Decks in Programming

Transcript

  1. !
    NSQ
    a realtime distributed messaging platform

    https://github.com/bitly/nsq

    @imsnakes
    April 24th 2014 - GopherCon
    Matt Reiferson - CTO at Torando Labs

    View full-size slide

  2. HI, I’M MATT

    View full-size slide

  3. HI, I’M MATT

    View full-size slide

  4. WHAT EVEN IS NSQ?
    NOT STABLE QUEUE
    NAMING SUCKS QUEUE

    View full-size slide

  5. NEW SIMPLE QUEUE

    View full-size slide

  6. SPRAY SOME NSQ ON IT

    View full-size slide

  7. BIG THINGS HAVE SMALL BEGINNINGS
    •boss says: “I want metrics!”

    •single host

    •synchronous writes
    API
    interweb
    1 2

    View full-size slide

  8. BIG THINGS HAVE SMALL BEGINNINGS
    •boss says: “I want metrics!”

    •single host

    •synchronous writes
    API metrics
    interweb
    1
    2
    3

    View full-size slide

  9. …AND THEN THEY EAT YOU

    View full-size slide

  10. nightmare
    …AND THEN THEY EAT YOU

    View full-size slide

  11. BUT THERE’S HOPE

    View full-size slide

  12. Distributed Message Bus
    THE GOAL
    provide a unifying distributed system to receive

    and disseminate event data

    View full-size slide

  13. MESSAGING PATTERNS

    View full-size slide

  14. PS
    Producer
    ConsumerA
    ConsumerB
    BROADCAST
    de-couple

    View full-size slide

  15. PS
    Producer
    ConsumerA
    ConsumerB
    BROADCAST
    de-couple
    m1
    m1

    View full-size slide

  16. PS
    Producer
    ConsumerA
    ConsumerB
    BROADCAST
    de-couple
    m1
    m1
    m2
    m2

    View full-size slide

  17. PS
    Producer
    ConsumerA
    ConsumerB
    BROADCAST
    de-couple
    m1
    m1
    m2
    m2

    View full-size slide

  18. PS
    Producer
    ConsumerA
    ConsumerB
    BROADCAST
    de-couple
    m2
    m2

    View full-size slide

  19. Q
    Producer
    ConsumerA
    ConsumerA
    horizontal scalability
    DISTRIBUTION

    View full-size slide

  20. Q
    Producer
    ConsumerA
    ConsumerA
    m1
    horizontal scalability
    DISTRIBUTION

    View full-size slide

  21. Q
    Producer
    ConsumerA
    ConsumerA
    m1
    horizontal scalability
    DISTRIBUTION
    m2

    View full-size slide

  22. Q
    Producer
    ConsumerA
    ConsumerA
    m1
    horizontal scalability
    DISTRIBUTION
    m2
    m3

    View full-size slide

  23. Q
    Producer
    ConsumerA
    ConsumerA
    horizontal scalability
    DISTRIBUTION
    m2
    m3

    View full-size slide

  24. Q
    Producer
    ConsumerA
    ConsumerA
    horizontal scalability
    DISTRIBUTION
    m2
    m3

    View full-size slide

  25. Q
    Producer
    ConsumerA
    ConsumerA
    fault tolerance
    FAILURE

    View full-size slide

  26. Q
    Producer
    ConsumerA
    ConsumerA
    fault tolerance
    FAILURE
    m1

    View full-size slide

  27. Q
    Producer
    ConsumerA
    ConsumerA
    fault tolerance
    FAILURE
    m2 m1

    View full-size slide

  28. Q
    Producer
    ConsumerA
    ConsumerA
    fault tolerance
    FAILURE
    m2
    m1

    View full-size slide

  29. Q
    Producer
    ConsumerA
    fault tolerance
    FAILURE

    View full-size slide

  30. Q
    Producer
    ConsumerA
    fault tolerance
    FAILURE
    m3

    View full-size slide

  31. Q
    Producer
    ConsumerA
    fault tolerance
    FAILURE
    m3

    View full-size slide

  32. Q
    Producer
    ConsumerA
    fault tolerance
    FAILURE
    m3

    View full-size slide

  33. Q
    Producer
    ConsumerA
    fault tolerance
    FAILURE

    View full-size slide

  34. Q
    Producer
    ConsumerA
    ConsumerA
    EVEN MORE FAILURE
    welp

    View full-size slide

  35. Q
    Producer
    EVEN MORE FAILURE
    welp
    m1

    View full-size slide

  36. Q
    Producer
    EVEN MORE FAILURE
    welp
    m2 m1

    View full-size slide

  37. Q
    Producer
    EVEN MORE FAILURE
    welp
    m2
    m3
    m1

    View full-size slide

  38. Q
    Producer
    EVEN MORE FAILURE
    welp
    m2
    m3
    m1

    View full-size slide

  39. Q
    Producer
    EVEN MORE FAILURE
    welp
    m2
    m3
    m1

    View full-size slide

  40. Q
    Producer
    ConsumerA
    EVEN MORE FAILURE
    welp
    m2
    m3
    m1

    View full-size slide

  41. Q
    Producer
    ConsumerA
    EVEN MORE FAILURE
    welp
    m2
    m3
    m1

    View full-size slide

  42. Q
    Producer
    ConsumerA
    EVEN MORE FAILURE
    welp
    m2
    m3

    View full-size slide

  43. Q
    Producer
    ConsumerA
    EVEN MORE FAILURE
    welp
    m3

    View full-size slide

  44. Q
    Producer
    ConsumerA
    EVEN MORE FAILURE
    welp

    View full-size slide

  45. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “clicks”
    Topics

    View full-size slide

  46. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics

    View full-size slide

  47. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”

    View full-size slide

  48. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”

    View full-size slide

  49. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers

    View full-size slide

  50. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers

    View full-size slide

  51. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers

    View full-size slide

  52. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A

    View full-size slide

  53. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A

    View full-size slide

  54. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A

    View full-size slide

  55. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A

    View full-size slide

  56. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A
    B
    B
    B

    View full-size slide

  57. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A
    B
    B
    B

    View full-size slide

  58. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A
    B
    B
    B

    View full-size slide

  59. TOPICS AND CHANNELS
    •a topic is a distinct stream of
    messages

    •a topic has one or more channels

    •topics and channels are created at
    runtime

    •messages are pushed to consumers
    combine pubsub, distribution, and queueing
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A
    B
    B
    B

    View full-size slide

  60. UNDER THE HOOD
    •topics and channels are independent

    •configurable high water mark

    (disk persistence)

    •“bounded” memory footprint
    DiskQueue
    In-Memory
    goroutine
    Input
    Chan
    Output
    Chan

    View full-size slide

  61. THERE AND BACK AGAIN

    View full-size slide

  62. DE-COUPLE
    •PUB locally to nsqd via HTTP

    •perform work async

    •co-locate everything (silo)
    API
    nsqd “metrics”

    channel
    “api_event”

    topic
    metrics
    consumer
    consumer

    View full-size slide

  63. SCALE HORIZONTALLY
    API
    nsqd “metrics”

    channel
    API
    nsqd
    metrics
    consumer
    consumer
    consumer
    consumer
    “api_event”

    topic
    “api_event”

    topic
    “metrics”

    channel
    … …

    View full-size slide

  64. A MESSAGE QUEUE

    IS BORING

    (IN ISOLATION)

    View full-size slide

  65. NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    TYPICAL NSQ CLUSTER
    •enable distributed and
    decentralized topologies

    •no centralized broker

    •nsqlookupd instances
    are independent (no
    coordinatation)

    View full-size slide

  66. NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    PUBLISH
    TYPICAL NSQ CLUSTER
    •enable distributed and
    decentralized topologies

    •no centralized broker

    •nsqlookupd instances
    are independent (no
    coordinatation)

    View full-size slide

  67. NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    PUBLISH
    REGISTER
    TYPICAL NSQ CLUSTER
    •enable distributed and
    decentralized topologies

    •no centralized broker

    •nsqlookupd instances
    are independent (no
    coordinatation)

    View full-size slide

  68. NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    PUBLISH
    REGISTER
    DISCOVER
    TYPICAL NSQ CLUSTER
    •enable distributed and
    decentralized topologies

    •no centralized broker

    •nsqlookupd instances
    are independent (no
    coordinatation)

    View full-size slide

  69. NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    PUBLISH
    REGISTER
    DISCOVER
    SUBSCRIBE
    TYPICAL NSQ CLUSTER
    •enable distributed and
    decentralized topologies

    •no centralized broker

    •nsqlookupd instances
    are independent (no
    coordinatation)

    View full-size slide

  70. … AND AGAIN

    View full-size slide

  71. DISCOVERABILITY
    •introduce nsqlookupd

    •producers and consumers

    come and go

    •other services can discover
    and subscribe to this topic
    API
    nsqd
    “metrics”

    channel
    “api_event”

    topic
    API
    nsqd “metrics”

    channel
    “api_event”

    topic

    View full-size slide

  72. DISCOVERABILITY
    •introduce nsqlookupd

    •producers and consumers

    come and go

    •other services can discover
    and subscribe to this topic
    API
    nsqd
    “metrics”

    channel
    “api_event”

    topic
    API
    nsqd “metrics”

    channel
    “api_event”

    topic
    nsqlookupd
    TCP
    nsqlookupd

    TCP

    View full-size slide

  73. DISCOVERABILITY
    •introduce nsqlookupd

    •producers and consumers

    come and go

    •other services can discover
    and subscribe to this topic
    API
    nsqd
    “metrics”

    channel
    “api_event”

    topic
    API
    nsqd “metrics”

    channel
    “api_event”

    topic
    nsq_to_file nsq_to_file
    nsqlookupd
    TCP
    nsqlookupd

    TCP
    “archive”

    channel

    View full-size slide

  74. DISCOVERABILITY
    •introduce nsqlookupd

    •producers and consumers

    come and go

    •other services can discover
    and subscribe to this topic
    API
    nsqd
    “metrics”

    channel
    “api_event”

    topic
    API
    nsqd “metrics”

    channel
    “api_event”

    topic
    nsq_to_file nsq_to_file
    nsqlookupd
    TCP
    nsqlookupd

    TCP
    “archive”

    channel
    1 - query nsqlookupd!
    for topic “api_event”
    HTTP

    View full-size slide

  75. DISCOVERABILITY
    •introduce nsqlookupd

    •producers and consumers

    come and go

    •other services can discover
    and subscribe to this topic
    API
    nsqd
    “metrics”

    channel
    “api_event”

    topic
    API
    nsqd “metrics”

    channel
    “api_event”

    topic
    2 - connect to all discovered

    producers
    nsq_to_file nsq_to_file
    nsqlookupd
    TCP
    nsqlookupd

    TCP
    “archive”

    channel
    1 - query nsqlookupd!
    for topic “api_event”
    HTTP

    View full-size slide

  76. TO INFINITY AND BEYOND
    nsqlookupd
    nsqd
    Datacenter A

    View full-size slide

  77. TO INFINITY AND BEYOND
    nsqlookupd
    nsqd
    Datacenter A
    nsq_to_nsq

    View full-size slide

  78. TO INFINITY AND BEYOND
    nsqlookupd nsqlookupd
    nsqd
    nsqd
    Datacenter A Datacenter B
    nsq_to_nsq

    View full-size slide

  79. TO INFINITY AND BEYOND
    nsqlookupd nsqlookupd
    nsqd
    nsqd
    Datacenter A Datacenter B
    nsq_to_nsq

    View full-size slide

  80. TO INFINITY AND BEYOND
    nsqlookupd nsqlookupd
    nsqd
    nsqd
    Datacenter A Datacenter B
    nsq_to_nsq

    View full-size slide

  81. GUARANTEES
    •messages are delivered at least once

    •messages are not durable (by default)

    •messages received are un-ordered

    •consumers eventually find all topic producers

    View full-size slide

  82. BACK PRESSSSSSSSSSSSURE
    •clients control flow - RDY state

    •concurrency knob

    •processing fails?

    •slow down rate

    •bad message?

    •limit attempts

    •delay re-processing
    Incoming Msgs
    Msg
    Handler
    Timeout - -
    Check
    Timeout
    Full Throttle
    success
    Timeout == 0
    RDY N

    View full-size slide

  83. BACK PRESSSSSSSSSSSSURE
    •clients control flow - RDY state

    •concurrency knob

    •processing fails?

    •slow down rate

    •bad message?

    •limit attempts

    •delay re-processing
    Incoming Msgs
    Msg
    Handler Timeout ++
    Timeout - -
    Check
    Timeout
    Full Throttle
    success
    fail
    Timeout == 0
    RDY N

    View full-size slide

  84. BACK PRESSSSSSSSSSSSURE
    •clients control flow - RDY state

    •concurrency knob

    •processing fails?

    •slow down rate

    •bad message?

    •limit attempts

    •delay re-processing
    Incoming Msgs
    Msg
    Handler Pause
    Timeout ++
    Timeout - -
    Check
    Timeout
    Full Throttle
    success
    fail
    Timeout == 0
    RDY N
    RDY 0

    View full-size slide

  85. BACK PRESSSSSSSSSSSSURE
    •clients control flow - RDY state

    •concurrency knob

    •processing fails?

    •slow down rate

    •bad message?

    •limit attempts

    •delay re-processing
    Incoming Msgs
    Msg
    Handler
    Backoff Expired
    Pause
    Timeout ++
    Timeout - -
    Check
    Timeout
    Full Throttle
    success
    fail
    Timeout == 0
    RDY N
    RDY 0

    View full-size slide

  86. BACK PRESSSSSSSSSSSSURE
    •clients control flow - RDY state

    •concurrency knob

    •processing fails?

    •slow down rate

    •bad message?

    •limit attempts

    •delay re-processing
    Incoming Msgs
    Msg
    Handler
    Test Water Backoff Expired
    Pause
    Timeout ++
    Timeout - -
    Check
    Timeout
    Full Throttle
    success
    fail
    Timeout == 0
    RDY N
    RDY 0
    RDY 1

    View full-size slide

  87. BACK PRESSSSSSSSSSSSURE
    •clients control flow - RDY state

    •concurrency knob

    •processing fails?

    •slow down rate

    •bad message?

    •limit attempts

    •delay re-processing
    Incoming Msgs
    Msg
    Handler
    Test Water Backoff Expired
    Pause
    Timeout ++
    Timeout - -
    Check
    Timeout
    Full Throttle
    success
    fail
    Timeout > 0
    Timeout == 0
    RDY N
    RDY 0
    RDY 1

    View full-size slide

  88. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    “achrive”
    C

    A

    C

    H

    E
    DB

    View full-size slide

  89. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    “achrive”
    C

    A

    C

    H

    E
    DB
    A
    A
    A

    View full-size slide

  90. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    “achrive”
    C

    A

    C

    H

    E
    DB
    A
    A
    A
    B
    B
    B

    View full-size slide

  91. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    “achrive”
    C

    A

    C

    H

    E
    DB
    A
    A
    A
    B
    B
    B

    View full-size slide

  92. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    “achrive”
    C

    A

    C

    H

    E
    DB
    B
    B
    B

    View full-size slide

  93. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    “achrive”
    C

    A

    C

    H

    E
    DB

    View full-size slide

  94. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    “achrive”
    paused
    C

    A

    C

    H

    E
    DB
    C D
    C
    C D
    D

    View full-size slide

  95. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    paused
    C

    A

    C

    H

    E
    DB
    “archive”
    C D
    C
    C D
    D

    View full-size slide

  96. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    C

    A

    C

    H

    E
    DB
    “archive”
    C D
    C
    C D
    D

    View full-size slide

  97. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    C

    A

    C

    H

    E
    DB
    “archive”
    C
    D
    C
    C
    D
    D

    View full-size slide

  98. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    C

    A

    C

    H

    E
    DB
    “archive”
    C
    D
    C
    C
    D
    D

    View full-size slide

  99. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    C

    A

    C

    H

    E
    DB
    “archive”
    D
    D
    D

    View full-size slide

  100. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    C

    A

    C

    H

    E
    DB
    “archive”

    View full-size slide

  101. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    paused
    F
    F
    F
    C

    A

    C

    H

    E
    DB
    “archive”

    View full-size slide

  102. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    paused
    F
    F
    F
    C

    A

    C

    H

    E
    DB
    “archive”
    G
    G
    G

    View full-size slide

  103. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    paused
    F
    F
    F
    C

    A

    C

    H

    E
    DB
    “archive”
    G
    G
    G

    View full-size slide

  104. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    paused
    F
    C

    A

    C

    H

    E
    DB
    “archive”
    G
    G
    G

    View full-size slide

  105. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    paused
    F
    C

    A

    C

    H

    E
    DB
    “archive”
    G

    View full-size slide

  106. TOPIC & CHANNEL PAUSING
    “clicks”
    “metrics”
    “reporting”
    C

    A

    C

    H

    E
    DB
    “archive”

    View full-size slide

  107. ONE MORE THING
    •#ephemeral channels

    •channel sampling

    •TLS / Snappy

    •telemetry over statsd / HTTP

    View full-size slide

  108. SUMMARY
    •github.com/bitly/nsq

    •10,300 lines of Go

    •19 client libraries, 11 languages

    •over 2 years in production

    View full-size slide

  109. SUMMARY
    •github.com/bitly/nsq

    •10,300 lines of Go

    •19 client libraries, 11 languages

    •over 2 years in production

    View full-size slide

  110. IN PRODUCTION
    heavy water

    View full-size slide

  111. Thanks!
    @imsnakes
    !
    https://github.com/bitly/nsq

    !
    shoutout to @jehiah (co-author of NSQ)

    View full-size slide