Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Tale of Queues — from ActiveMQ over Hazelcast to Disque

Philipp Krenn
December 01, 2016

A Tale of Queues — from ActiveMQ over Hazelcast to Disque

After all the attention databases have been getting over the last years, it is high time to give some thought to queues. We will kick off with some considerations why you need queues in distributed systems and what their limitations are — in particular the at-least-once and at-most-once decision. Next we discuss our specific use case and why
* we started off with ActiveMQ,
* it's working ok for us,
* we are looking for a better solution.

While looking for a better solution, we considered Amazon SQS and RabbitMQ, but finally selected Hazelcast — which seemed to do everything for us. After the initial phase of enchantment, we came to realize that Hazelcast is actually not the right tool for us and why we do not want to fully rely on it. Luckily, Disque has just been released and looks really promising. And we have already started migrating to it, even though it's currently marked as beta code.

Philipp Krenn

December 01, 2016
Tweet

More Decks by Philipp Krenn

Other Decks in Programming

Transcript

  1. A tale of queues from ActiveMQ over Hazelcast to Disque

    Philipp Krenn̴̴̴̴̴̴̴̴̴̴@xeraa
  2. !

  3. !

  4. Networked message queues like ActiveMQ, RabbitMQ, ZeroMQ, and a host

    of other Java inspired software tumors are crutches of systems design. — Ted Dziuba, http://widgetsandshit.com/teddziuba/2011/02/the-case- against-queues.html
  5. There are only two hard problems in distributed systems: 2.

    Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery — Mathias Verraes, https://twitter.com/mathiasverraes/status/632260618599403520
  6. <bean id="mysql-ds" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="jdbcUrl" value="jdbc:mysql://{{rds.hostname}}:3306/{{rds.activemq.database}}"/>

    <property name="username" value="{{rds.activemq.user}}"/> <property name="password" value="{{rds.activemq.pass}}"/> <property name="validationTimeout" value="5000"/> <property name="minimumIdle" value="2"/> <property name="maximumPoolSize" value="20"/> <property name="idleTimeout" value="600000"/> <property name="maxLifetime" value="1800000"/> <property name="initializationFailFast" value="false"/> <property name="allowPoolSuspension" value="false"/> <property name="dataSourceProperties"> <props> <prop key="relaxAutoCommit">true</prop> <prop key="socketTimeout">10000</prop> <prop key="connectTimeout">10000</prop> <prop key="useJDBCCompliantTimezoneShift">true</prop> </props> </property> </bean>
  7. In-memory data grid Set, list, map, queue, topic, lock, atomic

    long,... Query, aggregate, MapReduce Hibernate 2nd level cache, session replication
  8. Currently our action is to create bug for these scenarios

    and try to ensure exactly-once. — Enes Akar, https://groups.google.com/forum/#!msg/hazelcast/ u_KLHVnvT_U/Qx5Km8COk_oJ
  9. In the event of network failure (or a node crashing),

    messages can be duplicated, and consumers must be prepared to handle them. — https://www.rabbitmq.com/reliability.html
  10. [...] no one try to use N Redis independent nodes

    and the offered primitives as a building block for a distributed system [...] — Salvatore Sanfilippo, http://antirez.com/news/78
  11. !

  12. WARNING: This is beta code and may not be suitable

    for production usage. — https://github.com/antirez/disque
  13. Image Credit · Schnitzel https://flic.kr/p/9m27wm · Architektur https://flic.kr/p/6dwCAe · Conchita

    https://flic.kr/p/nBqSHT · Papier https://flic.kr/p/83thLf · Lu!brücke https://commons.wikimedia.org/wiki/File:C-54landingattemplehof.jpg · Bier https://flic.kr/p/aBSmtY · Gum https://flic.kr/p/ALQ3b · Eierlegende Wollmilchsau https://flic.kr/p/GzQTT · Li! https://flic.kr/p/iBNr9x