A Tale of Queues — from ActiveMQ over Hazelcast to Disque

After all the attention databases have been getting over the last years, it is high time to give some thought to queues. We will kick off with some considerations why you need queues in distributed systems and what their limitations are — in particular the at-least-once and at-most-once decision. Next we discuss our specific use case and why
* we started off with ActiveMQ,
* it's working ok for us,
* we are looking for a better solution.

While looking for a better solution, we considered Amazon SQS and RabbitMQ, but finally selected Hazelcast — which seemed to do everything for us. After the initial phase of enchantment, we came to realize that Hazelcast is actually not the right tool for us and why we do not want to fully rely on it. Luckily, Disque has just been released and looks really promising. And we have already started migrating to it, even though it's currently marked as beta code.

Transcript

1. A tale of queues from ActiveMQ over Hazelcast to Disque Philipp Krenn̴̴̴̴̴̴̴̴̴̴@xeraa
5. Infrastructure | Developer Advocate
6. ViennaDB Papers We Love Vienna
7. Electronic Data Interchange
9. Automotive Retail
10. Transmission & Message Flow AS2, X.400, OFTP2, FTP, SMTP, IMAP, web services,...
11. Document format EDIFACT, X12, ODETTE, VDA,...
17. Queues Decoupling producer and consumer
18. !
19. !
20. Networked message queues like ActiveMQ, RabbitMQ, ZeroMQ, and a host of other Java inspired software tumors are crutches of systems design. — Ted Dziuba, http://widgetsandshit.com/teddziuba/2011/02/the-case- against-queues.html
21. "The Blocking Consumer"
22. "Collecting Data for Offline Processing"
23. "Everybody Loves System Complexity"
24. There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery — Mathias Verraes, https://twitter.com/mathiasverraes/status/632260618599403520
25. Order As if it was on a single machine http://book.mixu.net/distsys/time.html
26. http://blog.acolyer.org/2016/02/26/distributed-consistency-and- session-anomalies/
27. Exactly once
28. Removal / ACK Before processing: at-most-once A!er processing: at-least-once
29. At-most-once Requires strict consistency with communication overhead Network split — majority
30. At-least-once Easier to distribute Idempotent consumption Timeout tuning
31. Queues and databases Salvatore Sanfilippo 870 days ago, http://antirez.com/news/78
32. Cheating Order
33. Cheating 1. Receive 2a. Persist in RDS (MySQL) 2b. Camel routing + queue 3. Acknowledge
34. Goals Reliability Liveness At-most-once
35. Machines Wait Retry
36. Amazon Simple Queue Service (SQS) Fast, reliable, scalable, and fully managed
38. But At-least-once 64KB (now 256KB and 2GB on S3) payload
40. Master Slave Shared file system JDBC Replicated LevelDB + ZooKeeper
41. <bean id="mysql-ds" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="jdbcUrl" value="jdbc:mysql://{{rds.hostname}}:3306/{{rds.activemq.database}}"/> <property name="username" value="{{rds.activemq.user}}"/> <property name="password" value="{{rds.activemq.pass}}"/> <property name="validationTimeout" value="5000"/> <property name="minimumIdle" value="2"/> <property name="maximumPoolSize" value="20"/> <property name="idleTimeout" value="600000"/> <property name="maxLifetime" value="1800000"/> <property name="initializationFailFast" value="false"/> <property name="allowPoolSuspension" value="false"/> <property name="dataSourceProperties"> <props> <prop key="relaxAutoCommit">true</prop> <prop key="socketTimeout">10000</prop> <prop key="connectTimeout">10000</prop> <prop key="useJDBCCompliantTimezoneShift">true</prop> </props> </property> </bean>
43. RDS down !
46. In-memory data grid Set, list, map, queue, topic, lock, atomic long,... Query, aggregate, MapReduce Hibernate 2nd level cache, session replication
47. Eierlegende Wollmilchsau
48. "egg-laying wool- milk-sow"
49. Great Does everything for Java apps Multi-master HA Fast
50. Bad feeling Very few queue users Bugs with IP / DNS changes Painful upgrades
51. Currently our action is to create bug for these scenarios and try to ensure exactly-once. — Enes Akar, https://groups.google.com/forum/#!msg/hazelcast/ u_KLHVnvT_U/Qx5Km8COk_oJ
52. Kafka Real-time focus At-least-once https://cwiki.apache.org/confluence/display/KAFKA/ Idempotent+Producer
53. Kestrel Dead !
54. RabbitMQ Too much middleware
55. In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. — https://www.rabbitmq.com/reliability.html
57. [...] no one try to use N Redis independent nodes and the offered primitives as a building block for a distributed system [...] — Salvatore Sanfilippo, http://antirez.com/news/78
58. Disque, an in-memory, distributed job queue — https://github.com/antirez/disque
59. Great Does one thing Multi-master HA Configurable ACK Similar to Redis
60. Driver: Spinach "A scalable Java Disque client" based on Lettuce, "A scalable Java Redis client"
61. Mark Paluch https://github.com/mp911de
62. !
63. Current configuration Per message Replication factor: 1 Retry time: 0 No persistence
64. WARNING: This is beta code and may not be suitable for production usage. — https://github.com/antirez/disque
65. Disque 1.0 RC1 is out! — Salvatore Sanfilippo 333 days ago, http://antirez.com/news/100
68. Back to Redis?
69. Conclusion
75. Options Disque (Redis) ActiveMQ Kafka RabbitMQ Hazelcast SQS
76. Thank you! Questions?! @xeraa
77. Image Credit · Schnitzel https://flic.kr/p/9m27wm · Architektur https://flic.kr/p/6dwCAe · Conchita https://flic.kr/p/nBqSHT · Papier https://flic.kr/p/83thLf · Lu!brücke https://commons.wikimedia.org/wiki/File:C-54landingattemplehof.jpg · Bier https://flic.kr/p/aBSmtY · Gum https://flic.kr/p/ALQ3b · Eierlegende Wollmilchsau https://flic.kr/p/GzQTT · Li! https://flic.kr/p/iBNr9x