Slide 1
Slide 1 text
para//el 2016
Erfahrungen mit Queues
von ActiveMQ über Hazelcast zu Disque
Philipp Krenn̴̴̴̴̴̴̴̴̴̴̴@xeraa
Slide 2
Slide 2 text
Wien
Slide 3
Slide 3 text
Wien
Slide 4
Slide 4 text
Wien
Slide 5
Slide 5 text
ViennaDB
Papers We Love Vienna
Slide 6
Slide 6 text
Elastic
Infrastructure | Developer Advocate
Slide 7
Slide 7 text
Electronic Data Interchange EDI
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
Einzelhandel
Automotive
Slide 10
Slide 10 text
Übertragungsprotokolle
AS2, X.400, OFTP, FTP, SMTP, IMAP,
Web Services,...
Dokumentenformate
EDIFACT, X12, ODETTE, VDA,...
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
Architektur
+--------+
| | +----+
| +--------+ | |
| | | | | |
| | +--------+ v +
| | | | | | +------------> Verarbeitung
+-|-|----+ | | +
| | | | |
+-|------+ | |
| | <--------------+
+--------+
Schnittstellen
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
Datenfluss
Camel Komponente / Protokoll-Implementierung
+ ^
| |
| |
v +
Camel Routing
+ ^
| |
| |
v +
Queue
+ ^
| |
| |
v +
Verarbeitung
Slide 16
Slide 16 text
Queues
Entkoppeln Erstellung und
Verarbeitung
Slide 17
Slide 17 text
!
Slide 18
Slide 18 text
!
Slide 19
Slide 19 text
Networked message queues like
ActiveMQ, RabbitMQ, ZeroMQ, and a
host of other Java inspired software
tumors are crutches of systems design.
— Ted Dziuba, http://widgetsandshit.com/teddziuba/2011/02/the-case-
against-queues.html
Slide 20
Slide 20 text
"The Blocking Consumer"
Slide 21
Slide 21 text
"Collecting Data for
Offline Processing"
Slide 22
Slide 22 text
"Everybody Loves System
Complexity"
Slide 23
Slide 23 text
There are only two hard problems in
distributed systems:
2. Exactly-once delivery
1. Guaranteed order of messages
2. Exactly-once delivery
— Mathias Verraes, https://twitter.com/mathiasverraes/status/
632260618599403520
Slide 24
Slide 24 text
Order
Als wäre nur eine Instanz involviert
http://book.mixu.net/distsys/time.html
Slide 25
Slide 25 text
http://blog.acolyer.org/2016/02/26/distributed-consistency-and-
session-anomalies/
Slide 26
Slide 26 text
Exactly-Once
Das gewünschte Verhalten
Slide 27
Slide 27 text
Bestätigung / ACK
Vor der Verarbeitung: At-Most-Once
Nach der Verarbeitung: At-Least-Once
Slide 28
Slide 28 text
At-Most-Once
Strikte Konsistenz, die
Kommunikations-Overhead erzeugt
Netzwerk-Split — Mehrheit
Slide 29
Slide 29 text
At-Least-Once
Leichter zu verteilen
Idempotente Konsumation
Timeouts anpassen
Slide 30
Slide 30 text
Queues and databases
http://antirez.com/news/78
antirez 632 days ago
Slide 31
Slide 31 text
Schummeln
Order
Slide 32
Slide 32 text
Schummeln
1. Empfangen
2a. In RDS speichern (MySQL)
2b. Camel Routing + Queue
3. Acknowledge
Slide 33
Slide 33 text
Ziele
Verfügbarkeit
Liveness
At-Most-Once
Slide 34
Slide 34 text
Maschinen
Warten
Versuchen es später erneut
Slide 35
Slide 35 text
Amazon Simple Queue
Service (SQS)
Schnell, zuverlässig, skalierbar und
von AWS verwaltet
Slide 36
Slide 36 text
No content
Slide 37
Slide 37 text
Aber...
At-Least-Once
64KB (mittlerweile 256KB) Payload
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
Master Slave
Gemeinsames Dateisystem
JDBC
Replizierte LevelDB
+ ZooKeeper
Slide 40
Slide 40 text
true
10000
10000
true
Slide 41
Slide 41 text
RDS down !
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
In-memory data grid
Sets, Listen, Maps, Queues, Topics,
Locks, Atomic Long,...
Query, Aggregate, MapReduce
Hibernate 2nd Level Cache, Session
Replication
Slide 45
Slide 45 text
Eierlegende
Wollmilchsau
Slide 46
Slide 46 text
Gut
Löst alle Probleme für Java Apps
Multi-Master HA
Schnell
Slide 47
Slide 47 text
Weniger gut
Wenige Anwender der Queue-
Komponente
Bugs bei IP / DNS Änderungen
Stop-the-World Upgrades
Slide 48
Slide 48 text
Currently our action is to create
bug for these scenarios and try to
ensure exactly-once.
— Enes Akar, https://groups.google.com/forum/#!msg/hazelcast/
u_KLHVnvT_U/Qx5Km8COk_oJ
Slide 49
Slide 49 text
Kafka
Echtzeit Fokus
At-Least-Once
https://cwiki.apache.org/confluence/display/KAFKA/Idempotent
+Producer
Slide 50
Slide 50 text
Kestrel
Tot !
Slide 51
Slide 51 text
RabbitMQ
Zu viel Middleware
Slide 52
Slide 52 text
In the event of network failure
(or a node crashing), messages
can be duplicated, and consumers
must be prepared to handle them.
— https://www.rabbitmq.com/reliability.html
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
[...] no one try to use N Redis
independent nodes and the
offered primitives as a building
block for a distributed system [...]
— Salvatore Sanfilippo, http://antirez.com/news/78
Slide 55
Slide 55 text
Disque is a distributed message
broker
— https://github.com/antirez/disque
Slide 56
Slide 56 text
Gut
Dokumentation
Fokus auf Queue
Multi-Master HA
Konfigurierbare ACK
Ähnlich Redis
Slide 57
Slide 57 text
Client: Spinach
"A scalable Java Disque client"
basierend auf Lettuce: "A scalable Java
Redis client"
Slide 58
Slide 58 text
Mark Paluch
https://github.com/mp911de
Slide 59
Slide 59 text
!
Kinderkrankheiten
Slide 60
Slide 60 text
Aktuelle Konfiguration
Pro Nachricht
Replikations-Faktor: 1
Retry-Time: 0
Keine Persistenz
Slide 61
Slide 61 text
Aktuelle Verwendung
Verrechnung seit 07/2015
Bald voller Ersatz für ActiveMQ?
Slide 62
Slide 62 text
WARNING: This is beta code and
may not be suitable for
production usage.
— https://github.com/antirez/disque
Slide 63
Slide 63 text
Oder doch Redis?
Slide 64
Slide 64 text
Zusammenfassung
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
Optionen
Disque (Redis)
ActiveMQ
Kafka
RabbitMQ
Hazelcast
SQS
Slide 67
Slide 67 text
PS: Elastic
Slide 68
Slide 68 text
Danke!
Fragen?!
@xeraa
Slide 69
Slide 69 text
Bilder
· Schnitzel https://flic.kr/p/9m27wm
· Architektur https://flic.kr/p/6dwCAe
· Conchita https://flic.kr/p/nBqSHT
· Papier https://flic.kr/p/83thLf
· Lu!brücke https://commons.wikimedia.org/wiki/
File:C-54landingattemplehof.jpg
· Bier https://flic.kr/p/aBSmtY
· Eierlegende Wollmilchsau https://flic.kr/p/GzQTT
· Li! https://flic.kr/p/iBNr9x