OrientDB 

OrientDB ● Support document and graph natively ● Used in the following DSP services: ○ Mapping service ○ Albums service ○ Snafus service ○ Revenue service ● Supposed to have replication but unreliable in the version we use ○ Version 2.0.0 for Albums (built specifically for Tango, and later in data app) ○ Version 1.7.7 for everything else ● Low maintenance 

OrientDB concepts Relational database Document Model Graph Model table Class and Cluster row Document Vertex column Key/Values Vertex, Edge properties relationship Links Edge Pet Owner dog cat NY NJ {“gender: (mandatory), “Pet_id” (mandatory, unique) “Name”: , “DOB”: (mandatory)} dog #1 cat# 15 John Queen 

Monitoring & Backup We monitor these: ● Port 80 (Studio) ● Port 2424 (binary connection) ● Port 2480 (HTTP connection) ● S3 Backup ● Log file ● Disk, Host, Memory Backup is created every 15 minutes (based on LVM snapshot), uploads to S3, and is downloaded to standby every 5 minutes. 

Hazeclast 

Hazelcast ● Distributed, Peer-to-Peer key-value NoSQL database ● Often used as distributed cache (esp due to support of locks and distributed id generator) ● Supports queue and topic so you can publish and subscribe, and even register events ● Used by most DSP applications and services to store session data ● Low maintenance 

Locking ● Cassandra data is indexed in ES for better search support (otherwise we’d build many secondary indices in Cassandra) ● To avoid concurrent reindexing, one of the many Albums instances acquires a lock, performs the reindex, and cache index result in the cache. 

Management console ● Provides overview of cluster statistics and overview ● Use “Console” feature to read keys ○ ns ○ m.keys (list keys of a Map type) ○ m.key (read the value of a Key) 

RabbitMQ 

“RabbitMQ is a message broker; it accepts and forwards messages like a post office.” 

No content

RabbitMQ concepts Written in Erlang, RabbitMQ supports multiple protocols. ● Advanced Message Queuing Protocol (AMQP) - default, most popular standard ● Simple/Streaming Text Oriented Message Protocol (STOMP) - text-based ● Message Queue Telemetry Transport (MQTT) - compacted, pub/sub protocol without actually having a queue. https://www.rabbitmq.com/protocols.html 

RabbitMQ concepts: Producer connects to RabbitMQ and sends messages to a queue Consumer connects to RabbitMQ and fetch message from a queue. Like a named mailbox, holds messages. Note: You can have multiple procducers and consumers. It makes sense right? 

RabbitMQ concepts ● By default messages must be acknolwedged by the consumer, otherwise, RabbitMQ will re-queue the message. ● By default messages in a queue are delivered to each connected consumer in round-robin manner. One consumer gets one message at a time. 

RabbitMQ channels Application connects to RabbitMQ over a TCP connection, and every action we do thereafter is communicated over a channel. 

Exchange and queues 

RabbitMQ concept ● Exchange is like your post office. ● Exchange receives messages from producers. ● Based on routing algorithm (topic, fanout, or direct), route messages to one or more queues matching the routing policy (or none if no match). 

Direct exchange 

Fanout exchange: ● Message sent to a fanout exchange is delivered to all queues attached to the exchange. ● Use case: A new EC2 instance is discovered, the message contains some information about this instance (id, IP address, SSH key name, etc). Action: 1. Log this discovery, 2. Analyze audit log 3. Adds to our dynamic monitoring system 4. Run some assurance tests on this instance. 

Topic exchange ● Almost like direct exchange, except useful if you want the same message delivered to multiple queues based on a pattern-matching routing key. ● If message is sent with routing key “log.critical”, the message is delivered to both QA and Q2. 

RabbitMQ topology 

RabbitMQ gotcha Queues and messages are local to a RabbitMQ node, even in cluster setup. a. Node is down for maintenance, queues and messages won’t be available b. Node is terminated, queues and messages are lost To remediate this issue, use mirrored queue. The downside is performance hit. Messaging queuing and dequeuing happens with the “master” node where the queue was first declared and created. 

RabbitMQ clustering gotcha ● Joining a cluster requires cookie to be same (/var/lib/rabbitmq/.erlang.cookie) as the first node ● Joining a cluster means the new candidate will be reset (config and data). ● Erlang version must be homogeneous otherwise node cannot join the cluster. ● RabbitMQ minor version difference is OKAY (3.6.0 and 3.6.1, NOT 3.6.0 and 3.7.0) 

Join Rabbit cluster To join, do this on the second node (or the next candidate) $ sudo rabbitmqctl stop_app Stopping node rabbit@demo-2 ... ...done. $ sudo rabbitmqctl join_cluster rabbit@demo-1 Clustering node rabbit@rmq-prod-2 with rabbit@demo-1 ... ...done. $ sudo rabbitmqctl start_app Starting node rabbit@demo-2 ... ...done. $ sudo rabbitmqctl cluster_status Cluster status of node rabbit@demo-1 ... [{nodes,[{disc,[rabbit@demo-2,rabbit@demo-1]}]}, {running_nodes,[rabbit@demo-2,rabbit@demo-1]}, {partitions,[]}] ...done.