Getting started with MemoryDB for Redis

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon MemoryDB for Redis Getting Started Abhishek Gupta Principal Developer Advocate Amazon Web Services abhi_tweeter abhirockzz

Slide 4

Slide 4 text

Slide 5

Slide 5 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Web and mobile User data stores, session management, geospatial indexing, chat and message queues Retail Customer proﬁles, inventory tracking, fulﬁllment Gaming Player data stores, session history, leaderboards Banking and finance User transactions, fraud detection Media and entertainment User data stores, real-time streaming IoT Streaming device data, operational insights Redis use cases Microservices Highly scalable, performant, simple to complex modularity Customer 360 Profiles, history, preferences, contacts, likes/dislikes, risk category

Slide 6

Slide 6 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Redis data structures and APIs Data type Description Example use cases Associated commands String Sequence of bytes Fast data access APPEND,GET,SET,INCR,DECR,GETSET... List* A list of strings Work queue LSET,LLEN,LPUSH,LPOP,LTRIM,RPOP... Set* Nonrepeating, unordered collection of strings Cardinality SADD,SCARD,SDIFF,SUNION,SINTER,SMEMBERS... Sorted set* Nonrepeating, ordered collection of strings Leaderboards ZADD,ZCARD,ZCOUNT,ZRANK,ZSCORE... Hash* Map of key-value pairs Objects HGET,HGETALL,HKEYS,HVALS,HMSET,HMGET... Streams Log data structure Streaming XADD,XRANGE,XREAD,XACK,XCLAIM,XLEN... Geospatial Longitude-/latitude-based entries Maps, "nearby" GEOADD,GEODIST,GEOPOS,GEORADIUS... Bitmaps Special usage of string type GETBIT,BITCOUNT,SETBIT,SETRANGE,GETRANGE... HyperLogLogs Special usage of string type PFADD,PFCOUNT,PFMERGE

Slide 7

Slide 7 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Diﬃcult to scale Online scaling can be error prone, replication performance needs to be monitored during scaling operations, data loss can occur if issues arise. Expensive Invest in people, processes, hardware, and software Hard to make robust Many in-memory databases do not have data durability, which can lead to data loss during outages or failover. Also, need to implement fast error detection and remediation. Diﬃcult to manage Manage server provisioning, software patching, setup, configuration, and backups Challenges with Self Managed In-memory datastores

Slide 8

Slide 8 text

Slide 9

Slide 9 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Durability and high availability Multi-AZ transactional log for durability and replicas for high availability Redis compatibility Flexible and friendly Redis APIs and data structures Fully managed AWS-managed hardware and software setup, configuration, monitoring, and snapshots Ultra-fast performance Lowest latency database offered by AWS Security Amazon VPC, encryption at-rest and in-transit, Access Control Lists (ACLs) Amazon MemoryDB for Redis Redis-compatible, in-memory database with Multi-AZ durability High scalability Up to 500 nodes and 128 TB of in-memory storage per cluster (with 1 replica per shard)

Slide 10

Slide 10 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Core Components Replica Node Primary Node Replica Node Max 5 replicas Shard Replica Node Primary Node Replica Node Max 5 replicas Shard Cluster

Slide 11

Slide 11 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scalability Scale out in minutes Scale up in minutes Scale reads in minutes Scale up to 500 shards Scale from 13 to 419 GiB of RAM per node Scale up to 5 replicas

Slide 12

Slide 12 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availability Zone B Availability Zone C Redis cluster Availability Zone A P R Slots 0–5454 Slots 10910–16363 R Slots 5455–10909 R R Slots 0–5454 Slots 10910–16363 P Slots 5455–10909 R P Slots 0–5454 Slots 10910–16363 R Slots 5455–10909 Shards, primaries, and read replicas Example 3 shards 2 replicas per shard Multi-AZ S S S Multi-AZ High availability architecture • Add shards to scale reads/writes, increase in-memory capacity • Add replicas to scale reads, increase availability

Slide 13

Slide 13 text

© 2022, Amazon Web Services, Inc. or its aﬃliates. All rights reserved. Use open-source cluster-aware Redis client Authenticate with ACLs Use open-source Redis 6.2 API commands U S E E X I S T I N G O P E N - S O U R C E R E D I S C L U S T E R - A W A R E C L I E N T S – P Y T H O N E X A M P L E Getting connected to MemoryDB

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

© 2022, Amazon Web Services, Inc. or its aﬃliates. All rights reserved. Multi-AZ Transaction Log • All writes committed to Multi-AZ transactional log • Guaranteed delivery of writes to replicas Amazon MemoryDB for Redis MemoryDB cluster

Slide 18

Slide 18 text

© 2022, Amazon Web Services, Inc. or its aﬃliates. All rights reserved. Data Durability and HA Redis Client SET A 100 ACK SET A 100 SET A 100 ACK SET A 100 Amazon MemoryDB for Redis Cluster Availability Zone A Primary Availability Zone B Replica Availability Zone C Replica Synchronous Asynchronous Multi-AZ Transaction Log Acknowledgement sent to client only after transaction is written. Data is replicated via the transaction log and consumed by replicas. Guaranteed delivery of writes to replicas

Slide 19

Slide 19 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Strong read consistency from primary GET A(nil) Data Consistency Redis Client SET A 100 ACK SET A 100 SET A 100 ACK SET A 100 Amazon MemoryDB for Redis Cluster Availability Zone A Primary Availability Zone B Replica Availability Zone C Replica Multi-AZ Transaction Log GET A(100) GET A(100) Eventual read consistency from replicas REMEMBER: Data is always in memory, so reads are super fast

Slide 20

Slide 20 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary Handling Failover Redis Client Amazon MemoryDB for Redis Cluster Availability Zone A Primary Availability Zone B Replica Availability Zone C Replica Synchronous Asynchronous Multi-AZ Transaction Log SET A,B,C Roll-up ACK x3 SET A,B,C SET A SET B SET C ACK x3 SET A SET B SET C Replica reads all transactions from the log, then is promoted to primary Replicas continue to consume from log asynchronously Client is now redirected to newly promoted primary for all writes

Slide 21

Slide 21 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary Handling Failover Redis Client Amazon MemoryDB for Redis Cluster Availability Zone A Availability Zone B Availability Zone C Replica Multi-AZ Transaction Log Zero data loss during failover! Replica

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Best practices • High available (HA) setup § Multi-AZ – typically across 3 AZs § 3 shards and 1 primary 2 replicas (3 nodes) per shard • Redis Cluster? Pick the right client, configure for read/write • Connection pool, Pipeline • Timeouts – client and server-side • Monitoring

Slide 29

Slide 29 text

Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Abhishek Gupta Principal Developer Advocate Amazon Web Services abhi_tweeter abhirockzz https://eventbox.dev/survey/IKIEG9U