COMMUNITY DAY MENA How to select the right EBS

Ahmed Shendy Senior DevOps Engineer

COMMUNITY DAY MENA Agenda What is Amazon EBS Amazon EBS Volume Types Choosing an Amazon EBS volume type Performance Best Practices for EBS Saving Cost with EBS

COMMUNITY DAY MENA What is Amazon EBS? o Block storage as service o Create, attach volumes through an API o Service accessed over network o Volumes persist independent of EC2 o Detach and attach between instances within the same availability zone AWS Cloud Availability Zone NAT gateway Instance Instance Amazon EC2 Auto Scaling Amazon EC2 Amazon Elastic Block Store

COMMUNITY DAY MENA Amazon EBS Volume Types

COMMUNITY DAY MENA Amazon EBS Volume Types SSD HDD gp2 General Purpose SSD io1 Provisioned IOPS SSD st1 Throughput Optimized HDD sc1 Cold HDD

COMMUNITY DAY MENA Amazon EBS volume: General Purpose SSD Baseline: 100 to 16,000 IOPS @ 3 IOPS/GB Burst: 3,000 IOPS (< 1000 GB) Throughput: Up to 256 MiB/s Latency: Single-digit ms Capacity: 1 GiB to 16 TiB I/O Size: Up to 256 KiB (logical merge) gp2 General Purpose SSD Great for boot volumes, low-latency applications, and bursty databases

COMMUNITY DAY MENA Burst bucket: gp2 gp2 General Purpose SSD

COMMUNITY DAY MENA Burst and baseline: gp2

COMMUNITY DAY MENA How long can I burst on gp2? 0 0 150 300 450 600 750 Minutes of Burst 200 400 600 800 Volume size in GiB

COMMUNITY DAY MENA Amazon EBS volume: Provisioned IOPS Baseline: 100 – 64,000 IOPS Throughput: Up to 1000 MiB/s Latency: Single-digit ms Capacity: 4GiB – 16 TiB I/O Size: Up to 256 KiB (logical merge) io1 Provisioned IOPS Ideal for critical applications and databases with sustained IOPS

COMMUNITY DAY MENA Scaling Provisioned IOPS SSD : io1 0 0 40,000 20,000 30,000 10,000 50,000 64,000 IOPS 1.28 2 6 9 16

COMMUNITY DAY MENA Amazon EBS volume: Throughput Optimized Baseline: 40 MiB/s per TiB, up to 500 MiB/s Burst: 250 MiB/s per TiB, up to 500 MiB/s Capacity: 500 GiB to 16 TiB I/O Size: Up to 1 MiB (logical merge) st1 Throughput Optimized HDD Ideal for large-block, high-throughput sequential workloads

COMMUNITY DAY MENA Amazon EBS volume: Cold HDD Baseline: 12 MiB/s per TiB, up to 192 MiB/s Burst: 80 MiB/s per TiB, up to 250 MiB/s Capacity: 500 GiB to 16 TiB I/O Size: Up to 1 MiB (logical merge) sc1 Cold HDD Ideal for sequential throughput workloads, such as logging and backup

COMMUNITY DAY MENA Choosing an Amazon EBS volume type

COMMUNITY DAY MENA Understand your mission Databases PostgreSQL, MySQL Cassandra, MongoDB Data & Analytics Kafka, Splunk, Hadoop, Data Warehousing Media Transcoding, Encoding, Render Farms File CIFS, NFS Archive

COMMUNITY DAY MENA Select the right volume • Typically high performance requirements • Mostly random I/O • Journal is sequential • Highly workload dependent Recommendation: SSD Databases PostgreSQL, MySQL Cassandra, MongoDB

COMMUNITY DAY MENA Select the right volume • Typically high throughput requirements • Mostly sequential I/O • Sustained I/O Recommendation: Throughput Optimized HDD (st1) Media Transcoding, Encoding, Render Farms

COMMUNITY DAY MENA Select the right volume • Typically high throughput requirements • Mostly sequential I/O • Daily periodicity Recommendation: HDD Data & Analytics Kafka, Splunk, Hadoop, Data Warehousing

COMMUNITY DAY MENA Select the right volume • Typically low throughput requirements • Bursty workloads • Cost sensitive Recommendation: Cold HDD (sc1) File CIFS, NFS Archive

COMMUNITY DAY MENA How do AWS count I/O When possible, AWS logically merge sequential I/O, to minimize I/O charges on io1, and maximize burst on gp2, sc1, and st1 io1 and gp2 : up to 256 KiB st1 and sc1: up to 1 MiB

COMMUNITY DAY MENA How do AWS count I/O Example 1: Random I/Os: • 4 random I/Os (non-sequential) • Each I/O 64 KiB

COMMUNITY DAY MENA How do AWS count I/O Example 2: Sequential I/Os: • 4 sequential I/Os • Each I/O 64 KiB

COMMUNITY DAY MENA How do AWS count I/O Example 2: Sequential I/Os: • 4 sequential I/Os • Each I/O 64 KiB

COMMUNITY DAY MENA How do AWS count I/O Example 2: Sequential I/Os: • 4 sequential I/Os • Each I/O 64 KiB

COMMUNITY DAY MENA How do AWS count I/O Example 3: Large I/O: • 1 I/O • 1024 KiB

COMMUNITY DAY MENA How do AWS count I/O Example 4: Mixed I/Os: • 2 * 512 KB sequential I/O s • 2 * 64 KB random I/O s • 2 * 128 KB sequential I/O s

COMMUNITY DAY MENA Need more details

COMMUNITY DAY MENA Performance Best Practices for EBS

COMMUNITY DAY MENA EBS Optimized • Dedicated EBS bandwidth • Up to 19 Gbps (2,375 MiB/s) per instance • Burst available for smaller instances Tip: Ensure EBS optimized bandwidth can support your volumes

COMMUNITY DAY MENA EBS Optimized • Dedicated EBS bandwidth • Up to 19 Gbps (2,375 MiB/s) per instance • Burst available for smaller instances Tip: Ensure EBS optimized bandwidth can support your volumes

COMMUNITY DAY MENA Performance: EBS Optimized

COMMUNITY DAY MENA Best practices: RAID When to RAID? - Storage requirements > 16 TiB - Throughput requirement > 1,000 MiB /s - IOPS requirement > 64,000

COMMUNITY DAY MENA Best practices: RAID Avoid RAID for redundancy • EBS data is already replicated • RAID 1 halves available EBS bandwidth • RAID 5 / 6 loses 20 % to 30% of usable I/O to parity

COMMUNITY DAY MENA Monitor I/O characteristics using CloudWatch You can monitor the I/O characteristics with each volume's CloudWatch volume metrics. Important metrics to consider include the following: • BurstBalance • VolumeReadBytes • VolumeWriteBytes • VolumeReadOps • VolumeWriteOps • VolumeQueueLength BurstBalance: displays the burst bucket balance for gp2, st1, and sc1 volumes as a percentage of the remaining balance To determine your volume's average I/O size, divide VolumeWriteBytes by VolumeWriteOps. The same calculation applies to read operations. If your I/O latency is higher than you require, check VolumeQueueLength to make sure your application is not trying to drive more IOPS than you have provisioned.

COMMUNITY DAY MENA Saving Cost with EBS

COMMUNITY DAY MENA EBS Cost gp2 $0.10 per GiB io1 $0.125 per GiB $0.065 per PIOPS st1 $0.045 per GiB sc1 $0.025 per GiB Snapshot storage for all volume types is $0.05 per GiB per month * All prices are per month, prorated to the second, and from the us-west-2 Region as of Nov 2019

COMMUNITY DAY MENA Elastic Volumes: Size your volumes correctly!

COMMUNITY DAY MENA Tag Volumes and Snapshots on Create

COMMUNITY DAY MENA Set DeleteOnTermination = TRUE if storage lifecycle = instance lifecycle DeleteOnTermination = FALSE DeleteOnTermination = TRUE USE WITH CARE – You might delete volumes you need

COMMUNITY DAY MENA Tier to lower cost volume type if volume is not in use

COMMUNITY DAY MENA Don't use io1 unless you are sure that you need it

COMMUNITY DAY MENA Thank You