Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Growing Up With AWS
Search
Michael Hamrah
January 24, 2019
0
38
Growing Up With AWS
Michael Hamrah
January 24, 2019
Tweet
Share
More Decks by Michael Hamrah
See All by Michael Hamrah
A Whirlwind Tour of Infra
mhamrah
0
66
gRPC All The Things
mhamrah
0
370
Sailing With Spinnaker
mhamrah
0
60
Frankenstein's Microservices
mhamrah
1
33
Uber: Anomaly Detection At Scale
mhamrah
0
750
The New Enterprise: RabbitMq, Akka and AWS at Getty Images
mhamrah
5
1.8k
Clean Code
mhamrah
0
330
Things to Know About Web Performance
mhamrah
2
670
Intro to SPDY
mhamrah
0
320
Featured
See All Featured
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
800
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
53k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.8k
Thoughts on Productivity
jonyablonski
69
4.7k
4 Signs Your Business is Dying
shpigford
184
22k
A Tale of Four Properties
chriscoyier
160
23k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
124
52k
We Have a Design System, Now What?
morganepeng
53
7.7k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Transcript
Growing Up With AWS Michael Hamrah / Riadh Amari 1
Namely An HR SaaS 550+ Employees 140+ Engineers
3 A Story of Growth
4 AWS Bills Payments
5 Observability Continuous Integration Continuous Delivery/Deployment Runtime (Kubernetes/Docker/App Servers) Server
Infrastructure (AWS) Operations Configuration Management Environment Management Foundational Infrastructure Ease of Development, Testing and Delivery Meeting Production SLOs Security
6 SignalFx, Logz.io, New Relic Jenkins Spinnaker/Octopus Kubernetes + Istio
AWS Spinnaker, kubectl, Kubernetes, ad-hoc Kubernetes/Octopus Terraform Foundational Infrastructure Ease of Development, Testing and Delivery Meeting Production SLOs IAM, RBAC, Networking, VPN, Secrets
There is no right answer, only various degrees of wrong.
We experiment, learn, decide, act, rinse, repeat and improve!
8 Welcome to Namely Infra AWS (Virginia) Production 10.50.0.0/16 Int
10.52.0.0/16 Stage 10.51.0.0/16 VendorX 10.53.0.0/16 Ops 10.54.0.0/16 Portal IT 172.16.0.0/16 An environment is: • An AWS account and permissions • A VPC • Route tables • Everything required to run Namely • The ability to deploy components
None
None
None
12 Environment Basics a Peering Public ELB(s) Internet Gateway Server1
Server2 Jumpboxes Server3 VPC A bunch of RDS Alotta ElasticCache Some Aurora CloudFront S3 Kubernetes 15 workers 3 masters 5 etcd
13 $17,000/mo on NAT Gateways 380 TB
VPC Endpoints
15 $1,500/mo on NAT Gateways 33 TB
16 Kubernetes Cluster Etcd0 Etcd1 Etcd2 Etcd3 Etcd4 Master0 Master1
Master2 Worker0 Worker2 Worker3 Worker1 Worker... Worker15 State is stored here Does most of the k8s work Where stuff runs
17 EKS Cluster Worker0 Worker2 Worker3 Worker1 Worker... Worker15 Where
stuff runs Better networking
18 Latency Improvements with EKS
OOPS, I IOPS’ED Datawarehouse : EC2 + MSSQL Server (Data
across multiple EBS) I/O Concurrency issues => Latency Transition Volume Type from gp2 to io1 with no downtime I IOPS’ED
20 RDS : Multiple DB flavors (MySQL, Postgres, Aurora) Issue:
Replication lag too high (Read Replica =Latency) for the nightly refreshes. Solution: Enabling IOPS (Master + Read Replicas)! I IOPS’ED
DB Encryption Master KMS keys shared across accounts! EBS volumes
encrypted at rest for EC2. RDS Encryption enabled! Encryption of data in Transit Compliance NY State Cybersecurity Requirements for Financial Services
ElastiCache : Redis Heavy usage of Redis: 50+ instances(Millions Keys)!
Transitioning from Redis instances to Redis clusters: Sharding + data partitioning
23 Automation: Terraform
None
25 200+ Load Balancers
Reserved Instances
None
Things we’re curious about • Container evolution (EKS -> Fargate)
• Transit Gateway • Aurora Postgres HA • Lambda • EC2 Optimizations • Better Account Management
Everything is an Investment We want a return. We must
build up on what we’ve done.
None
None
None
None
None