AWS Deployment Mechanics

In this workshop we are discussing the mechanics of a reliable, scalable, and cost-effective deployment on AWS. This includes how to structure your network (VPC), the use of auto-scaling groups, and how to work with storage like EBS, Glacier, S3, or instance storage. Additionally, we will talk about Elastic Load Balancers and their limitations, how to save money by using reserved or on-demand instances, and how to monitor your servers and applications. While we will show some possible solutions, this should be very interactive — tell us how and why you are doing things differently and we will discuss varying scenarios and experiences!

Transcript

1. Java$Vienna,$JSUG AWS Deployment*Mechanics Philipp&Krenn&@xeraa
2. ViennaDB Papers'We'Love'Vienna
3. Electronic*Data*Interchange!EDI Automated)exchange)of)B2B)documents
4. Outline Basics&&&Overview&&&Demo !!" Hands&On Our$Setup
5. It's%the%future... Everybody's+using+it...
10. Region'+'Availability'Zone eu#central#1 eu#central#1a+and+eu#central#1b AZ#different#per#account
12. $ ec2-describe-availability-zones --region us-east-1 AVAILABILITYZONE us-east-1a available us-east-1 AVAILABILITYZONE us-east-1b available us-east-1 AVAILABILITYZONE us-east-1c available us-east-1 AVAILABILITYZONE us-east-1d available us-east-1 AVAILABILITYZONE us-east-1e available us-east-1
13. SLA:%Up(me More%than%one%AZ Reboots,(failed(instances,...(not(covered h"p:/ /aws.amazon.com/ec2/sla/
14. AWS$Management$Console:$Demo h"ps:/ /console.aws.amazon.com1root1account h"ps:/ /<account>.signin.aws.amazon.com/console7IAM7user
15. EC2 Elas%c'Compute'Cloud
16. Features Xen:%Host%server%+%Hypervisor%+%Guests Linux,'Windows Intel&only,&now&custom&built
17. Every&day,&AWS&adds&enough&new& server&capacity&to&support&all&of& Amazon's&global&infrastructure&when& it&was&a&$7B&enterprise. —"h$p://www.slideshare.net/AmazonWebServices/intro<to<aws< amazon<ec2<and<compute<services
18. Instance(type:(c4.large c:"Instance"family 4:"Instance"genera,on large:"Instance"size
19. Families(&(Genera-ons General'purpose:'M3,'M4 Compute(op)mized:(C3,(C4 Memory'op)mized:'R3 Dense%storage:%D2 I/O$op'mize:$I2
20. Families(&(Genera-ons GPU:%G2 Bursty:(T2 Old$instances:$T1,$M1,$C1,$CC2,$M2,$CR1,$ CG1,$HS1,$HI1 h"ps:/ /aws.amazon.com/ec2/instance3types/
21. Storage Instance(storage((locally(a/ached) Elas%c'Block'Storage'(EBS):'General'purpose' (SSD),'Provisioned'IOPS'(SSD),'Magne%c Elas%c'File'System'(EFS)
22. Demo
23. EBS$Limita*ons Single'instance'usage Data$on$mul*ple$instances$but$single$AZ Main%cause%of%outages%in%the%past
24. EBS$Tip If#it#doesn't#boot,#a/ach#it#to#a#working# instance
25. Volumes Snapshots Amazon'Machine'Image'(AMI)
26. On#Demand:*per*hour Reserved:(no(/(par.al(/(all(upfront,(1( or(3(years,(low(/(medium(/(high( u.liza.on Spot:&bidding
27. Auto%Scaling%Group%(ASG) Start%/%stop%instances%based%on%0me%or%load Very%expensive%mistakes
28. ASG$Limita*ons Not$instant Does%your%database,%cache,...%scale%as%well?
29. ASG$Usage Define%AMI Fetch&current&build&on&boot Also%used%for%single%instances?
30. Elas%c'Load'Balancers'(ELB) Load%distribu,on Detects&available&instances SSL#termina+on
31. ELB$Internals Round&Robin&or&Session&S-ckiness TCP$ports:$25,$80,$443,$and$1024665535
32. ELB$Limita)ons No#sta'c#IP Single'region Single'Cer*ficates'<=2KB No#500#pages
33. ELB$Tips Support'can'configure'.meouts Configure)logs)(S3) Warm%up Prefer%Round%Robin
34. ELB$Tips SSL#configura,on h"ps:/ /wiki.mozilla.org/Security/Server_Side_TLS
36. Elas%c'IPs'(EIP) Quicker(to(switch(than(DNS Firewall(rules
37. EIP$Limita*ons Default(5(per(region Bound&to&a&region&and&EC20classic&/&VPC No#tags Too#easy#to#release
38. Tip m1.large)!=)m1.large)—)genera.ons)ma1er $ grep ‘model name’ /proc/cpuinfo
39. Experience Instances(get(stuck((mostly(bad(hardware) Instances(break Noisy&neighbors
42. VPC Virtual(Private(Cloud
44. Features Custom'route'table'(private'subnet) Internet&Gateway&(public&subnet) NAT$instance$/$bas.on$host$(private$subnet) Network(ACL Subnet,(allow(/(deny,(stateless
45. 10.0.0.0/16 - Production 10.0.0.0/18 — AZ A 10.0.0.0/20 — Private (4094 addresses) 10.0.16.0/20 — Public 10.0.32.0/20 - Spare 10.0.48.0/20 — Spare 10.0.64.0/18 — AZ B 10.0.64.0/20 — Private 10.0.80.0/20 - Public 10.0.96.0/20 — Spare 10.0.112.0/20 - Spare 10.0.128.0/18 — Spare 10.0.192.0/18 - Spare 10.1.0.0/16 - Development 10.2.0.0/16 - Stage
46. Limita&ons VPC$IP$rangs$cannot$be$changed Networks)can)be)/16)to)/28
47. Route53
48. Features DNS Loca%on'awareness Health'checks API$(Boto)
49. Use Instances(update(their(CNAME(on(start #!/bin/bash # Add to /etc/rc.local # Requires Boto DNS_ADDRESS="`ec2metadata | grep 'public-hostname:' | cut -d ' ' -f 2`" /usr/local/bin/route53 change_record <KEY> xxx.ecosio.com. CNAME $DNS_ADDRESS 300
50. ELB$Alterna,ve TCP$+$UDP,$any$port,$direct$resolu7on (Weighted)*Round*Robin Failover)health)checks Cheaper No#SSL#termina-on
51. S3 Simple'Storage'Service
52. Concepts Bucket Object
53. Guarantees 99.999999999%$durability 99.99%$availability/year
54. Features Higly&reliable: Rela%vely(cheap Host%sta'c%sites%(XML%schema) Reduced&Redundancy&S3 Logging
55. Demo
56. Tip It's%not%a%good%(private)%Maven%repo Parent'POM'/'plugin,'access'control,'snapshot'cleanup,...
57. Misc%Services
58. RDS Rela%onal(Database(Service
59. Features MySQL&,(Oracle&,(SQL(Server&,(or(PostgreSQL Replica(on Read%Replicas
60. Limita&ons Some%permissions%not%available No#filesystem#opera.ons#(XtraBackup) Backups(are(totally(opaque
61. Tip Point&in&'me*recovery
62. Tip Failover)via)DNS)—)up)to)5min c3p0 Hard%to%test
63. Aurora
64. Elas%Cache
65. Features memcached'and'Redis 213MB&to&237GB
66. Limita&ons No#authen*ca*on Custom'memcached'client Fork%of%Spymemcached%maintained%by%@daschl
67. SQS Simple'Queue'Service
68. Features Powerful)queue)(DLQ,)batches) Not$JMS
69. Limita&ons At#last#once Messages&<256KB
70. Monitoring
71. CloudWatch
72. Features EC2$(CPU,$network,...) RDS$(replica$lag,$free$disk$space,...) ELB$(>=$1$healthy$instance)
73. Limita&ons No#RAM#on#EC2 Some%mes'bogus'alerts
74. Usage EC2:%CPU%>%70%,%incoming%network%>%5Mbit/ s,%any%failure RDS:%Free%disk,%replica%lag,%connec5ons ELB:%Healthy%instances%>=%1
75. NewRelic
76. Features Middle&'er&free&on&AWS Many%programming%languages%and% infrastructure
77. Limita&ons Web$applica*on$centric Bogus&alerts Plugins(are(useless
78. Usage Pinger Pingdom(for(important(services Disk%space%monitor
79. LogEntries
80. Features Logback(appender Instance(monitoring Analy&cs)of)AWS)logs Alerts
81. Limita&ons No#aggregated#reports#(Papertrail)
82. OpsGenie
83. Features Cheaper'than'PagerDuty Reliable Many%integra+ons
84. Usage CloudWatch:,Any,EC2,instance,failure,on, produc8on CloudWatch:,All,RDS,produc3on,metrics Pingdom(integra-on
85. Security
88. [...]$our$data,$backups,$machine$ configura8ons$and$offsite$backups$ were$either$par8ally$or$completely$ deleted. —"h$p://www.codespaces.com
90. The$person(s)$used$our$account$to$ order$hundreds$of$expensive$ servers,$likely$to$mine$Bitcoin$or$ other$cryptocurrencies. —"h$p://blog.drawquest.com
92. This%outage%was%the%result%of%an% a1ack%on%our%systems%using%a% compromised%API%key. —"h$p://status.bonsai.io/incidents/qt70mqtjbf0s
94. Lock%away%your%root%account%and% never%use%it
95. Always'use'IAM Iden%ty(and(Access(Management
96. Create&an&IAM&user&for&every&service& or&ac3on
97. Use$groups$to$manage$permissions$ for$users
98. Lock%users%and%groups%down%as% much%as%possible
99. { "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets", "s3:ListBucket" ], "Resource": "arn:aws:s3:::*" }, { "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Effect": "Allow", "Resource": "arn:aws:s3:::com.example.backup/*" } ] }
100. Strong'password
101. h"p:/ /xkcd.com/936/
102. 2"Factor"Authen.ca.on"(2FA)
104. Do#not#lose#your#root#account's#2FA# codes#and#backup#ques7ons#—#this# requires#a#notary
105. Never%commit%your%creden/als
106. Enable'IP'restric/ons
107. { "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" }, { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "NotIpAddress": { "aws:SourceIp": ["10.0.0.0/24", "10.10.0.0/24"] } } } ] }
109. Enable'billing'alerts
111. Enable'CloudTrail
112. { "Records": [ { "eventVersion": "1.0", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::123456789012:user/Alice", "accountId": "123456789012", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-09-09T19:01:59Z", "eventSource": "ec2.amazonaws.com", "eventName": "StopInstances", "awsRegion": "eu-west-1", "sourceIPAddress": "205.251.233.176", "userAgent": "ec2-api-tools 1.6.12.2", "requestParameters": { "instancesSet": { "items": [ { "instanceId": "i-ebeaf9e2" } ] }, "force": false }, ... }, ... ] }
113. Check&Your&Security&Status
115. Trusted(Advisor(Security
116. 140$servers$running$on$my$AWS$ account?$What?$How?$I$only$had$S3$ keys$on$my$GitHub$and$they$where$ gone$within$5$minutes! —"h$p://www.devfactor.net/2014/12/30/2375;amazon;mistake/
118. Automate Automate Automate
119. Pets%vs%Ca)le?
120. Puppet&vs&Chef&vs&Ansible&vs&Salt?
121. Ansible(demo
122. Conclusion &"Discussion
125. Main%difference%of%data%center%and% cloud:%in%data%center%you%are%need%2% of%everything,%in%cloud%you%are%need% 20%of%everything. —"@DEVOPS_BORAT, h1ps://twi1er.com/DEVOPS_BORAT/status/274366602252804096
127. #1#Tip Paid%support And$try$to$get$a$contact$person
128. #2#Tip Know%the%service%limits h"p:/ /docs.aws.amazon.com/general/latest/gr/ aws_service_limits.html
129. Try$it$out$—$free$,er!