Healthy cloud for a HealthTech company - AWS Berlin meetup

https://github.com/bsideup/healthy-cloud-demo

When we talk about the Health and related technologies, we usually think about some big mainframes and private datacenters.
And while your health data is indeed important and has the high privacy requirements, it must also be stored reliably and always be available to you. At Vivy, we think the same, this is why we decided to use AWS as our infrastructure provider.
This talk will explain how we scaled from 1 to 20 microservices in less than 6 months by deploying them to Elastic Container Service with Docker, how successfully followed Infrastructure-as-Code approach with CloudFormation and Troposphere and automated the creation of new environments and services.
Ever wondered how to securely store secrets in AWS without Vault? Do service discovery without Consul? Or how AWS Lambda can help you to operate your infrastructure & provisioning? Me too, and now I want to share it with you! :)

Transcript

1. Healthy cloud for a HealthTech company Sergei Egorov @bsideup
2. About me • Head of Backend at Uvita GmbH • OSS enthusiast (TestContainers, Apache Foundation, …) • Before: N26, Zalando, ZeroTurnaround, TransferWise, … • First time tried AWS in 2010-ish, Docker in 2014 @bsideup
4. Digital health companion
5. WHAT HOW WHY What drives us forward
6. WHAT HOW WHY What drives us forward We believe a healthier life is a happier life.
7. WHAT HOW WHY What drives us forward We believe a healthier life is a happier life. Utilizing technology and data, we therefore provide personal health insights and thus empower to understand and improve your well-being.
8. WHAT HOW WHY What drives us forward We believe a healthier life is a happier life. Utilizing technology and data, we therefore provide personal health insights and thus empower to understand and improve your well-being. This guidance we deliver through a personal digital health companion that connects you to a ecosystem of health services.
9. WHAT HOW WHY What drives us forward We believe a healthier life is a happier life. Utilizing technology and data, we therefore provide personal health insights and thus empower to understand and improve your well-being. This guidance we deliver through a personal digital health companion that connects you to a ecosystem of health services.
10. WHY
11. @bsideup Number of micro-services 0 2 4 6 8 10 12 14 16 18 20 August '17 September '17 … January '18 20 3 1
12. @bsideup Expected number of micro-services 0 250 500 750 1000 August '17 September '17 … January '18 January '19 1000 20 3 1
13. @bsideup Sanely expected number of micro-services 0 4 8 12 16 20 24 28 32 36 40 August '17 September '17 … January '18 January '19 40 20 3 1
14. HOW
16. @bsideup
17. @bsideup
18. @bsideup
20. Elastic Container Service
21. No managed Kubernetes on AWS (yet) Well-integrated in the rest of AWS services Shared pool of EC2 instances (cattle) Blue/Green, Rolling and other deployment strategies @bsideup
22. Challenges
23. Service discovery @bsideup Internal AWS ALB AWS VPC Private VPC DNS zone, i.e.
 *.vivy.services
24. Service discovery @bsideup Internal AWS ALB AWS VPC Private VPC DNS zone, i.e.
 *.vivy.services there is cheap .services TLD
25. Service discovery Staging VPC https:/ /auth.vivy.services Staging internal ALB Routes to i.e. 10.20.30.40:36836 Prod VPC https:/ /auth.vivy.services Prod internal ALB Routes to i.e. 10.50.60.70:48162 @bsideup
26. Service discovery benefits @bsideup ✅ Devs can “hardcode” 
 https:/ /something.mycorp.services ✅ Load Balanced (on the server side) ✅ Yet great performance ✅ You don’t need Consul until you need it
27. Service discovery downsides @bsideup ⚠ Every Target in ALB must have a unique priority, even if “host” filter is already unique ⚠ ALB does not work with gRPC
 (Current solution: NLB next to ALB) ⚠ Bottleneck, but only if you’re Netflix
28. Secrets management & config @bsideup Parameters Store AWS KMS
 encrypted params + + https://github.com/remind101/ssm-env
29. @bsideup FROM ****/ssm-env:0.0.2 AS ssm-env FROM openjdk:8u151-jre COPY --from ssm-env /ssm-env /bin/ssm-env COPY build/libs/*.jar /app.jar CMD ["ssm-env", "-with-decryption", "sh", "-c", "java -jar /app.jar”] …and then with env vars: TWILIO_AUTH_TOKEN=ssm:///twilio/accountSid
30. @bsideup FROM ****/ssm-env:0.0.2 AS ssm-env FROM openjdk:8u151-jre COPY --from ssm-env /ssm-env /bin/ssm-env COPY build/libs/*.jar /app.jar CMD ["ssm-env", "-with-decryption", "sh", "-c", "java -jar /app.jar”] …and then with env vars: TWILIO_AUTH_TOKEN=ssm:///twilio/accountSid
31. SSM+KMS benefits @bsideup ✅ Encrypted & managed by AWS ✅ Key rotation ✅ key/value storage of params, both encrypted and not ✅ IAM-controlled ✅ You don’t need Vault until you need it
32. SSM+KMS downsides @bsideup ⚠ No manual key rotation in KMS ⚠ SSM Parameters Store’s UI ⚠ Not integrated with ECS out of the box
33. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2
34. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2 Auto-scaling group v2
35. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2 Auto-scaling group v2 EC2 EC2
36. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2 Auto-scaling group v2 EC2 EC2 Drain instances in ECS
37. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2 Auto-scaling group v2 EC2 EC2
38. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2 Auto-scaling group v2 EC2 EC2
39. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2 Auto-scaling group v2 EC2 EC2
40. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2 Auto-scaling group v2 EC2 EC2
41. Cattle-like instance upgrades @bsideup Auto-scaling group v1 EC2 EC2 Auto-scaling group v2 EC2 EC2
42. Cattle-like instance upgrades @bsideup Auto-scaling group v2 EC2 EC2
43. How @bsideup 1. AWS ASG “EC2_INSTANCE_TERMINATING” LifecycleHook 2. AWS Lambda will mark EC2 instance in ECS as “DRAINING” 3. Complete the lifecycle once no running containers left on EC2 instance
44. Benefits @bsideup ✅ Fargate-like ECS experience ✅ …but in eu-central-1 ✅ Up/Down scaling with no downtime
45. DoS protection @bsideup AWS ALB AWS WAF
 (Web Application Firewall) +
46. AWS WAF benefits @bsideup ✅ Attach it to your ALB ✅ Analyses the access log ✅ Flexible rules (regex by path, headers, …) ✅ Minutes to trigger
47. AWS WAF benefits @bsideup ✅ Attach it to your ALB ✅ Analyses the access log ✅ Flexible rules (regex by path, headers, …) ✅ Minutes to trigger no ALB config changes needed
48. AWS WAF downsides @bsideup ⚠ No “block by user” ⚠ No DDoS protection
49. How to manage?
51. CloudFormation
52. Terraform CloudFormation No vendor lock-in Native service by AWS OSS Managed state Modular New APIs support is fast More than just AWS resources Custom resources with Lambda @bsideup
53. CloudFormation because • The state is managed by AWS, not by us (as it is with TF) • Better documentation and examples • Some APIs/Resources are still not supported in TF • Custom Resources with AWS Lambda @bsideup
54. How to cook templates?
55. @bsideup
56. @bsideup
57. @bsideup
58. @bsideup
60. @bsideup from uvita import ECSMicroService, SSMParameter, DynamoDBTable from troposphere import Template, Parameter t = Template() t.add_parameter(Parameter('Env', Type="String")) t.add_parameter(Parameter('DockerImage', Type="String")) questionnaire_table = DynamoDBTable('QuestionnaireDynamoDBTable', ('subjectId', 'S')) questionnaire_table.inject_to(t) subject_info_table = DynamoDBTable('SubjectInfoDynamoDBTable', ('subjectId', 'S'), ('id', 'S')) subject_info_table.inject_to(t) goal_table = DynamoDBTable('GoalDynamoDBTable', ('subjectId', 'S')) goal_table.inject_to(t) service = ECSMicroService( "sensemaking", priority=2100, public=True, envs = { "kafka_bootstrapServers": "kafka.uvita.services:9092", "kafka_eventBusTopic": "user-event-log", "aws_dynamodb_questionnairesTableName": questionnaire_table.table_ref(), "aws_dynamodb_subjectInfoTableName": subject_info_table.table_ref(), "aws_dynamodb_goalTableName": goal_table.table_ref(), "security_oauth2_resource_jwt_keyValue": SSMParameter('/${Env}/jwt/password') } ) service.inject_to(t)
61. @bsideup from uvita import ECSMicroService, SSMParameter, DynamoDBTable from troposphere import Template, Parameter t = Template() t.add_parameter(Parameter('Env', Type="String")) t.add_parameter(Parameter('DockerImage', Type="String")) questionnaire_table = DynamoDBTable('QuestionnaireDynamoDBTable', ('subjectId', 'S')) questionnaire_table.inject_to(t) subject_info_table = DynamoDBTable('SubjectInfoDynamoDBTable', ('subjectId', 'S'), ('id', 'S')) subject_info_table.inject_to(t) goal_table = DynamoDBTable('GoalDynamoDBTable', ('subjectId', 'S')) goal_table.inject_to(t) service = ECSMicroService( "sensemaking", priority=2100, public=True, envs = { "kafka_bootstrapServers": "kafka.uvita.services:9092", "kafka_eventBusTopic": "user-event-log", "aws_dynamodb_questionnairesTableName": questionnaire_table.table_ref(), "aws_dynamodb_subjectInfoTableName": subject_info_table.table_ref(), "aws_dynamodb_goalTableName": goal_table.table_ref(), "security_oauth2_resource_jwt_keyValue": SSMParameter('/${Env}/jwt/password') } ) service.inject_to(t) Standard Troposphere APIs
62. @bsideup from uvita import ECSMicroService, SSMParameter, DynamoDBTable from troposphere import Template, Parameter t = Template() t.add_parameter(Parameter('Env', Type="String")) t.add_parameter(Parameter('DockerImage', Type="String")) questionnaire_table = DynamoDBTable('QuestionnaireDynamoDBTable', ('subjectId', 'S')) questionnaire_table.inject_to(t) subject_info_table = DynamoDBTable('SubjectInfoDynamoDBTable', ('subjectId', 'S'), ('id', 'S')) subject_info_table.inject_to(t) goal_table = DynamoDBTable('GoalDynamoDBTable', ('subjectId', 'S')) goal_table.inject_to(t) service = ECSMicroService( "sensemaking", priority=2100, public=True, envs = { "kafka_bootstrapServers": "kafka.uvita.services:9092", "kafka_eventBusTopic": "user-event-log", "aws_dynamodb_questionnairesTableName": questionnaire_table.table_ref(), "aws_dynamodb_subjectInfoTableName": subject_info_table.table_ref(), "aws_dynamodb_goalTableName": goal_table.table_ref(), "security_oauth2_resource_jwt_keyValue": SSMParameter('/${Env}/jwt/password') } ) service.inject_to(t) … but with our reusable definitions
63. @bsideup from uvita import ECSMicroService, SSMParameter, DynamoDBTable from troposphere import Template, Parameter t = Template() t.add_parameter(Parameter('Env', Type="String")) t.add_parameter(Parameter('DockerImage', Type="String")) questionnaire_table = DynamoDBTable('QuestionnaireDynamoDBTable', ('subjectId', 'S')) questionnaire_table.inject_to(t) subject_info_table = DynamoDBTable('SubjectInfoDynamoDBTable', ('subjectId', 'S'), ('id', 'S')) subject_info_table.inject_to(t) goal_table = DynamoDBTable('GoalDynamoDBTable', ('subjectId', 'S')) goal_table.inject_to(t) service = ECSMicroService( "sensemaking", priority=2100, public=True, envs = { "kafka_bootstrapServers": "kafka.uvita.services:9092", "kafka_eventBusTopic": "user-event-log", "aws_dynamodb_questionnairesTableName": questionnaire_table.table_ref(), "aws_dynamodb_subjectInfoTableName": subject_info_table.table_ref(), "aws_dynamodb_goalTableName": goal_table.table_ref(), "security_oauth2_resource_jwt_keyValue": SSMParameter('/${Env}/jwt/password') } ) service.inject_to(t) … and higher level abstractions
64. @bsideup from uvita import ECSMicroService, SSMParameter, DynamoDBTable from troposphere import Template, Parameter t = Template() t.add_parameter(Parameter('Env', Type="String")) t.add_parameter(Parameter('DockerImage', Type="String")) questionnaire_table = DynamoDBTable('QuestionnaireDynamoDBTable', ('subjectId', 'S')) questionnaire_table.inject_to(t) subject_info_table = DynamoDBTable('SubjectInfoDynamoDBTable', ('subjectId', 'S'), ('id', 'S')) subject_info_table.inject_to(t) goal_table = DynamoDBTable('GoalDynamoDBTable', ('subjectId', 'S')) goal_table.inject_to(t) service = ECSMicroService( "sensemaking", priority=2100, public=True, envs = { "kafka_bootstrapServers": "kafka.uvita.services:9092", "kafka_eventBusTopic": "user-event-log", "aws_dynamodb_questionnairesTableName": questionnaire_table.table_ref(), "aws_dynamodb_subjectInfoTableName": subject_info_table.table_ref(), "aws_dynamodb_goalTableName": goal_table.table_ref(), "security_oauth2_resource_jwt_keyValue": SSMParameter('/${Env}/jwt/password') } ) service.inject_to(t) … and conventions
65. How to operate?
66. Bastion instances @bsideup https:/ /github.com/widdix/aws-ec2-ssh Users upload their SSH public keys
 in IAM Docker-based commands over SSH
67. @bsideup EC2 SSH connect
68. @bsideup EC2 SSH connect Download public
 SSH key from IAM
69. @bsideup EC2 SSH connect Download public
 SSH key from IAM Sync IAM users
 to local system
70. DEMO
71. What @bsideup 1. Deploy VPC, ECS Cluster, Public + Internal ALBs 2. Deploy an internal micro-service 3. Deploy one public micro-service 4. Debug them with a Bastion instance
72. @bsideup bsideup