Configuration management and service discovery in a serverless world

Your system is composed of highly decoupled, independent, fast, and modular microservices. But how can they share common configurations, dynamic endpoints, database references, and properly rotate secrets? Based on the size and complexity of your serverless system, you may simply use environment variables or eventually opt for some sort of centralized store. And then how do integrate all of this with monitoring and automation tooling?

During this session, I will present the ideal solutions and some of the alternatives available on AWS (such as AWS Systems Manager Parameter Store and AWS Secrets Manager). I will also discuss the best use cases for each solution and the corresponding best practices to achieve the highest standards for security and performance.

Transcript

1. Alex Casalboni Technical Evangelist, AWS @alex_casalboni @ 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Configuration management and service discovery in a serverless world
2. About me • Software Engineer & Web Developer • Worked in a startup for 4.5 years • ServerlessDays Organizer • AWS Customer since 2013
3. Agenda 1. Serverless security & configuration management 2. Live demo 3. Cost optimization techniques
4. Serverless security & configuration management @alex_casalboni
5. Lambda permission model Fine-grained security controls for both execution and invocation Execution policies Define what AWS resources/API calls can this function access via AWS IAM Used in streaming invocations For example, “Lambda function A can read from DynamoDB table users” Function policies Used for sync and async invocations Resource policies allow for cross account access For example, “Actions on bucket X can invoke Lambda function Z"
6. Action: “s3:*” … make puppies cry! Action: “dynamodb:*" Action: “sns:*“ Photo by Matthew Henry on Unsplash
7. MyFunction: Type: AWS::Serverless::Function Properties: Handler: index.handler Runtime: python2.7 Policies: - AWSLambdaExecute # Managed Policy - Version: '2012-10-17' Statement: - Effect: Allow Action: - dynamodb:GetItem Resource: !GetAtt MyDynamoDBTable.Arn Fine-grained IAM policy with AWS SAM MyFunction: Type: AWS::Serverless::Function Properties: Handler: index.handler Runtime: python2.7 Policies: - AWSLambdaExecute # Managed Policy - Version: '2012-10-17' Statement: - Effect: Allow Action: - dynamodb:GetItem Resource: !GetAtt MyDynamoDBTable.Arn
8. Hardcoded secrets make fish cry! Photo by Julieann Ragojo on Unsplash
9. AWS Lambda environment variables Key-value pairs that you can dynamically pass to your function Available via standard environment variable APIs (based on runtime) Can optionally be encrypted via AWS KMS Allows you to specify in IAM what roles have access to the keys to decrypt the information Useful for creating environments per stage (such as dev, test, prod)
10. AWS Systems Manager―Parameter Store Centralized store to manage your configuration data Supports hierarchies Plaintext or encrypted with AWS KMS Can send notifications of changes to Amazon SNS or Lambda Can be secured with IAM Calls recorded in AWS CloudTrail Can be tagged Available via API/SDK Useful for centralized environment variables, secrets control, feature flags
11. Parameter Store access via SDK import json, boto3 ssm = boto3.client('ssm') def get_parameter(): response = ssm.get_parameter( Name=‘my_param’, WithDecryption=True ) return response['Parameter']['Value'] def lambda_handler(event, context): value = get_parameter() print(”value = %s" % value)
12. Parameter Store access via SDK with ssm_cache import json, boto3 ssm = boto3.client('ssm') def get_parameter(): response = ssm.get_parameter( Name=‘my_param’, WithDecryption=True ) return response['Parameter']['Value'] def lambda_handler(event, context): value = get_parameter() print(”value = %s" % value) from ssm_cache import SSMParameter param = SSMParameter(‘my_param’) def lambda_handler(event, context): value = param.value print(”value = %s" % value) github.com/alexcasalboni/ssm-cache-python
13. AWS Secrets Manager Allows you to manage, retrieve, and rotate credentials Helps you rotate secrets regularly without breaking stuff Keeps track of different password versions Implements security controls associated with credential management Built-in support for Amazon RDS
14. AWS Secrets Manager + Parameter Store Uniform and consistent access to both services You can reference Secrets Manager secrets with Parameter Store APIs Rotation & Refresh delegated to the client As simple as using a prefix: /aws/reference/secretsmanager/ +
15. Secrets access via Parameter Store import json, boto3 ssm = boto3.client('ssm’) prefix = ‘/aws/reference/secretsmanager’ def get_secret(): response = ssm.get_parameter( Names=[‘%s/my_secret’ % prefix], WithDecryption=True ) return response['Parameter']['Value'] def lambda_handler(event, context): value = get_secret() print(”value = %s" % value)
16. Secrets access via Parameter Store with ssm_cache import json, boto3 ssm = boto3.client('ssm’) prefix = ‘/aws/reference/secretsmanager’ def get_secret(): response = ssm.get_parameter( Names=[‘%s/my_secret’ % prefix], WithDecryption=True ) return response['Parameter']['Value'] def lambda_handler(event, context): value = get_secret() print(”value = %s" % value) from ssm_cache import SecretsManagerParameter secret = SecretsManagerParameter(‘my_secret’) def lambda_handler(event, context): value = secret.value print(”value = %s" % value) github.com/alexcasalboni/ssm-cache-python
17. Demo time! @alex_casalboni amzn.to/serverless-security
18. Serverless cost optimization techniques @alex_casalboni
19. Anatomy of a function Your function Language runtime Function container Compute substrate
20. The request lifecycle Bootstrap the runtime Start your code Cold start Warm start Download your code Start new container AWS optimization Your optimization
21. Same view in AWS X-Ray
22. Efficient function code Avoid monolithic functions (or “fat”) Control the dependencies in your function's deployment package Optimize for your language Node.js – Browserfy, Minify, Webpack
23. Ephemeral function environment Lambda processes a single event per-container No need for non-blocking execution on the frontend REMEMBER – containers are reused Lazily load variables in global scope Don’t load it if you don’t need it
24. Lazy initialization example (Python & boto3) import boto3 S3_client = None ddb_client = None def get_objects(event, context): if not s3_client: s3_client = boto3.client("s3") # business logic def get_items(event, context): if not ddb_client: ddb_client = boto3.client(”dynamodb") # business logic
25. Optimized dependency usage (Node.js SDK & X-Ray) // const AWS = require('aws-sdk’) const DynamoDB = require('aws-sdk/clients/dynamodb’) // 125ms faster // const AWSXRay = require('aws-xray-sdk’) const AWSXRay = require('aws-xray-sdk-core’) // 5ms faster // const AWS = AWSXRay.captureAWS(require('aws-sdk’)) const dynamodb = new DynamoDB.DocumentClient() AWSXRay.captureAWSClient(dynamodb.service) // 140ms faster @theburningmonk theburningmonk.com/2019/03/just-how-expensive-is-the-full-aws-sdk/
26. Concise function logic Separate Lambda handler from core logic Use functions to TRANSFORM, not TRANSPORT Read only what you need Query filters in Amazon Aurora Use Amazon S3 select
27. Concise function logic (example) from mylib import MyLibClass def lambda_handler(event, context): operation = event['Operation’] myobj = MyLibClass() if operation == ‘do_this’: my_obj.do_this() elif operation == ‘do_that’: myobj.do_that() else: raise ValueError(‘Invalid op’)
28. Concise function logic (example) import boto3 ddb = boto3.client(‘dynamodb’) class MyLibClass(object): MY_CONSTANT = ‘blabla’ def __init__(…): # constructor def do_this(self): # use ddb to do this def do_that(self): # use ddb to do that from mylib import MyLibClass def lambda_handler(event, context): operation = event['Operation’] myobj = MyLibClass() if operation == ‘do_this’: my_obj.do_this() elif operation == ‘do_that’: myobj.do_that() else: raise ValueError(‘Invalid op’)
29. Small changes, big difference # Download and process all keys for key in src_keys: response = s3_client.get_object(…) contents = response['Body'].read() for line in contents.split('\n')[:-1]: line_count +=1 try: data = line.split(',') srcIp = data[0][:8] … # Select IP Address and Keys for key in src_keys: response = s3_client.select_object_content( expression=“SELECT SUBSTR(obj._1, 1, 8), obj._2 FROM s3object as obj”) contents = response['Body'].read() for line in contents: line_count +=1 try: … After (95s, $0.028) Before (200s, $0.112) https://github.com/awslabs/lambda-refarch-mapreduce
30. Smart resource allocation Match resource allocation (up to 3 GB!) to logic Stats for Lambda function that calculates 1000 times all prime numbers <= 1000000 128 MB 11.722s $0.024628 256 MB 6.6789s $0.028035 512 MB 3.1949s $0.026830 1024 MB 1.4659s $0.024638
31. “AWS Lambda Power Tuning” Data-driven cost & performance optimization for AWS Lambda github.com/alexcasalboni/aws-lambda-power-tuning Don’t guesstimate!
32. No orchestration in code START JOB JOB #X STARTED HTTP POST HTTP POST ARE WE THERE YET? NOPE! WE’RE DONE! ZzZz OR time.sleep(10)
33. No orchestration in code
34. Gateways & routers Choose suitable entry point for client applications Single, custom client? Use the AWS SDK Not end user facing? Use regional endpoints on API Gateway Discard uninteresting events ASAP S3 – Event prefix SNS – Message filtering
35. Resilient: retry policies Understand retry policies Sync never retried Async retried 2 times Streams retried all the time Leverage Dead Letter Queues (DLQ) SQS or SNS for replays REMEMBER: Retries count as invokes
36. Concurrency Controls Concurrency a shared pool by default Separate using per function concurrency settings Acts as reservation Also acts as max concurrency per function Especially critical for data sources like RDS “Kill switch” – set per function concurrency to zero
37. Should my Lambda function be in a VPC? Does my function need to access any specific resources in a VPC? Does it also need to access resources or services in the public internet? Don’t put the function in a VPC Put the function in a private subnet Put the function in a subnet with a NAT’d route to the internet Yes Yes No No Do I need a VPC?
38. Alex Casalboni Technical Evangelist, AWS @alex_casalboni @ 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Thank you!