Configuration management and service discovery in a serverless world

Configuration management and service discovery in a serverless world

Your system is composed of highly decoupled, independent, fast, and modular microservices. But how can they share common configurations, dynamic endpoints, database references, and properly rotate secrets? Based on the size and complexity of your serverless system, you may simply use environment variables or eventually opt for some sort of centralized store. And then how do integrate all of this with monitoring and automation tooling?

During this session, I will present the ideal solutions and some of the alternatives available on AWS (such as AWS Systems Manager Parameter Store and AWS Secrets Manager). I will also discuss the best use cases for each solution and the corresponding best practices to achieve the highest standards for security and performance.

36ca8ae7e19067021d0e39c0b72acc2a?s=128

Alex Casalboni

April 06, 2019
Tweet

Transcript

  1. 1.

    Alex Casalboni Technical Evangelist, AWS @alex_casalboni @ 2019, Amazon Web

    Services, Inc. or its Affiliates. All rights reserved Configuration management and service discovery in a serverless world
  2. 2.

    About me • Software Engineer & Web Developer • Worked

    in a startup for 4.5 years • ServerlessDays Organizer • AWS Customer since 2013
  3. 5.

    Lambda permission model Fine-grained security controls for both execution and

    invocation Execution policies Define what AWS resources/API calls can this function access via AWS IAM Used in streaming invocations For example, “Lambda function A can read from DynamoDB table users” Function policies Used for sync and async invocations Resource policies allow for cross account access For example, “Actions on bucket X can invoke Lambda function Z"
  4. 7.

    MyFunction: Type: AWS::Serverless::Function Properties: Handler: index.handler Runtime: python2.7 Policies: -

    AWSLambdaExecute # Managed Policy - Version: '2012-10-17' Statement: - Effect: Allow Action: - dynamodb:GetItem Resource: !GetAtt MyDynamoDBTable.Arn Fine-grained IAM policy with AWS SAM MyFunction: Type: AWS::Serverless::Function Properties: Handler: index.handler Runtime: python2.7 Policies: - AWSLambdaExecute # Managed Policy - Version: '2012-10-17' Statement: - Effect: Allow Action: - dynamodb:GetItem Resource: !GetAtt MyDynamoDBTable.Arn
  5. 9.

    AWS Lambda environment variables Key-value pairs that you can dynamically

    pass to your function Available via standard environment variable APIs (based on runtime) Can optionally be encrypted via AWS KMS Allows you to specify in IAM what roles have access to the keys to decrypt the information Useful for creating environments per stage (such as dev, test, prod)
  6. 10.

    AWS Systems Manager―Parameter Store Centralized store to manage your configuration

    data Supports hierarchies Plaintext or encrypted with AWS KMS Can send notifications of changes to Amazon SNS or Lambda Can be secured with IAM Calls recorded in AWS CloudTrail Can be tagged Available via API/SDK Useful for centralized environment variables, secrets control, feature flags
  7. 11.

    Parameter Store access via SDK import json, boto3 ssm =

    boto3.client('ssm') def get_parameter(): response = ssm.get_parameter( Name=‘my_param’, WithDecryption=True ) return response['Parameter']['Value'] def lambda_handler(event, context): value = get_parameter() print(”value = %s" % value)
  8. 12.

    Parameter Store access via SDK with ssm_cache import json, boto3

    ssm = boto3.client('ssm') def get_parameter(): response = ssm.get_parameter( Name=‘my_param’, WithDecryption=True ) return response['Parameter']['Value'] def lambda_handler(event, context): value = get_parameter() print(”value = %s" % value) from ssm_cache import SSMParameter param = SSMParameter(‘my_param’) def lambda_handler(event, context): value = param.value print(”value = %s" % value) github.com/alexcasalboni/ssm-cache-python
  9. 13.

    AWS Secrets Manager Allows you to manage, retrieve, and rotate

    credentials Helps you rotate secrets regularly without breaking stuff Keeps track of different password versions Implements security controls associated with credential management Built-in support for Amazon RDS
  10. 14.

    AWS Secrets Manager + Parameter Store Uniform and consistent access

    to both services You can reference Secrets Manager secrets with Parameter Store APIs Rotation & Refresh delegated to the client As simple as using a prefix: /aws/reference/secretsmanager/ +
  11. 15.

    Secrets access via Parameter Store import json, boto3 ssm =

    boto3.client('ssm’) prefix = ‘/aws/reference/secretsmanager’ def get_secret(): response = ssm.get_parameter( Names=[‘%s/my_secret’ % prefix], WithDecryption=True ) return response['Parameter']['Value'] def lambda_handler(event, context): value = get_secret() print(”value = %s" % value)
  12. 16.

    Secrets access via Parameter Store with ssm_cache import json, boto3

    ssm = boto3.client('ssm’) prefix = ‘/aws/reference/secretsmanager’ def get_secret(): response = ssm.get_parameter( Names=[‘%s/my_secret’ % prefix], WithDecryption=True ) return response['Parameter']['Value'] def lambda_handler(event, context): value = get_secret() print(”value = %s" % value) from ssm_cache import SecretsManagerParameter secret = SecretsManagerParameter(‘my_secret’) def lambda_handler(event, context): value = secret.value print(”value = %s" % value) github.com/alexcasalboni/ssm-cache-python
  13. 20.

    The request lifecycle Bootstrap the runtime Start your code Cold

    start Warm start Download your code Start new container AWS optimization Your optimization
  14. 22.

    Efficient function code Avoid monolithic functions (or “fat”) Control the

    dependencies in your function's deployment package Optimize for your language Node.js – Browserfy, Minify, Webpack
  15. 23.

    Ephemeral function environment Lambda processes a single event per-container No

    need for non-blocking execution on the frontend REMEMBER – containers are reused Lazily load variables in global scope Don’t load it if you don’t need it
  16. 24.

    Lazy initialization example (Python & boto3) import boto3 S3_client =

    None ddb_client = None def get_objects(event, context): if not s3_client: s3_client = boto3.client("s3") # business logic def get_items(event, context): if not ddb_client: ddb_client = boto3.client(”dynamodb") # business logic
  17. 25.

    Optimized dependency usage (Node.js SDK & X-Ray) // const AWS

    = require('aws-sdk’) const DynamoDB = require('aws-sdk/clients/dynamodb’) // 125ms faster // const AWSXRay = require('aws-xray-sdk’) const AWSXRay = require('aws-xray-sdk-core’) // 5ms faster // const AWS = AWSXRay.captureAWS(require('aws-sdk’)) const dynamodb = new DynamoDB.DocumentClient() AWSXRay.captureAWSClient(dynamodb.service) // 140ms faster @theburningmonk theburningmonk.com/2019/03/just-how-expensive-is-the-full-aws-sdk/
  18. 26.

    Concise function logic Separate Lambda handler from core logic Use

    functions to TRANSFORM, not TRANSPORT Read only what you need Query filters in Amazon Aurora Use Amazon S3 select
  19. 27.

    Concise function logic (example) from mylib import MyLibClass def lambda_handler(event,

    context): operation = event['Operation’] myobj = MyLibClass() if operation == ‘do_this’: my_obj.do_this() elif operation == ‘do_that’: myobj.do_that() else: raise ValueError(‘Invalid op’)
  20. 28.

    Concise function logic (example) import boto3 ddb = boto3.client(‘dynamodb’) class

    MyLibClass(object): MY_CONSTANT = ‘blabla’ def __init__(…): # constructor def do_this(self): # use ddb to do this def do_that(self): # use ddb to do that from mylib import MyLibClass def lambda_handler(event, context): operation = event['Operation’] myobj = MyLibClass() if operation == ‘do_this’: my_obj.do_this() elif operation == ‘do_that’: myobj.do_that() else: raise ValueError(‘Invalid op’)
  21. 29.

    Small changes, big difference # Download and process all keys

    for key in src_keys: response = s3_client.get_object(…) contents = response['Body'].read() for line in contents.split('\n')[:-1]: line_count +=1 try: data = line.split(',') srcIp = data[0][:8] … # Select IP Address and Keys for key in src_keys: response = s3_client.select_object_content( expression=“SELECT SUBSTR(obj._1, 1, 8), obj._2 FROM s3object as obj”) contents = response['Body'].read() for line in contents: line_count +=1 try: … After (95s, $0.028) Before (200s, $0.112) https://github.com/awslabs/lambda-refarch-mapreduce
  22. 30.

    Smart resource allocation Match resource allocation (up to 3 GB!)

    to logic Stats for Lambda function that calculates 1000 times all prime numbers <= 1000000 128 MB 11.722s $0.024628 256 MB 6.6789s $0.028035 512 MB 3.1949s $0.026830 1024 MB 1.4659s $0.024638
  23. 31.

    “AWS Lambda Power Tuning” Data-driven cost & performance optimization for

    AWS Lambda github.com/alexcasalboni/aws-lambda-power-tuning Don’t guesstimate!
  24. 32.

    No orchestration in code START JOB JOB #X STARTED HTTP

    POST HTTP POST ARE WE THERE YET? NOPE! WE’RE DONE! ZzZz OR time.sleep(10)
  25. 34.

    Gateways & routers Choose suitable entry point for client applications

    Single, custom client? Use the AWS SDK Not end user facing? Use regional endpoints on API Gateway Discard uninteresting events ASAP S3 – Event prefix SNS – Message filtering
  26. 35.

    Resilient: retry policies Understand retry policies Sync never retried Async

    retried 2 times Streams retried all the time Leverage Dead Letter Queues (DLQ) SQS or SNS for replays REMEMBER: Retries count as invokes
  27. 36.

    Concurrency Controls Concurrency a shared pool by default Separate using

    per function concurrency settings Acts as reservation Also acts as max concurrency per function Especially critical for data sources like RDS “Kill switch” – set per function concurrency to zero
  28. 37.

    Should my Lambda function be in a VPC? Does my

    function need to access any specific resources in a VPC? Does it also need to access resources or services in the public internet? Don’t put the function in a VPC Put the function in a private subnet Put the function in a subnet with a NAT’d route to the internet Yes Yes No No Do I need a VPC?
  29. 38.

    Alex Casalboni Technical Evangelist, AWS @alex_casalboni @ 2019, Amazon Web

    Services, Inc. or its Affiliates. All rights reserved Thank you!