Upgrade to Pro — share decks privately, control downloads, hide ads and more …

One AWS Lambda to Rule Them All

1e2ead439777ff94d9b2dd11a0607e01?s=47 Wolf Paulus
November 13, 2020

One AWS Lambda to Rule Them All

Whenever I got a new laptop, or was just (re-) installing Mac OS from scratch, a Java JDK, IntelliJ IDEA, and Tomcat, the "pure Java" HTTP web server environment, were always among the 1st things I installed. How times have changed. Now it's Docker, Python3, PyCharm, and AWS and SAM CLIs that go on first. I still do Java, quite a bit actually, but Python and AWS Lambda are on the rise. An AWS Lambda function can be simple but still quite powerful, doing many things I used to do with Tomcat.
This talk will show an AWS Lambda function, implemented in Python, performing things like:
- Serving an HTML page
- Consuming HTTP Post requests sent from that page HTML page
- Securely storing received information in a Dynamo DB
- Synthesizing text into speech, i.e. returning MP3 (digital audio)
- Calling others AWS Lambda functions
- Calling native libraries or executable that were deployed with the lambda function .. and more.

We will be using the AWS Web UI only very sparingly, but use a YAML file instead wherever it makes sense, like for declaring the DynamoDB.

1e2ead439777ff94d9b2dd11a0607e01?s=128

Wolf Paulus

November 13, 2020
Tweet

More Decks by Wolf Paulus

Other Decks in Technology

Transcript

  1. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com One Lambda to Rule

    Them All Wolf Paulus
 Principal Engineer at Intuit Adjunct Professor at Embry–Riddle Aeronautical University Advisory Committee Member at the University of California, Irvine
  2. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com In the past .

    . .
  3. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com t o d a

    y . . .
  4. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com FEBRUARY 2020

  5. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com FEBRUARY 2020

  6. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com FEBRUARY 2020

  7. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com Runtime Environment Limits •

    The /tmp directory storage is limited to 512 MB. • File descriptors: 1024 • Processes / threads: 1024 • Deployment package size is 50 MB (compressed) 250MB (uncompressed) • Invocation payload (request and response) synchronous calls 6 MB, async. 256KB • Invocation frequency per function: 10x provisioned concurrency • Function timeout: 900s (15 minutes) • Memory range is from 128 to 3008 MB, in 64MB increments http://www.lambdashell.com/ microVM Intel / AMD x86_64
  8. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com https://wolfpaulus.com/one-lambda/

  9. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com Web Service + UI

    : - capture text - optionally translate the text to German - synthesize the text into an MP3 or WAV file - store requests in a DynamoDB - play the audio file back in the user's browser
  10. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com https://le95wslzi8.execute-api.us-west-2.amazonaws.com/ Prod/ui/index.html

  11. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com translate Lambda Function: EN2DE

    Lambda Function: One
  12. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com Prerequisite • Python 3.x

    • PyCharm + AWS Plugin • AWS CLI (AWS Command Line Interface) • SAM CLI (AWS Serverless Application Model Command Line Interface)
  13. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com Prerequisite • AWS Account

    • S3 Bucket • IAM Account (Identity and Access Management) • IAMFullAccess • PowerUserAccess • Create a profile (id, key) on your computer
 (e.g. in ~/.aws/credentials)
  14. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com translate Lambda Function: EN2DE

    Lambda Function: One 1 1
  15. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com import logging from boto3

    import client logger = logging.getLogger(__name__) logging.getLogger().setLevel(logging.INFO) def lambda_handler(event: dict, context) -> dict: """ :param event: input data, usually a dict, but can also be list, str, int, float, or NoneType type. :param context: object providing info about invocation, function, and execution environment :return: dict with TranslatedText, SourceLanguageCode, TargetLanguageCode """ logging.info(str(event)) text = event.get('text', 'no input text provided') translate = client(service_name='translate', region_name='us-west-2', use_ssl=True) return translate.translate_text(Text=text, SourceLanguageCode='en', TargetLanguageCode='de') ./lambda/app.py Calling AWS Translate service and logging in CloudWatch
  16. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com AWSTemplateFormatVersion: '2010-09-09' Transform: AWS::Serverless-2016-10-31

    Description: Lambda, translating English to German Resources: TranslateFunction: Type: AWS::Serverless::Function Properties: FunctionName: EN2DE CodeUri: lambda/ Handler: app.lambda_handler Runtime: python3.8 Timeout: 30 MemorySize: 512 Policies: - TranslateFullAccess LogsLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub '/aws/lambda/${TranslateFunction}' RetentionInDays: 7 Outputs: TranslateFunction: Description: "Translate EN to DE Lambda Function ARN" Value: !GetAtt TranslateFunction.Arn ./template.yaml
  17. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com

  18. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com https://insidelambda.com https://boto3.amazonaws.com/v1/documentation/ api/latest/reference/services/translate.html AWS

    SDK for Python (Boto3)
  19. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com translate Lambda Function: EN2DE

    Lambda Function: One 2
  20. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com ./template.yaml AWSTemplateFormatVersion: '2010-09-09' Transform:

    AWS::Serverless-2016-10-31 Description: One Lambda to rule them all Globals: Api: Cors: AllowMethods: "'GET,POST,OPTIONS'" AllowHeaders: "'content-type'" AllowOrigin: "'*'" AllowCredentials: "'*'" Resources: DynamoDBTable: Type: AWS::DynamoDB::Table Properties: AttributeDefinitions: - AttributeName: "key" AttributeType: "S" - AttributeName: "scope" AttributeType: "S" KeySchema: - AttributeName: "key" # Partition key KeyType: "HASH" - AttributeName: "scope" # Sort key KeyType: "RANGE" ProvisionedThroughput: ReadCapacityUnits: 5 WriteCapacityUnits: 5 SSESpecification: SSEEnabled: true TableName: "lambdaOneRequests" LambdaOneFunction: Type: AWS::Serverless::Function Properties: FunctionName: ONE CodeUri: lambda/ Handler: app.lambda_handler Runtime: python3.8 Timeout: 3 MemorySize: 512 Policies: - AmazonDynamoDBFullAccess - AmazonPollyFullAccess - Version: '2012-10-17' Statement: - Effect: Allow Action: - 'polly:SynthesizeSpeech' Resource: '*' - Effect: Allow Action: - logs:* Resource: arn:aws:logs:*:*:* - Effect: Allow Action: - lambda:InvokeFunction Resource: '*' Events: GetEvent: Type: Api Properties: Path: /ui/{filename} Method: get PostEvent: Type: Api Properties: Path: /{function} Method: post UpdateSchedule: Type: Schedule Properties: Schedule: rate(5 minutes) Input: '{"req":"poll"}' LogsLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub '/aws/lambda/${LambdaOneFun RetentionInDays: 7 Outputs: LambdaOneApi: Description: "Prod stage API Gateway endpoint UR Value: !Sub "https://${ServerlessRestApi}.execut LambdaOneFunction: Description: "LambdaOne Function ARN" Value: !GetAtt LambdaOneFunction.Arn LambdaOneIamRole: Description: "Implicit IAM Role created for Lamb Value: !GetAtt LambdaOneFunction.Arn • The API declaration contains the CORS settings, allowing requests from any origin. • The declaration of the DynamoDBTable will create the table “lambdaOneRequests” with a hash key and a search key • The Lambda function declaration contains, the handler, runtime information, and also all the needed policies, like accessing DynamoDB, Speech Synthesis, accessing the log, or calling a Lambda function. • Events declares the request PATH for HTTP GET and POST requests. Moreover, the UpdateSchedule configures that the Lambda function will be called every 5 minutes, which should keep it warm, i.e. remove startup delays. • The declaration of the LogsLogGroup will create the / aws/lambda/ONE log group and retain entries for 7 days. • The Output declaration will expose the function’s ARN and API Gateway URL CORS DynamoDB Lambda Function HTTP GET / POST Schedule LOG INFO Policies
  21. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com def lambda_handler(event, context) ->

    dict: """ :param event: input data, usually a dict, but can also be list, str, int, float, or NoneType type. :param context: object providing info about invocation, function, and execution environment :return: dict with http header, status code, and body """ if not event.get('httpMethod'): # invoked directly e.g. to keep it warm return {"status_code": 400, "message": "Bad Request"} status_code, content_type, content = get(event['path']) if 'GET' == event['httpMethod'] else post(event) return { "statusCode": status_code, "headers": { # Cross-Origin Resource Sharing (CORS) allows a server to indicate any other origins than its own, # from which a browser should permit loading of resources. 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': True, 'Access-Control-Allow-Methods': 'OPTIONS,POST,GET', 'Content-Type': content_type, }, "body": content } Return Type:
  22. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com HTTP GET /ui/* Request

    -> returns text files def get(file_path: str) -> (int, str, str): """ :return: http code, content-type, file content of the requested text file. """ logging.info(file_path) if file_path.endswith('.css'): content_type = 'text/css; charset=UTF-8' elif file_path.endswith('.js'): content_type = 'text/javascript; charset=UTF-8' else: content_type = 'text/html; charset=UTF-8' file_path = '/ui/index.html' try: with open('.' + file_path, 'r') as file: content = file.read() except OSError: return 404, 'text/html; charset=UTF-8', 'not found' return 200, content_type, content } ui
  23. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com HTTP POST/{function} Request ->

    returns base 64 encoded MP3 or WAV audio def post(event: dict) -> (int, str, str): """ :returns HTTP code, content-type, and message """ params = json.loads(event['body']) # if event.get('body') else event if params: function = event['path'].split('/')[-1].lower() if function == 'synthesize': RequestDB.create(SCOPE, event['requestContext']['identity']['sourceIp'], params['text']) content_type = "audio/wav" if params['format'] == 'wav' else 'audio/mpeg' content = synthesize(params['text'], params['translate'], params['format']) return 200, content_type, json.dumps({"b64": content}) return 400, 'application/json; charset=UTF-8', 'Bad Request'
  24. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com translate Lambda Function: EN2DE

    Lambda Function: One 4 4
  25. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com import base64 from boto3

    import client def synthesize(text: str = "", translate: bool = False, fmt='mp3') -> str: """ Synthesize the provided text to base64 encoded mp3 :param text: text to be synthesized :param translate: bool needs to be translated into German :param fmt: 'wav' or 'mp3' :return: string, the base64 encoded bytes or a wav or mp3 file """ response = client("polly").synthesize_speech(Engine="standard" if translate else "neural", VoiceId="Marlene" if translate else "Joanna", Text=to_german(text) if translate else text, OutputFormat="mp3") binary_data = response["AudioStream"].read() return base64.b64encode(mpeg2wav(binary_data) if fmt == 'wav' else binary_data).decode("utf-8") Calling AWS Polly Service
  26. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com from boto3 import client

    def to_german(english: str) -> str: """ Calling our translate lambda function""" response = client('lambda').invoke( FunctionName='EN2DE', InvocationType='RequestResponse', Payload=bytes(json.dumps({'text': english}), encoding='utf8') ) logger.info(str(response)) response = json.load(response['Payload']) return response.get('TranslatedText') Custom AWS Lambda Calling another Lambda Function Custom
  27. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com translate Lambda Function: EN2DE

    Lambda Function: One 5
  28. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com Calling a native executable

    import logging from os import chmod from shutil import copyfile from dynamo import RequestDB from polly import synthesize logger = logging.getLogger(__name__) logger.setLevel(logging.INFO) copyfile("./ffmpeg", '/tmp/ffmpeg') chmod("/tmp/ffmpeg", 755) 
 
 RequestDB.init("dynamodb") SCOPE = "awscd" ./lambda/app.py
  29. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com Calling a native linux

    64 bit x86 executable import subprocess def mpeg2wav(mp3_bytes: bytes) -> bytes: """ :param mp3_bytes: raw mp3 data :return: wav bytes prefixed with the proper header """ wav_bytes = subprocess.Popen(["/tmp/ffmpeg", "-i", "pipe:0", "-f", "wav", "pipe:1"], shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE).communicate(mp3_bytes)[0] return create_header(22000, 16, 1, len(str(wav_bytes))) + wav_bytes
  30. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com translate Lambda Function: EN2DE

    Lambda Function: One 6
  31. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com Storing request information in

    a DynamoDB from boto3 import resource from botocore.exceptions import ClientError class RequestDB: TABLE_NAME = "lambdaOneRequests" dynamodb = None table = None @classmethod def init(cls, res: str): if res == "dynamodb": cls.dynamodb = resource("dynamodb") cls.table = cls.dynamodb.Table(cls.TABLE_NAME) @classmethod def _create_key(cli): return uuid.uuid4().hex @classmethod def create(cls, scope: str, ip: str, text: str): if not scope or not ip or not text: return 400, "Bad Request" try: key = scope + RequestDB._create_key() logging.info("Checking if key is new " + str(key)) response = cls.table.get_item(Key={'key': key, 'scope': scope}) if "Item" in response: return 500, "Unique key creation failed" logging.info("Creating record for " + str(key)) response = cls.table.put_item( Item={'key': key, 'scope': scope, 'ip': cls.hash_pin(ip), 'created': datetime.utcnow().isoformat(), 'text': text}) logging.info(str(response)) return 200, "Record Created" except ClientError as e: logging.error(str(e)) return 500, e.response['Error']['Message']
  32. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com def post(event: dict) ->

    (int, str, str): """ :returns HTTP code, content-type, and message """ params = json.loads(event['body']) # if event.get('body') else event if params: function = event['path'].split('/')[-1].lower() if function == 'synthesize': RequestDB.create(SCOPE, event['requestContext']['identity']['sourceIp'], params['text']) content_type = "audio/wav" if params['format'] == 'wav' else 'audio/mpeg' content = synthesize(params['text'], params['translate'], params['format']) return 200, content_type, json.dumps({"b64": content}) return 400, 'application/json; charset=UTF-8', 'Bad Request' Storing request information in a DynamoDB
  33. Wolf Paulus - https://wolfpaulus.com - wolf@paulus.com https://wolfpaulus.com/one-lambda/