CNN Digital Publishing Platform

Transcript

1. Digital Publishing Platform & AWS Lambda By Federico Cargnelutti Senior Software Engineer, CNN
2. AWS Lambda is an event-driven, serverless computing platform provided by AWS. Applications react to events, and Amazon runs the server and dynamically manages the allocation of resources. Node and Python are the leading languages for Lambda: • Node: 67.6% • Python: 26.4% • Others: 6% (*) Source: epsagon.com AWS Lambda
3. Stellar uses the Pub/Sub pattern to notify other services when certain events take place. CMS Events: • clay:createPage • clay:publishLayout • clay:publishPage • clay:save • clay:saveMeta • clay:saveUser • clay:schedulePage • clay:unpublishPage • clay:unschedulePag CMS Events
4. The amphora-event-bus package uses the Amazon SNS module to publish CMS Events to an SNS topic so that any number of distributed systems can subscribe to messages. Publisher
5. An SNS Topic (event bus) is used to achieve anonymity between publisher and subscriber. Lambdas subscribe to the topic to receive messages published to that topic. Event Bus
6. AWS SNS + SQS + Lambda • Amazon SNS sends an Amazon SQS message to the subscribed queue. • Amazon SQS notifies AWS Lambda (hey, wake up!) • AWS Lambda polls the SQS queue and invokes the Lambda function synchronously with an event that contains queue messages. • The Lambda function (or handler) reads messages in batches and invokes your function once for each batch.
7. SQS Message The Amazon SQS message will look similar to this: When your function successfully processes a batch, Lambda deletes its messages from the queue.
8. Lambda Hibernation AWS has the ability to launch Lambda functions, set them up as desired, hibernate them, and then bring them back to life when needed. The hibernation process stores the in-memory state of the instance, along with its private and elastic IP addresses, allowing it to pick up exactly where it left off.
9. Stellar 2.0 Serverless Architecture
10. Infrastructure (...gulp!)
11. CloudFormation Templates CloudFormation is an AWS tool for deploying infrastructure. AWS uses the CloudFormation template specification to define a Lambda function: • Template sections • Resources type • Resource properties • Data types
12. CloudFormation Templates You describe your desired infrastructure in YAML then submit your CloudFormation template for deployment:
13. Create Stack The aws-stack-builder.sh script automates the deployment of the Lambda stack using CloudFormation templates. ./aws-stack-builder.sh -c create-deploy -t ec2-compute-pipeline -b ec2-clay -p ps0001 -y -L all The script is responsible for: • Creating an SQS Standard Queue • Configuring the SQS queue to trigger an AWS Lambda function • Creating an SNS topic • Adding an SNS Subscription Filter Policy • Subscribing the SQS queue to an SNS topic
14. To deploy only lambda functions: ./aws-stack-builder.sh -c deploy -y -k 4da396 -b lambdas-clay -L all Deploy Lambdas