Tweet
Tweet

Slide 1

Slide 1 text

AWS LAMBDA @ . Juan Manuel Torres César López-Natarén Jan. 2016

Slide 2

Slide 2 text

JUAN AND CÉSAR MindTouch Engineers

Slide 3

Slide 3 text

JUAN MANUEL TORRES ➤ Born and raised in Bogotá Colombia ➤ Software Engineer @ ➤ Love ➤ Programming ➤ Photography ➤ Lizards ➤ Traveling ➤ twitter.com/onema ➤ [email protected]

Slide 4

Slide 4 text

CÉSAR LÓPEZ-NATARÉN ➤ Senior Software Engineer @ ➤ Love to work on: ➤ APIs ➤ Data engineering ➤ Making code fast ➤ Infrastructure automation ➤ twitter.com/nataren ➤ [email protected]

Slide 5

Slide 5 text

Customer Success SaaS

Slide 6

Slide 6 text

➤ Authoring and publishing platform for Product Help ➤ We use your product’s knowledge to: ➤ Accelerate User Adoption ➤ Build Customer Loyalty ➤ We help you: ➤ Understand your Customers ➤ Accelerate your Sales Cycle

Slide 7

Slide 7 text

SOME MINDTOUCH CUSTOMERS

Slide 8

Slide 8 text

MINDTOUCH’S SCALE ➤ Up to 2M req/day ➤ Up to 5M events/day

Slide 9

Slide 9 text

AWS LAMBDA Serverless computing

Slide 10

Slide 10 text

WHAT IS AWS LAMBDA? ➤ Compute service that hosts an execution environment for your code on high availability compute infrastructure. ➤ The service: ➤ Administers the compute resources (CPU, memory, network and OS maintenance). ➤ Guarantees the selected capacity. ➤ Scales your software automatically. ➤ Provides monitoring and logging services.

Slide 11

Slide 11 text

WHY DOES IT MATTER? ➤ Because all the tasks described before consume programmer’s time ➤ And instead of focusing on your problem domain you are dealing with those other things ➤ It is actually complicated to get all those pieces right

Slide 12

Slide 12 text

HOW DO I USE IT? ➤ You provide your Node.js, Java, or Python code. ➤ Configure: ➤ Event sources ➤ Memory ➤ Execution role ➤ Handler function

Slide 13

Slide 13 text

LET’S GET ONE PYTHON LAMBDA FUNCTION DEPLOYED photo by Kool Cats Photography

Slide 14

Slide 14 text

DEMO Lambda S3 to Tweet

Slide 15

Slide 15 text

LAMBDA FUNCTION LIFE CYCLE ➤ Develop ➤ Test ➤ Deploy ➤ Repeat

Slide 16

Slide 16 text

LAMBDA FUNCTION DEVELOPMENT photo by Lachlan Hardy

Slide 17

Slide 17 text

PROGRAMMING MODEL / ENTRY POINT Handler 1 def lambda_handler(event, context): 2 s3_info = event['Records'][0]['s3'] 3 ... 4 tweet_images.send_image( 5 s3_info['bucket']['name'], 6 s3_info['object']['key'], 7 cleanup=True 8 )

Slide 18

Slide 18 text

PROGRAMMING MODEL / LOGGING Logging 1 import logging 2 3 logger = logging.getLogger() 4 logger.setLevel(logging.INFO) 5 6 def lambda_handler(event, context): 7 logger.info("Invoked with {0} events".format( 8 len(event['Records'])))

Slide 19

Slide 19 text

LAMBDA FUNCTION TESTING

Slide 20

Slide 20 text

TESTING OPTIONS ➤ Local testing through unit testing ➤ Make sure you can swap out implementations of components that perform I/O ➤ Mock testing through the AWS Console in the Lambda service itself ➤ Create your own simulation environment for Lambda

Slide 21

Slide 21 text

photo by NASA's Marshall Space Flight Center LAMBDA FUNCTION DEPLOYMENT

Slide 22

Slide 22 text

DEPLOYMENT ➤ Creating the lambda function on the service ➤ Granting permissions to event sources and any other components that the functions needs to interact with ➤ Sending the packaged code to the service ➤ Configuring the application

Slide 23

Slide 23 text

AWS LAMBDA USAGE @ MINDTOUCH ➤ Mirror data streams across accounts and regions ➤ Archive events to S3 ➤ Generate SQS messages from S3 object creation ➤ Publish Auto Scaling Group’s notifications to DataDog

Slide 24

Slide 24 text

CHALLENGES FACED while using AWS Lambda

Slide 25

Slide 25 text

CHALLENGES WHILE USING AWS LAMBDA ➤ Consistent function configuration ➤ Automated function deployment on different environments (dev, test, production)

Slide 26

Slide 26 text

MINDTOUCH LAMBDA OPS MindTouch deployment tool for Lambda functions

Slide 27

Slide 27 text

MINDTOUCH LAMBDA OPS DESIGN OBJECTIVE To enable a consistent development/test/ deployment lifecycle for AWS Lambda functions.

Slide 28

Slide 28 text

TOOL REQUIREMENTS ➤ Developers + DevOps ➤ Configuration must be simple • Single file • Checked-in to version control • Many deployments ➤ Permission management ➤ Managing Event Source Mappings

Slide 29

Slide 29 text

IMPLEMENTATION DETAILS ➤ Single YAML Configuration file ➤ Automatic generation of config.json including names for all sources and resources ➤ Prepares lambda function for deployment ➤ Creates a CloudFormation stack

Slide 30

Slide 30 text

DEMO Kinesis to S3

Slide 31

Slide 31 text

KINESIS2S3XMLARCHIVER CONFIGURATION 1 app: 2 3 resources: 4 source: 5 type: kinesis 6 name: events-stream 7 event-source: 8 batch-size: 10000 9 starting-position: latest 10 11 s3bucket: 12 type: s3 13 name: events-archive 14 allow: 15 - s3:PutObject 16 - s3:GetObject

Slide 32

Slide 32 text

photo by Andreas Levers WARNINGS

Slide 33

Slide 33 text

LIMITS ➤ Execution time is limited to 300s (5 minutes) ➤ 1.5GB memory limits photo by Navalarp Teratanatorn

Slide 34

Slide 34 text

SECURITY ➤ As Jan 2016 there is no support for accessing resources on a VPC (Coming soon)

Slide 35

Slide 35 text

DEPLOYMENT ➤ No straight forward deployment other than the console or using a third party tool photo by NASA's Marshall Space Flight Center

Slide 36

Slide 36 text

CONCLUSIONS ➤ Huge developer productivity gains ➤ Reduces complexity in large infrastructure systems ➤ They do what they claim to do ➤ We will choose AWS Lambda for suitable projects in the future

Slide 37

Slide 37 text

MINDTOUCH IS HIRING! www.mindtouch.com/jobs Join the Nerds