Deploying your python backend with (almost) zero cost

Srinivasan Rangarajan Systems Architect

Physical Servers • Takes few days to provision • Heavily under-utilized • Priced per month • Administrative nightmare

Cloud Servers • Few minutes to provision • Create new servers using APIs • Can be turned off when not in use • Priced per hour

Lambda Functions • Milliseconds to provision • 100% utilization • Priced every 100ms • No servers to manage • Continuous & automatic scaling

How to do it? 1. Create your lambda function. 2. Specify the resource which triggers the function. 3. ??? 4. Profit.

Triggers • File gets uploaded into S3 • Receive new message in SQS • Inserts into DynamoDB • Plain old HTTP request

hello_world.py def my_handler(event, context): message = 'Hello {} {}!'.format(event['first_name'], event['last_name']) return { 'message' : message }

event • Usually a dict • list • str • int • float • Nonetype

context • LambdaContext type • how much time is remaining for termination • function_name • function_version • memory_limit_in_mb

Deployment Package • Create a project-dir folder • Write your .py files in that folder • pip install all dependencies 
 pip install module_name --target /path/to/project-dir • Zip the project-dir

Chalice

$ pip install chalice $ chalice new-project helloworld $ cd helloworld

app.py from chalice import Chalice app = Chalice(app_name="helloworld") @app.route("/") def index(): return {"hello": “world"} @app.route("/city/{city}") def city(city): return {"city": city}

Deploy $ chalice deploy Creating deployment package. Lambda deploy done. API Gateway rest API already found. Deleting root resource id Done deleting existing resources. Deploying to: dev https://68mxz0v402.execute-api.us-east-1.amazonaws.com/dev/

Browser/curl 1. https://68mxz0v402.execute-api.us- east-1.amazonaws.com/dev/ • http://bit.do/lambda_index 2. https://68mxz0v402.execute-api.us- east-1.amazonaws.com/dev/city/delhi • http://bit.do/lambda_city

request object • app.current_request.query_params • app.current_request.headers • app.current_request.uri_params • app.current_request.method • app.current_request.json_body • app.current_request.raw_body • app.current_request.context • app.current_request.stage_vars

Examples

Update Product Metadata r = redis.Redis() @app.route('/product/{pid}/update', methods=['POST']) def product_update(pid): """Update price, availability info about a product in redis""" body = app.current_request.json_body client_id = body['client_id'] prod_key = '{}:{}'.format(client_id, pid) price = body.get('price') if price: r.hset(prod_key, 'price', price) availability = body.get('availability') if availability: r.hset(prod_key, 'availability', availability)

Thumbnail Creation @app.route('/', methods=['POST']) def index(): body = app.current_request.json_body image = base64.b64decode(body['data']) format = {'jpg': 'jpeg', 'png': 'png'}[body.get('format', 'jpg').lower()] mode = {'max': '', 'min': '^', 'exact': '!'}[body.get('mode', 'max').lower()] width = int(body.get('width', 128)) height = int(body.get('height', 128)) https://github.com/Schweigi/thumbnail-service

Thumbnail Creation cmd = [ 'convert', # ImageMagick Convert '-', # Read original picture from StdIn '-auto-orient', # Detect picture orientation from metadata '-thumbnail', '{}x{}{}'.format(width, height, mode), # Thumbnail size '-extent', '{}x{}'.format(width, height), # Fill if original picture i smaller than thumbnail '-gravity', 'Center', # Extend (fill) from the thumbnail middle '-unsharp',' 0x.5', # Un-sharpen slightly to improve small thumbnails '-quality', '80%', # Thumbnail JPG quality '{}:-'.format(format), # Write thumbnail with `format` to StdOut ] p = Popen(cmd, stdout=PIPE, stdin=PIPE) thumbnail = p.communicate(input=image)[0] https://github.com/Schweigi/thumbnail-service

Thumbnail Creation if not thumbnail: raise BadRequestError('Image format not supported') filename = '{}_{}x{}.{}'.format(uuid.uuid4(), width, height, format) S3.put_object( Bucket=S3_BUCKET, Key=filename, Body=thumbnail, ACL='public-read', ContentType='image/{}'.format(format), ) return { 'url': 'https://s3.amazonaws.com/{}/{}'.format(S3_BUCKET, filename) } https://github.com/Schweigi/thumbnail-service

Other Use Cases

Data Pipeline Log file parsing and data cleansing

Static Blog commenting system HTTP Gateway + DynamoDB

Genome Editing & Sub-second Searches http://benchling.engineering/crispr-aws-lambda/

Continuous Integration

lambci • Node.js (multiple versions via nave) • Python 2.7 • Java (OpenJDK 1.8 and 1.7) • Go (any version) • Ruby (2.3.1, 2.2.5, 2.1.10, 2.0.0-p648) • PHP (7.0.10, 5.6.25) • Native compilation with a pre-built gcc 4.8.5 • Rust (1.11.0, 1.10.0, but any version should work) https://github.com/lambci/lambci

Chat Bots

Stateless A: I love Stateless Systems B: Don’t they have drawbacks? A: Don’t what have drawbacks?

Limitations • Only 300 seconds of execution time. • Max 1.5GB RAM. • Max 512 MB /tmp disk space. • Max 6MB Request/Response payload • Only python 2.7

Advantages • Automatically scalable • Built-in Fault Tolerance • Automatic Administration • Pay per use

Pricing • $0.000000208 per 100ms for 128MB RAM • $0.000002501 per 100ms for 1.5GB RAM • $0.0000002 per request • Free tier level • 400,000 GB-seconds per month • 1Million requests

Links • https://aws.amazon.com/lambda/ • http://chalice.readthedocs.io/ • https://github.com/Schweigi/thumbnail-service • https://github.com/lambci/lambci • http://benchling.engineering/crispr-aws-lambda/

• https://github.com/cnu • https://twitter.com/cnu • [email protected]