How to send gzipped requests with boto3 Luciano Mammino ( ) @loige AWS UserGroup Dublin 2021-07-01 loige.link/gzip-boto3 1

loige.link/gzip-boto3 Get these slides! loige 2

Let me introduce myself first... 3

Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋 3

Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋 Senior Architect @ fourTheorem (Dublin ) 3

Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋 Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 3

Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋 Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 Connect with me: (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 3

We are business focused technologists that deliver. | | Accelerated Serverless AI as a Service Platform Modernisation We are hiring: do you want to ? work with us loige 4

loige In the previous episodes... SLIC WATCH: SLIDES: fth.link/slic-watch fth.link/o11y-no-pain 5

loige ⚠ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland) Threshold crossed: 1 out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition). @wake_me_up_bot 6

loige 7

loige 8

loige This guy was failing... a lot! 8

loige 9

for event in payload['logEvents']: # ... cw_client.put_metric_data(...) loige 10

for event in payload['logEvents']: # ... cw_client.put_metric_data(...) loige Sending 1 metric per log line... 🙈 10

[ { 'MetricName': 'SomeMetric1', 'Dimensions': [ { 'Name': 'Dimension1Name', 'Value': 'Dimension1Value' }, # Up to other 9 dimensions here ... ], 'Unit': 'Count', 'Values': [217, 220, 221], # Up to 150 items here ... 'Timestamp': 1624290910000 }, # Up to other 19 metric items here ... ] loige 11

[ { 'MetricName': 'SomeMetric1', 'Dimensions': [ { 'Name': 'Dimension1Name', 'Value': 'Dimension1Value' }, # Up to other 9 dimensions here ... ], 'Unit': 'Count', 'Values': [217, 220, 221], # Up to 150 items here ... 'Timestamp': 1624290910000 }, # Up to other 19 metric items here ... ] loige ⚠ Payload size limit: 40 KB 11

loige 12

loige 🤔 12

loige Maybe boto3 gzips the data automatically! 😏 13

import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14

import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14

import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14

loige 15

loige ... and there is no magic flag like GzipPayload=True! 😭 16

loige Can we extend boto3 somehow? 🤔 17

loige boto3 has an event system! 🤩 youtu.be/eM8uoGJO2AI @thekyleknapp 18

import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 19

import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19

import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19

import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19

import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19

import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # invoke a lambda function lambda_client.invoke(FunctionName='my-function') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 15 16 loige 19

import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # invoke a lambda function lambda_client.invoke(FunctionName='my-function') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 19

loige Event naming convention .. 20

loige Event naming convention .. provide-client-params request-created before-sign before-send response-received 20

loige Event naming convention .. provide-client-params request-created before-sign before-send response-received s3 cloudwatch lambda ... 20

loige Event naming convention .. provide-client-params request-created before-sign before-send response-received s3 cloudwatch lambda ... ListBuckets PutMetricData Invoke ... 20

loige You can be a 🌟 too! * (every event) after-call.*.* (all responses) after-call.lambda.* (all responses for lambda) 21

loige OK, now we know enough to write our own event handler for gzip compression! 🤓 22

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 cw_client.put_metric_data(...) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 17 loige 23

import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 cw_client.put_metric_data(...) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 17 import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 23

loige 24

loige Other potential use cases Log data before/after it is sent to AWS Add additional headers to AWS requests (tracing) Additional input validation (your custom rules) Payload enrichment (e.g. make sure certain tags exist) 25

loige What if you don't use Python? Customizing the AWS SDK for Go V2 Client Requests Introducing Middleware Stack in Modular AWS SDK for JavaScript Handlers and Middleware in the AWS SDK for PHP Version 3 26

loige What did we learn? Always have alarms for your lambdas! (have you checked already? 😉) You can use Gzip compression to send big payloads to AWS Boto3 is extensible through its event system! SLIC Watch 27

loige Do you think boto3 should make it easier to send gzipped payloads? 👍 UPVOTE this issue: github.com/boto/botocore/issues/2425 28

loige Do you want more details? loige.link/gzip-boto3-article 29

Cover picturea by on Thanks doggie by Stefanie Shank Moritz Mentges Unsplash loige nodejsdp.link loige.link/gzip-boto3 30