Share
Embed
How to send gzipped requests with boto3
Most AWS APIs will have limits on the amount of data you can send in one request and sometimes you really need to send a lot of data! To try to maximize the amount of data you can send, while still staying within the limits, some APIs support sending gzip-compressed payloads. But how can you send a gzipped request when using the Python SDK for AWS (boto3)? Well, I needed to answer this question recently and it turned out not to be as easy as I anticipated… Let’s jump into this rabbit hole together and let’s find out the answer!
Luciano Mammino
July 01, 2021
Transcript
58 slides
-
How to send gzipped requests with boto3 Luciano Mammino (
) @loige AWS UserGroup Dublin 2021-07-01 loige.link/gzip-boto3 1 -
Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin ) 3 -
Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 3 -
Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 Connect with me: (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 3 -
We are business focused technologists that deliver. | | Accelerated
Serverless AI as a Service Platform Modernisation We are hiring: do you want to ? work with us loige 4 -
loige ⚠ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland) Threshold crossed: 1
out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition). @wake_me_up_bot 6 -
for event in payload['logEvents']: # ... cw_client.put_metric_data(...) loige Sending 1
metric per log line... 🙈 10 -
[ { 'MetricName': 'SomeMetric1', 'Dimensions': [ { 'Name': 'Dimension1Name', 'Value':
'Dimension1Value' }, # Up to other 9 dimensions here ... ], 'Unit': 'Count', 'Values': [217, 220, 221], # Up to 150 items here ... 'Timestamp': 1624290910000 }, # Up to other 19 metric items here ... ] loige 11 -
[ { 'MetricName': 'SomeMetric1', 'Dimensions': [ { 'Name': 'Dimension1Name', 'Value':
'Dimension1Value' }, # Up to other 9 dimensions here ... ], 'Unit': 'Count', 'Values': [217, 220, 221], # Up to 150 items here ... 'Timestamp': 1624290910000 }, # Up to other 19 metric items here ... ] loige ⚠ Payload size limit: 40 KB 11 -
import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch',
endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14 -
import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch',
endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14 -
import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch',
endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14 -
import boto3 lambda_client = boto3.client('lambda') # our event handler def
add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 19 -
import boto3 lambda_client = boto3.client('lambda') # our event handler def
add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19 -
import boto3 lambda_client = boto3.client('lambda') # our event handler def
add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19 -
import boto3 lambda_client = boto3.client('lambda') # our event handler def
add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19 -
import boto3 lambda_client = boto3.client('lambda') # our event handler def
add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19 -
import boto3 lambda_client = boto3.client('lambda') # our event handler def
add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # invoke a lambda function lambda_client.invoke(FunctionName='my-function') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 15 16 loige 19 -
import boto3 lambda_client = boto3.client('lambda') # our event handler def
add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # invoke a lambda function lambda_client.invoke(FunctionName='my-function') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 19 -
loige Event naming convention <event-type>.<service-name>.<operation-name> provide-client-params request-created before-sign before-send response-received
s3 cloudwatch lambda ... ListBuckets PutMetricData Invoke ... 20 -
loige You can be a 🌟 too! * (every event)
after-call.*.* (all responses) after-call.lambda.* (all responses for lambda) 21 -
loige OK, now we know enough to write our own
event handler for gzip compression! 🤓 22 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 cw_client.put_metric_data(...) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 17 loige 23 -
import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events
# Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 cw_client.put_metric_data(...) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 17 import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 23 -
loige Other potential use cases Log data before/after it is
sent to AWS Add additional headers to AWS requests (tracing) Additional input validation (your custom rules) Payload enrichment (e.g. make sure certain tags exist) 25 -
loige What if you don't use Python? Customizing the AWS
SDK for Go V2 Client Requests Introducing Middleware Stack in Modular AWS SDK for JavaScript Handlers and Middleware in the AWS SDK for PHP Version 3 26 -
loige What did we learn? Always have alarms for your
lambdas! (have you checked already? 😉) You can use Gzip compression to send big payloads to AWS Boto3 is extensible through its event system! SLIC Watch 27 -
loige Do you think boto3 should make it easier to
send gzipped payloads? 👍 UPVOTE this issue: github.com/boto/botocore/issues/2425 28 -
Cover picturea by on Thanks doggie by Stefanie Shank Moritz
Mentges Unsplash loige nodejsdp.link loige.link/gzip-boto3 30