How to send gzipped requests with boto3

Most AWS APIs will have limits on the amount of data you can send in one request and sometimes you really need to send a lot of data! To try to maximize the amount of data you can send, while still staying within the limits, some APIs support sending gzip-compressed payloads. But how can you send a gzipped request when using the Python SDK for AWS (boto3)? Well, I needed to answer this question recently and it turned out not to be as easy as I anticipated… Let’s jump into this rabbit hole together and let’s find out the answer!

Transcript

1. How to send gzipped requests with boto3 Luciano Mammino ( ) @loige AWS UserGroup Dublin 2021-07-01 loige.link/gzip-boto3 1
2. loige.link/gzip-boto3 Get these slides! loige 2
3. Let me introduce myself first... 3
4. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋 3
5. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋 Senior Architect @ fourTheorem (Dublin ) 3
6. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋 Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 3
7. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋 Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 Connect with me: (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 3
8. We are business focused technologists that deliver. | | Accelerated Serverless AI as a Service Platform Modernisation We are hiring: do you want to ? work with us loige 4
9. loige In the previous episodes... SLIC WATCH: SLIDES: fth.link/slic-watch fth.link/o11y-no-pain 5
10. loige ⚠ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland) Threshold crossed: 1 out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition). @wake_me_up_bot 6
11. loige 7
12. loige 8
13. loige This guy was failing... a lot! 8
14. loige 9
15. for event in payload['logEvents']: # ... cw_client.put_metric_data(...) loige 10
16. for event in payload['logEvents']: # ... cw_client.put_metric_data(...) loige Sending 1 metric per log line... 🙈 10
17. [ { 'MetricName': 'SomeMetric1', 'Dimensions': [ { 'Name': 'Dimension1Name', 'Value': 'Dimension1Value' }, # Up to other 9 dimensions here ... ], 'Unit': 'Count', 'Values': [217, 220, 221], # Up to 150 items here ... 'Timestamp': 1624290910000 }, # Up to other 19 metric items here ... ] loige 11
18. [ { 'MetricName': 'SomeMetric1', 'Dimensions': [ { 'Name': 'Dimension1Name', 'Value': 'Dimension1Value' }, # Up to other 9 dimensions here ... ], 'Unit': 'Count', 'Values': [217, 220, 221], # Up to 150 items here ... 'Timestamp': 1624290910000 }, # Up to other 19 metric items here ... ] loige ⚠ Payload size limit: 40 KB 11
19. loige 12
20. loige 🤔 12
21. loige Maybe boto3 gzips the data automatically! 😏 13
22. import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14
23. import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14
24. import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14
25. loige 15
26. loige ... and there is no magic flag like GzipPayload=True! 😭 16
27. loige Can we extend boto3 somehow? 🤔 17
28. loige boto3 has an event system! 🤩 youtu.be/eM8uoGJO2AI @thekyleknapp 18
29. import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 19
30. import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19
31. import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19
32. import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19
33. import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19
34. import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # invoke a lambda function lambda_client.invoke(FunctionName='my-function') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 15 16 loige 19
35. import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # invoke a lambda function lambda_client.invoke(FunctionName='my-function') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 19
36. loige Event naming convention <event-type>.<service-name>.<operation-name> 20
37. loige Event naming convention <event-type>.<service-name>.<operation-name> provide-client-params request-created before-sign before-send response-received 20
38. loige Event naming convention <event-type>.<service-name>.<operation-name> provide-client-params request-created before-sign before-send response-received s3 cloudwatch lambda ... 20
39. loige Event naming convention <event-type>.<service-name>.<operation-name> provide-client-params request-created before-sign before-send response-received s3 cloudwatch lambda ... ListBuckets PutMetricData Invoke ... 20
40. loige You can be a 🌟 too! * (every event) after-call.*.* (all responses) after-call.lambda.* (all responses for lambda) 21
41. loige OK, now we know enough to write our own event handler for gzip compression! 🤓 22
42. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 23
43. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
44. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
45. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
46. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
47. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
48. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
49. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 loige 23
50. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 cw_client.put_metric_data(...) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 17 loige 23
51. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 cw_client.put_metric_data(...) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 17 import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 23
52. loige 24
53. loige Other potential use cases Log data before/after it is sent to AWS Add additional headers to AWS requests (tracing) Additional input validation (your custom rules) Payload enrichment (e.g. make sure certain tags exist) 25
54. loige What if you don't use Python? Customizing the AWS SDK for Go V2 Client Requests Introducing Middleware Stack in Modular AWS SDK for JavaScript Handlers and Middleware in the AWS SDK for PHP Version 3 26
55. loige What did we learn? Always have alarms for your lambdas! (have you checked already? 😉) You can use Gzip compression to send big payloads to AWS Boto3 is extensible through its event system! SLIC Watch 27
56. loige Do you think boto3 should make it easier to send gzipped payloads? 👍 UPVOTE this issue: github.com/boto/botocore/issues/2425 28
57. loige Do you want more details? loige.link/gzip-boto3-article 29
58. Cover picturea by on Thanks doggie by Stefanie Shank Moritz Mentges Unsplash loige nodejsdp.link loige.link/gzip-boto3 30