Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to send gzipped requests with boto3

How to send gzipped requests with boto3

Most AWS APIs will have limits on the amount of data you can send in one request and sometimes you really need to send a lot of data! To try to maximize the amount of data you can send, while still staying within the limits, some APIs support sending gzip-compressed payloads. But how can you send a gzipped request when using the Python SDK for AWS (boto3)? Well, I needed to answer this question recently and it turned out not to be as easy as I anticipated… Let’s jump into this rabbit hole together and let’s find out the answer!

F3a6662b3cd161c3c2f13604965ed0f2?s=128

Luciano Mammino

July 01, 2021
Tweet

Transcript

  1. How to send gzipped requests with boto3 Luciano Mammino (

    ) @loige AWS UserGroup Dublin 2021-07-01 loige.link/gzip-boto3 1
  2. loige.link/gzip-boto3 Get these slides! loige 2

  3. Let me introduce myself first... 3

  4. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋

    3
  5. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋

    Senior Architect @ fourTheorem (Dublin ) 3
  6. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋

    Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 3
  7. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋

    Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 Connect with me: (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 3
  8. We are business focused technologists that deliver. | | Accelerated

    Serverless AI as a Service Platform Modernisation We are hiring: do you want to ? work with us loige 4
  9. loige In the previous episodes... SLIC WATCH: SLIDES: fth.link/slic-watch fth.link/o11y-no-pain

    5
  10. loige ⚠ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland) Threshold crossed: 1

    out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition). @wake_me_up_bot 6
  11. loige 7

  12. loige 8

  13. loige This guy was failing... a lot! 8

  14. loige 9

  15. for event in payload['logEvents']: # ... cw_client.put_metric_data(...) loige 10

  16. for event in payload['logEvents']: # ... cw_client.put_metric_data(...) loige Sending 1

    metric per log line... 🙈 10
  17. [ { 'MetricName': 'SomeMetric1', 'Dimensions': [ { 'Name': 'Dimension1Name', 'Value':

    'Dimension1Value' }, # Up to other 9 dimensions here ... ], 'Unit': 'Count', 'Values': [217, 220, 221], # Up to 150 items here ... 'Timestamp': 1624290910000 }, # Up to other 19 metric items here ... ] loige 11
  18. [ { 'MetricName': 'SomeMetric1', 'Dimensions': [ { 'Name': 'Dimension1Name', 'Value':

    'Dimension1Value' }, # Up to other 9 dimensions here ... ], 'Unit': 'Count', 'Values': [217, 220, 221], # Up to 150 items here ... 'Timestamp': 1624290910000 }, # Up to other 19 metric items here ... ] loige ⚠ Payload size limit: 40 KB 11
  19. loige 12

  20. loige 🤔 12

  21. loige Maybe boto3 gzips the data automatically! 😏 13

  22. import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch',

    endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14
  23. import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch',

    endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14
  24. import boto3 import gzip endpoint_url = "http://localhost:8000/" cw_client = boto3.client('cloudwatch',

    endpoint_url=endpoint_url, use_ssl=False) cw_client.put_metric_data( MetricData = [ { 'MetricName': 'TEST_BOTO', 'Dimensions': [ { 'Name': 'APP_VERSION', 'Value': '1.0' }, ], 'Unit': 'None', 'Value': 17 }, ], Namespace='BotoTest' ) loige 14
  25. loige 15

  26. loige ... and there is no magic flag like GzipPayload=True!

    😭 16
  27. loige Can we extend boto3 somehow? 🤔 17

  28. loige boto3 has an event system! 🤩 youtu.be/eM8uoGJO2AI @thekyleknapp 18

  29. import boto3 lambda_client = boto3.client('lambda') # our event handler def

    add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 19
  30. import boto3 lambda_client = boto3.client('lambda') # our event handler def

    add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19
  31. import boto3 lambda_client = boto3.client('lambda') # our event handler def

    add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19
  32. import boto3 lambda_client = boto3.client('lambda') # our event handler def

    add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19
  33. import boto3 lambda_client = boto3.client('lambda') # our event handler def

    add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 loige 19
  34. import boto3 lambda_client = boto3.client('lambda') # our event handler def

    add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # invoke a lambda function lambda_client.invoke(FunctionName='my-function') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 15 16 loige 19
  35. import boto3 lambda_client = boto3.client('lambda') # our event handler def

    add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') 1 2 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 5 6 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # get the event system for the lambda_client event_system = lambda_client.meta.events import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 9 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 12 13 14 # invoke a lambda function 15 lambda_client.invoke(FunctionName='my-function') 16 # invoke a lambda function lambda_client.invoke(FunctionName='my-function') import boto3 1 2 lambda_client = boto3.client('lambda') 3 4 # our event handler 5 def add_xtrace_header(request, **kwargs): 6 request.headers.add_header('x-trace-id', 'trace-trace') 7 8 # get the event system for the lambda_client 9 event_system = lambda_client.meta.events 10 11 # attach an event handler to the client for 12 event_system.register('before-sign.lambda.Invoke', add_xtrace_header) 13 14 15 16 import boto3 lambda_client = boto3.client('lambda') # our event handler def add_xtrace_header(request, **kwargs): request.headers.add_header('x-trace-id', 'trace-trace') # get the event system for the lambda_client event_system = lambda_client.meta.events # attach an event handler to the client for event_system.register('before-sign.lambda.Invoke', add_xtrace_header) # invoke a lambda function lambda_client.invoke(FunctionName='my-function') 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 19
  36. loige Event naming convention <event-type>.<service-name>.<operation-name> 20

  37. loige Event naming convention <event-type>.<service-name>.<operation-name> provide-client-params request-created before-sign before-send response-received

    20
  38. loige Event naming convention <event-type>.<service-name>.<operation-name> provide-client-params request-created before-sign before-send response-received

    s3 cloudwatch lambda ... 20
  39. loige Event naming convention <event-type>.<service-name>.<operation-name> provide-client-params request-created before-sign before-send response-received

    s3 cloudwatch lambda ... ListBuckets PutMetricData Invoke ... 20
  40. loige You can be a 🌟 too! * (every event)

    after-call.*.* (all responses) after-call.lambda.* (all responses for lambda) 21
  41. loige OK, now we know enough to write our own

    event handler for gzip compression! 🤓 22
  42. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 23
  43. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
  44. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
  45. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
  46. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
  47. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
  48. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 loige 23
  49. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 loige 23
  50. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 cw_client.put_metric_data(...) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 17 loige 23
  51. import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events

    # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 import gzip import boto3 1 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events import boto3 1 import gzip 2 3 4 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 7 8 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 request.headers.add_header('Content-Encoding', 'gzip') import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 gzipped_body = gzip.compress(request.body) request.data = gzipped_body import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 10 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 cw_client.put_metric_data(...) 17 # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 13 14 15 16 cw_client.put_metric_data(...) 17 cw_client.put_metric_data(...) import boto3 1 import gzip 2 3 cw_client = boto3.client('cloudwatch') 4 event_system = cw_client.meta.events 5 6 # Gzip handler 7 def gzip_request_body(request, **kwargs): 8 request.headers.add_header('Content-Encoding', 'gzip') 9 gzipped_body = gzip.compress(request.body) 10 request.data = gzipped_body 11 12 # registers the custom handler 13 event_system.register('before-sign.cloudwatch.PutMetricData', 14 gzip_request_body) 15 16 17 import boto3 import gzip cw_client = boto3.client('cloudwatch') event_system = cw_client.meta.events # Gzip handler def gzip_request_body(request, **kwargs): request.headers.add_header('Content-Encoding', 'gzip') gzipped_body = gzip.compress(request.body) request.data = gzipped_body # registers the custom handler event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body) cw_client.put_metric_data(...) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 23
  52. loige 24

  53. loige Other potential use cases Log data before/after it is

    sent to AWS Add additional headers to AWS requests (tracing) Additional input validation (your custom rules) Payload enrichment (e.g. make sure certain tags exist) 25
  54. loige What if you don't use Python? Customizing the AWS

    SDK for Go V2 Client Requests Introducing Middleware Stack in Modular AWS SDK for JavaScript Handlers and Middleware in the AWS SDK for PHP Version 3 26
  55. loige What did we learn? Always have alarms for your

    lambdas! (have you checked already? 😉) You can use Gzip compression to send big payloads to AWS Boto3 is extensible through its event system! SLIC Watch 27
  56. loige Do you think boto3 should make it easier to

    send gzipped payloads? 👍 UPVOTE this issue: github.com/boto/botocore/issues/2425 28
  57. loige Do you want more details? loige.link/gzip-boto3-article 29

  58. Cover picturea by on Thanks doggie by Stefanie Shank Moritz

    Mentges Unsplash loige nodejsdp.link loige.link/gzip-boto3 30