Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Tooling will make or Break Serverless Infrastructure

How Tooling will make or Break Serverless Infrastructure

From ServerlessConf Austin 2017

Recodring: https://www.youtube.com/watch?v=ELlQVTAKc8A

Florian Motlik

April 28, 2017
Tweet

More Decks by Florian Motlik

Other Decks in Technology

Transcript

  1. HOW TOOLING WILL MAKE OR
    BREAK SERVERLESS
    INFRASTRUCTURE

    View full-size slide

  2. WHO AM I
    Florian Motlik
    ‣Co-Founder and former CTO of Codeship
    ‣Former CTO of Serverless Inc
    AWS/Serverless/Development Process Consultant

    Need help in your infrastructure: flo@flomotlik.me
    T W I T T E R : @ F L O M O T L I K
    W E B S I T E : F L O M O T L I K . M E
    E M A I L : F L O @ F L O M O T L I K . M E

    View full-size slide

  3. WHAT IS SERVERLESS

    View full-size slide

  4. EVENT DRIVEN SYSTEM
    Service A
    F1 F2 F3
    Service C
    F1 F2
    Service B
    F1
    Service E
    F1 F2 F3
    Service F
    F1 F2 F3
    Service D
    F1 F2 F3 F4 F5
    S3
    Kinesis
    DynamoDB
    Api Gateway

    View full-size slide

  5. CLOUD 2.0, SUPER ADVANCED CLOUD, …

    View full-size slide

  6. MIGHTY SCALIN POWER CLOUD
    It’s Scalin’ Time!

    View full-size slide

  7. THE SERVERLESS TEAM
    ▸ Least Amount of Control
    ▸ Least Amount of Responsibility
    ▸ Complete Automation
    ▸ Gradual Escalation of Control and Responsibility
    ▸ Build for Replacement

    View full-size slide

  8. WHAT DOES THIS MEAN
    FOR OUR TOOLS

    View full-size slide

  9. HEAVY LIFTING IS DONE IN SERVICES
    ▸ Scheduling of Tasks
    ▸ Scaling of System
    ▸ Collecting of Log Data
    ▸ Authentication between Services
    ▸ Storing and managing data storage
    ▸ Provisioning and scaling Queue infrastructure
    ▸ …

    View full-size slide

  10. WHAT IS IMPORTANT, PAST VS PRESENT
    Past
    Features
    ‣ Replication
    ‣ Scalability
    ‣ …

    View full-size slide

  11. PAST
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER SERVER
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER

    View full-size slide

  12. PAST
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER SERVER
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER
    Application cluster
    Database cluster
    Data and ETL Pipeline

    View full-size slide

  13. PAST VS PRESENT
    Past Present
    Features
    ‣ Replication
    ‣ Scalability
    ‣ …
    Usability
    Accessibility
    Best Practices

    View full-size slide

  14. MAKE IT EASY TO COMBINE SERVICES
    Service A
    F1 F2 F3
    Service C
    F1 F2
    Service B
    F1
    Service E
    F1 F2 F3
    Service F
    F1 F2 F3
    Service D
    F1 F2 F3 F4 F5
    S3
    Kinesis
    DynamoDB
    Api Gateway

    View full-size slide

  15. BAD TOOLS WILL PUSH US AWAY FROM SERVICES
    ▸ Many examples in AWS from great services that aren’t used enough
    ▸ CloudFormation
    ▸ IAM
    ▸ CloudWatch Metrics, Logs and Alerts
    ▸ Custom Resources
    ▸ AWS Config

    View full-size slide

  16. SAME PROBLEM NO
    MATTER THE COMPANY SIZE

    View full-size slide

  17. UNREASONABLY
    COMPLEX TO START

    View full-size slide

  18. SO WHATS HOLDING US
    BACK

    View full-size slide

  19. CLOUDFORMATION IS THE
    MAKE OR BREAK SERVICE

    View full-size slide

  20. HORRIBLE CLI TOOLING FROM AWS
    ▸ Hard to learn and hard to read configuration
    ▸ No built-in modularity for one stack making config files huge
    ▸ There is no built-in command to deploy a stack, wait for it to be deployed
    and get the CF events as cli output

    View full-size slide

  21. CLOUDFORMATION IS
    OUR ASSEMBLY CODE

    View full-size slide

  22. ALL TOOLS SHOULD WORK TOWARDS CLOUDFORMATION
    ▸ Standard Endpoint allows us to swap out tools
    ▸ Not possible with other tools like Terraform as state is stored separately

    View full-size slide

  23. ALL TOOLS SHOULD WORK TOWARDS CLOUDFORMATION
    ▸ Standard Endpoint allows us to swap out tools
    ▸ Not possible with other tools like Terraform as state is stored separately
    ▸ Easier to support standards

    View full-size slide

  24. ALL TOOLS SHOULD WORK TOWARDS CLOUDFORMATION
    ▸ Standard Endpoint allows us to swap out tools
    ▸ Not possible with other tools like Terraform as state is stored separately
    ▸ Easier to support standards
    ▸ This also makes sure we’re building better low level tools with better UX

    View full-size slide

  25. SIMPLE TOOLS TO MAKE CF MORE USEFUL
    ▸ Awsie (https://github.com/flomotlik/awsie)
    awsie example-stack s3 ls s3://cf:DeploymentBucket: --region us-west-1
    aws s3 ls s3://formica-example-stack-deploymentbucket-1jjzisylxreh9 --region us-west-1

    awsie example-stack —command “echo //cf:DeploymentBucket:” --region us-west-1

    echo s3://formica-example-stack-deploymentbucket-1jjzisylxreh9

    View full-size slide

  26. WHAT WE NEED TO
    WATCH OUT FOR

    View full-size slide

  27. STOP BUILDING ABSTRACTIONS

    View full-size slide

  28. STOP BUILDING ABSTRACTIONS
    I need this small thing
    Hmm it could
    also do this
    Lets introduce this small abstraction
    to make it applicable here and here
    Wow this got really
    complex, new tool it is

    View full-size slide

  29. LETS JUST STOP THIS

    View full-size slide

  30. THERE IS NO ONE TOOL
    TO RULE THEM ALL

    View full-size slide

  31. PAST
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER SERVER
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER
    SERVER
    Application cluster
    Database cluster
    Data and Background Pipeline

    View full-size slide

  32. MONOLITHIC TOOLS IN FRONT OF
    MICRO-SERVICE ARCHITECTURE?
    (KEVIN MCGRATH, AUTHOR OF CONDENSATION)

    View full-size slide

  33. MICRO-TOOLING
    (KEVIN MCGRATH, AUTHOR OF CONDENSATION)

    View full-size slide

  34. POWER AND CONTROL OVER YOUR INFRASTRUCTURE
    HARDWARE CLOUD INSTANCES PAAS SERVERLESS

    View full-size slide

  35. THE POWER IS IN COMBINING SERVICES, SO DON'T HIDE MORE HERE
    Service A
    F1 F2 F3
    Service C
    F1 F2
    Service B
    F1
    Service E
    F1 F2 F3
    Service F
    F1 F2 F3
    Service D
    F1 F2 F3 F4 F5
    S3
    Kinesis
    DynamoDB
    Api Gateway

    View full-size slide

  36. CONSTANT NEW FEATURES ARE UNUSABLE DUE TO ABSTRACTIONS

    View full-size slide

  37. PROBLEMS WITH MULTI-PROVIDER TOOLING
    ▸ Always gets dragged down to lowest common denominator
    ▸ In a Serverless infrastructure you’re not going to move Everything between
    providers
    ▸ Small surface area per provider
    ▸ You want to get all you can out of that provider so you need specialised
    tooling

    View full-size slide

  38. WHAT CAN WE DO
    ABOUT IT

    View full-size slide

  39. LETS FIX
    CLOUDFORMATION

    View full-size slide

  40. LETS FIX
    CLOUDFORMATION
    NO, STOP IT

    View full-size slide

  41. IT NEEDS LOVE AND UX

    View full-size slide

  42. MY JOURNEY TO BUILD FORMICA
    ▸ https://github.com/flomotlik/formica
    ▸ Wanted to build and play with more AWS services so I needed a good CloudFormation tool
    ▸ Looked around at existing tools (Sceptre, Stacker, AWLess, Serverless, Gordon, Condensation, …) but all define their
    own abstractions
    ▸ Stack environments
    ▸ Dependencies between stacks
    ▸ Stack naming,
    ▸ Language to create the stack
    ▸ …
    ▸ So of course I built it using Troposphere so I could have nice python templates and added my own abstraction layer

    View full-size slide

  43. MAKEFILE IS MY
    DEPLOYMENT FRAMEWORK
    KAPIL THANGAVELU (@KAPILVT)
    FROM CAPITALONE

    View full-size slide

  44. THE UNIX PHILOSOPHY
    CLOUDFORMATION IS OUR PIPES

    View full-size slide

  45. THE UNIX PHILOSOPHY
    ALSO PIPES ARE OUR PIPES

    View full-size slide

  46. DOCUMENTAITON, NOT
    JUST TOOLING

    View full-size slide

  47. DOESN’T MEAN WE CAN’T IMPROVE CF SYNTAX
    ▸ Serverless/Gordon/Zappa are great for setting up an API Gateway with
    Lambdas
    ▸ We could have other tools as preprocessors for specific things
    ▸ Simple tool to translate HTTP endpoint definition to CF
    ▸ Tool to create CF templates for CloudWatch Alerts and Metrics

    View full-size slide

  48. SUITE OF SMALL FOCUSED TOOLS
    ▸ Built:
    ▸ Formica - Deploying CloudFormation templates
    ▸ AWSIE - Dynamically use CF ids in your shell commands
    ▸ Ideas:
    ▸ Orgonaut: Org/Account management and running commands against each command or group
    ▸ Auditor: Auditing and reporting setup with AWS Config and Cloudtrail across accounts
    ▸ Pipes: CI/CD setup through CodePipeline/CodeBuild
    ▸ Watchdog: CloudWatch config tool to make Monitoring and Alerts easier to configure
    ▸ Work together as a community to highlight those tools and define a best-practice suite

    View full-size slide

  49. THEY ARE SMALL
    THEY ARE SIMPLE
    GO BUILD SOME

    View full-size slide

  50. LEARN CLOUD
    FORMATION IN-DEPTH

    View full-size slide

  51. FLOMOTLIK.ME
    Florian Motlik
    AWS/Serverless/Development Process Consultant

    Need help in your infrastructure: flo@flomotlik.me
    T W I T T E R : @ F L O M O T L I K
    W E B S I T E : F L O M O T L I K . M E
    E M A I L : F L O @ F L O M O T L I K . M E

    View full-size slide

  52. CONCLUSIONS
    ▸ Composition and Accessibility, not Features and Abstractions
    ▸ Build and use small tools that do one thing well
    ▸ Compose larger systems out of several smaller tools
    ▸ Go check out Make, its reasonably good
    ▸ Join the Serverless Slack (Serverless-Forum, https://wt-serverless-
    seattle.run.webtask.io/serverless-forum-signup?webtask_no_cache=1)
    T W I T T E R : @ F L O M O T L I K
    W E B S I T E : F L O M O T L I K . M E
    E M A I L : F L O @ F L O M O T L I K . M E

    View full-size slide