How Tooling will make or Break Serverless Infrastructure

How Tooling will make or Break Serverless Infrastructure

From ServerlessConf Austin 2017

Recodring: https://www.youtube.com/watch?v=ELlQVTAKc8A

7a9e2c47c8719d221fd711b135b33341?s=128

Florian Motlik

April 28, 2017
Tweet

Transcript

  1. HOW TOOLING WILL MAKE OR BREAK SERVERLESS INFRASTRUCTURE

  2. WHO AM I Florian Motlik ‣Co-Founder and former CTO of

    Codeship ‣Former CTO of Serverless Inc AWS/Serverless/Development Process Consultant
 Need help in your infrastructure: flo@flomotlik.me T W I T T E R : @ F L O M O T L I K W E B S I T E : F L O M O T L I K . M E E M A I L : F L O @ F L O M O T L I K . M E
  3. WHAT IS SERVERLESS

  4. EVENT DRIVEN SYSTEM Service A F1 F2 F3 Service C

    F1 F2 Service B F1 Service E F1 F2 F3 Service F F1 F2 F3 Service D F1 F2 F3 F4 F5 S3 Kinesis DynamoDB Api Gateway
  5. CLOUD 2.0, SUPER ADVANCED CLOUD, …

  6. MIGHTY SCALIN POWER CLOUD It’s Scalin’ Time!

  7. THE SERVERLESS TEAM ▸ Least Amount of Control ▸ Least

    Amount of Responsibility ▸ Complete Automation ▸ Gradual Escalation of Control and Responsibility ▸ Build for Replacement
  8. WHAT DOES THIS MEAN FOR OUR TOOLS

  9. HEAVY LIFTING IS DONE IN SERVICES ▸ Scheduling of Tasks

    ▸ Scaling of System ▸ Collecting of Log Data ▸ Authentication between Services ▸ Storing and managing data storage ▸ Provisioning and scaling Queue infrastructure ▸ …
  10. WHAT IS IMPORTANT, PAST VS PRESENT Past Features ‣ Replication

    ‣ Scalability ‣ …
  11. PAST SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER

    SERVER SERVER SERVER
  12. PAST SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER

    SERVER SERVER SERVER Application cluster Database cluster Data and ETL Pipeline
  13. PAST VS PRESENT Past Present Features ‣ Replication ‣ Scalability

    ‣ … Usability Accessibility Best Practices
  14. MAKE IT EASY TO COMBINE SERVICES Service A F1 F2

    F3 Service C F1 F2 Service B F1 Service E F1 F2 F3 Service F F1 F2 F3 Service D F1 F2 F3 F4 F5 S3 Kinesis DynamoDB Api Gateway
  15. BAD TOOLS WILL PUSH US AWAY FROM SERVICES ▸ Many

    examples in AWS from great services that aren’t used enough ▸ CloudFormation ▸ IAM ▸ CloudWatch Metrics, Logs and Alerts ▸ Custom Resources ▸ AWS Config
  16. SAME PROBLEM NO MATTER THE COMPANY SIZE

  17. UNREASONABLY COMPLEX TO START

  18. SO WHATS HOLDING US BACK

  19. CLOUDFORMATION IS THE MAKE OR BREAK SERVICE

  20. HORRIBLE CLI TOOLING FROM AWS ▸ Hard to learn and

    hard to read configuration ▸ No built-in modularity for one stack making config files huge ▸ There is no built-in command to deploy a stack, wait for it to be deployed and get the CF events as cli output
  21. CLOUDFORMATION IS OUR ASSEMBLY CODE

  22. ALL TOOLS SHOULD WORK TOWARDS CLOUDFORMATION ▸ Standard Endpoint allows

    us to swap out tools ▸ Not possible with other tools like Terraform as state is stored separately
  23. ALL TOOLS SHOULD WORK TOWARDS CLOUDFORMATION ▸ Standard Endpoint allows

    us to swap out tools ▸ Not possible with other tools like Terraform as state is stored separately ▸ Easier to support standards
  24. ALL TOOLS SHOULD WORK TOWARDS CLOUDFORMATION ▸ Standard Endpoint allows

    us to swap out tools ▸ Not possible with other tools like Terraform as state is stored separately ▸ Easier to support standards ▸ This also makes sure we’re building better low level tools with better UX
  25. SIMPLE TOOLS TO MAKE CF MORE USEFUL ▸ Awsie (https://github.com/flomotlik/awsie)

    awsie example-stack s3 ls s3://cf:DeploymentBucket: --region us-west-1 aws s3 ls s3://formica-example-stack-deploymentbucket-1jjzisylxreh9 --region us-west-1
 awsie example-stack —command “echo //cf:DeploymentBucket:” --region us-west-1 
 echo s3://formica-example-stack-deploymentbucket-1jjzisylxreh9
  26. WHAT WE NEED TO WATCH OUT FOR

  27. STOP BUILDING ABSTRACTIONS

  28. STOP BUILDING ABSTRACTIONS I need this small thing Hmm it

    could also do this Lets introduce this small abstraction to make it applicable here and here Wow this got really complex, new tool it is
  29. LETS JUST STOP THIS

  30. THERE IS NO ONE TOOL TO RULE THEM ALL

  31. PAST SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER

    SERVER SERVER SERVER Application cluster Database cluster Data and Background Pipeline
  32. MONOLITHIC TOOLS IN FRONT OF MICRO-SERVICE ARCHITECTURE? (KEVIN MCGRATH, AUTHOR

    OF CONDENSATION)
  33. MICRO-TOOLING (KEVIN MCGRATH, AUTHOR OF CONDENSATION)

  34. POWER AND CONTROL OVER YOUR INFRASTRUCTURE HARDWARE CLOUD INSTANCES PAAS

    SERVERLESS
  35. THE POWER IS IN COMBINING SERVICES, SO DON'T HIDE MORE

    HERE Service A F1 F2 F3 Service C F1 F2 Service B F1 Service E F1 F2 F3 Service F F1 F2 F3 Service D F1 F2 F3 F4 F5 S3 Kinesis DynamoDB Api Gateway
  36. CONSTANT NEW FEATURES ARE UNUSABLE DUE TO ABSTRACTIONS

  37. PROBLEMS WITH MULTI-PROVIDER TOOLING ▸ Always gets dragged down to

    lowest common denominator ▸ In a Serverless infrastructure you’re not going to move Everything between providers ▸ Small surface area per provider ▸ You want to get all you can out of that provider so you need specialised tooling
  38. WHAT CAN WE DO ABOUT IT

  39. LETS FIX CLOUDFORMATION

  40. LETS FIX CLOUDFORMATION NO, STOP IT

  41. IT NEEDS LOVE AND UX

  42. MY JOURNEY TO BUILD FORMICA ▸ https://github.com/flomotlik/formica ▸ Wanted to

    build and play with more AWS services so I needed a good CloudFormation tool ▸ Looked around at existing tools (Sceptre, Stacker, AWLess, Serverless, Gordon, Condensation, …) but all define their own abstractions ▸ Stack environments ▸ Dependencies between stacks ▸ Stack naming, ▸ Language to create the stack ▸ … ▸ So of course I built it using Troposphere so I could have nice python templates and added my own abstraction layer
  43. MAKEFILE IS MY DEPLOYMENT FRAMEWORK KAPIL THANGAVELU (@KAPILVT) FROM CAPITALONE

  44. THE UNIX PHILOSOPHY CLOUDFORMATION IS OUR PIPES

  45. THE UNIX PHILOSOPHY ALSO PIPES ARE OUR PIPES

  46. DOCUMENTAITON, NOT JUST TOOLING

  47. DOESN’T MEAN WE CAN’T IMPROVE CF SYNTAX ▸ Serverless/Gordon/Zappa are

    great for setting up an API Gateway with Lambdas ▸ We could have other tools as preprocessors for specific things ▸ Simple tool to translate HTTP endpoint definition to CF ▸ Tool to create CF templates for CloudWatch Alerts and Metrics
  48. SUITE OF SMALL FOCUSED TOOLS ▸ Built: ▸ Formica -

    Deploying CloudFormation templates ▸ AWSIE - Dynamically use CF ids in your shell commands ▸ Ideas: ▸ Orgonaut: Org/Account management and running commands against each command or group ▸ Auditor: Auditing and reporting setup with AWS Config and Cloudtrail across accounts ▸ Pipes: CI/CD setup through CodePipeline/CodeBuild ▸ Watchdog: CloudWatch config tool to make Monitoring and Alerts easier to configure ▸ Work together as a community to highlight those tools and define a best-practice suite
  49. THEY ARE SMALL THEY ARE SIMPLE GO BUILD SOME

  50. LEARN CLOUD FORMATION IN-DEPTH

  51. FLOMOTLIK.ME Florian Motlik AWS/Serverless/Development Process Consultant
 Need help in your

    infrastructure: flo@flomotlik.me T W I T T E R : @ F L O M O T L I K W E B S I T E : F L O M O T L I K . M E E M A I L : F L O @ F L O M O T L I K . M E
  52. CONCLUSIONS ▸ Composition and Accessibility, not Features and Abstractions ▸

    Build and use small tools that do one thing well ▸ Compose larger systems out of several smaller tools ▸ Go check out Make, its reasonably good ▸ Join the Serverless Slack (Serverless-Forum, https://wt-serverless- seattle.run.webtask.io/serverless-forum-signup?webtask_no_cache=1) T W I T T E R : @ F L O M O T L I K W E B S I T E : F L O M O T L I K . M E E M A I L : F L O @ F L O M O T L I K . M E