Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Calling to the future

Eloy Coto
September 17, 2016

Calling to the future

Talk given by Eloy Coto on Pycon-Galicia. September 2016.

Eloy Coto

September 17, 2016

More Decks by Eloy Coto

Other Decks in Programming


  1. Calling to the future

  2. @eloycoto 2008 2011 2013

  3. This talk is all about buzzwords!

  4. Buzzwords? ✓ Containers ✓ Microservices ✓ Serverless ✓ Schedulers ✓

    12 Factor Apps
  5. Typical web app on 2000 Apache Server PHP Mysql

  6. Web on 2005 Ngninx Python PGSQL Memcached

  7. Web on 2016 Ngninx Python PGSQL Redis Bigdata Schedulers CI

    CD Stats
  8. Devs need to recycle every day.

  9. ✓ Buzzwords are double-edged sword ✓ Sometimes these tools are

    useful, but not production ready ✓ New tools implied share that knowledge into the company
  10. Remember, in the end, nobody wins unless everybody wins! Bruce

    springsteen, Born to run
  11. ✓ Buzzwords are double-edged sword ✓ Sometimes these tools are

    useful, but not production ready ✓ New tools implied share that knowledge into the company ✓ Buzzwords were born to fix problems
  12. Dev overhead Ops overhead Monolithic Microservices ƛ functions Stateless Stateful

  13. 12 factor apps manifesto

  14. Codebase ✓ One codebase tracked in revision control ✓ If

    multiple codebases, it’s not an app. It’s a distributed system $ git init
  15. Dependencies ✓ Explicitly declare and isolate dependencies ✓ Do not

    rely on the implicit existence of any system tools $ echo "click==6.6" > req.txt $ pip install -r req.txt
  16. Config $ docker run -d -e "DB=postgresql://py:[email protected]:6432/py" eloycoto/py $ envdir

    envs/prod/ python manage.py runserver $ vault read secret/DBURL ✓ Config stored in the environment ✓ Config is the specific information required to a host deployment of your codebase ✓ Keep your config outside the app or git.
  17. Backing Services ✓ Treat backing services as attached resources. ✓

    Make no distinction between local and third party services
  18. Build, Release, Run ✓ Strictly separate build & run stages

    ✓ Impossible to make changes at runtime ✓ Build = codebase + dependencies + assets ✓ Release = Build + config ✓ Run = run process against release
  19. Codebase Build Config Release Run Packer/Docker Terraform Kubernetes Nomad

  20. Processes ✓ Is stateless ✓ Shares nothing with other processes

    ✓ Session on DB storage ✓ Asset pre-compilation
  21. Port Binding ✓ All services are accessible by one Port

    binding ✓ One app can be backing service of another app using port binding
  22. Concurrency ✓ Scale out via process model ✓ Assign work

    to a process type (web, DB, worker) ✓ Can then handle own multiplexing ✓ Processes don’t daemonize or write PID files
  23. web.0 web.1 worker.0 worker.1 worker.2 billing.0 alerts.0 Scalability workload diversity

  24. Disposability ✓ Cattle, no pets! ✓ Maximise robustness with fast

    startup and graceful shutdown ✓ Then can be started or stopped at any moment. ✓ Job queues FTW
  25. Dev/Prod Parity Dev = Staging = Prod sqlite ≠ postgres

    = postgres postgres = postgres = postgres ✓ Keep development, staging, and production as similar as possible. ✓ CI & deploy ASAP after writing code ✓ Get devs involved in Operations
  26. Logs Cloudwatch + awslogs ElasticSearch + packetbeat + kibana Grafana

    + Influxdb ✓ Treat logs as event streams ✓ Apps never concern themselves with routing or storage of the output stream ✓ Logs should be indexed and capable to analysis
  27. Admin Processes $ python manage.py migrate ✓ Run admin/management task

    as one-off process in an identical environment ✓ Admin code ships with app code
  28. Eloy, where is the future?

  29. AWS Lamda Run code without thinking about servers.

  30. What is AWS lambda? ✓ A compute service that acts

    on events or requests ✓ Scaling is automatic to event rate ✓ Only pay for what is used ✓ Billed in 100ms chunks
  31. AWS Lambda - Benefits Serverless Event-Driven Scale Subsecond billing €

    1 2 3
  32. Validate image Create thumbnail Lambda Use Case New profile image

  33. Lambda Use Case

  34. Lambda design .java .js .py Dependencies ZIP S3

  35. Deploy AWS Lambdas Boto Hard Easy ✓ Lambdas are great,

    but deploy/manage is hard ✓ Amazon does not provide any useful tool for that
  36. ✓ Tool to create and deploy lambdas ✓ Heavy use

    of Cloudformation ✓ Python/Javascript/Golang/Java runtimes ✓ Supported integrations - ApiGateway - Cloudwatch events and cron - Dynamodb and S3 events - Kinesis streams
  37. /settings.yml --- project: pycongalicia default-region: us-east-1 code-bucket: gordon-pycongalicia-39b948b8 apps: -

    gordon.contrib.lambdas - pyday apigateway: helloapi: description: My first API resources: /: methods: [GET, POST] integration: lambda: pyday.pyday /pyday/settings.yml --- lambdas: pyday: code: pyday handler: code.handler runtime: python Gordon design
  38. /pyday/pyday/code.py import json def handler(event, context): data = "Hello World!"

    print(data) print("Received Event: " + json.dumps(event, indent=2)) return data
  39. $ gordon build Loading project resources ✓ apigateway:helloapi Loading installed

    applications contrib_lambdas: ✓ lambdas:version pyday: ✓ lambdas:pyday Building project... 0001_p.json 0002_pr_r.json 0003_r.json
  40. $ gordon apply Applying project... 0001_p.json (cloudformation) CREATE_COMPLETE waiting... |

    0002_pr_r.json (custom) ✓ code/contrib_lambdas_version.zip (de19e97c) ✓ code/pyday_pyday.zip (c6f54de9) 0003_r.json (cloudformation) CREATE_COMPLETE waiting... | Project Outputs: LambdaPydayPyday arn:aws:lambda:us-east-1:105374107590:function:dev-pycongal icia-r-PydayPyday-M79TC8KW3O49:current ApigatewayHelloapi https://qootpywo86.execute-api.us-east-1.amazonaws.com/dev
  41. Success stories ✓ 30K page views for $0.21 Examples ✓

  42. Email: [email protected] Blog: http://acalustra.com Twitter: @eloycoto Graciñas!