Calling to the future

Talk given by Eloy Coto on Pycon-Galicia. September 2016.

Transcript

1. Calling to the future
2. @eloycoto 2008 2011 2013
3. This talk is all about buzzwords!
4. Buzzwords? ✓ Containers ✓ Microservices ✓ Serverless ✓ Schedulers ✓ 12 Factor Apps
5. Typical web app on 2000 Apache Server PHP Mysql
6. Web on 2005 Ngninx Python PGSQL Memcached
7. Web on 2016 Ngninx Python PGSQL Redis Bigdata Schedulers CI CD Stats
8. Devs need to recycle every day.
9. ✓ Buzzwords are double-edged sword ✓ Sometimes these tools are useful, but not production ready ✓ New tools implied share that knowledge into the company
10. Remember, in the end, nobody wins unless everybody wins! Bruce springsteen, Born to run
11. ✓ Buzzwords are double-edged sword ✓ Sometimes these tools are useful, but not production ready ✓ New tools implied share that knowledge into the company ✓ Buzzwords were born to fix problems
12. Dev overhead Ops overhead Monolithic Microservices ƛ functions Stateless Stateful
13. 12 factor apps manifesto
14. Codebase ✓ One codebase tracked in revision control ✓ If multiple codebases, it’s not an app. It’s a distributed system $ git init
15. Dependencies ✓ Explicitly declare and isolate dependencies ✓ Do not rely on the implicit existence of any system tools $ echo "click==6.6" > req.txt $ pip install -r req.txt
16. Config $ docker run -d -e "DB=postgresql://py:py@127.0.0.1:6432/py" eloycoto/py $ envdir envs/prod/ python manage.py runserver $ vault read secret/DBURL ✓ Config stored in the environment ✓ Config is the specific information required to a host deployment of your codebase ✓ Keep your config outside the app or git.
17. Backing Services ✓ Treat backing services as attached resources. ✓ Make no distinction between local and third party services
18. Build, Release, Run ✓ Strictly separate build & run stages ✓ Impossible to make changes at runtime ✓ Build = codebase + dependencies + assets ✓ Release = Build + config ✓ Run = run process against release
19. Codebase Build Config Release Run Packer/Docker Terraform Kubernetes Nomad
20. Processes ✓ Is stateless ✓ Shares nothing with other processes ✓ Session on DB storage ✓ Asset pre-compilation
21. Port Binding ✓ All services are accessible by one Port binding ✓ One app can be backing service of another app using port binding
22. Concurrency ✓ Scale out via process model ✓ Assign work to a process type (web, DB, worker) ✓ Can then handle own multiplexing ✓ Processes don’t daemonize or write PID files
23. web.0 web.1 worker.0 worker.1 worker.2 billing.0 alerts.0 Scalability workload diversity
24. Disposability ✓ Cattle, no pets! ✓ Maximise robustness with fast startup and graceful shutdown ✓ Then can be started or stopped at any moment. ✓ Job queues FTW
25. Dev/Prod Parity Dev = Staging = Prod sqlite ≠ postgres = postgres postgres = postgres = postgres ✓ Keep development, staging, and production as similar as possible. ✓ CI & deploy ASAP after writing code ✓ Get devs involved in Operations
26. Logs Cloudwatch + awslogs ElasticSearch + packetbeat + kibana Grafana + Influxdb ✓ Treat logs as event streams ✓ Apps never concern themselves with routing or storage of the output stream ✓ Logs should be indexed and capable to analysis
27. Admin Processes $ python manage.py migrate ✓ Run admin/management task as one-off process in an identical environment ✓ Admin code ships with app code
28. Eloy, where is the future?
29. AWS Lamda Run code without thinking about servers.
30. What is AWS lambda? ✓ A compute service that acts on events or requests ✓ Scaling is automatic to event rate ✓ Only pay for what is used ✓ Billed in 100ms chunks
31. AWS Lambda - Benefits Serverless Event-Driven Scale Subsecond billing € 1 2 3
32. Validate image Create thumbnail Lambda Use Case New profile image
33. Lambda Use Case
34. Lambda design .java .js .py Dependencies ZIP S3
35. Deploy AWS Lambdas Boto Hard Easy ✓ Lambdas are great, but deploy/manage is hard ✓ Amazon does not provide any useful tool for that
36. ✓ Tool to create and deploy lambdas ✓ Heavy use of Cloudformation ✓ Python/Javascript/Golang/Java runtimes ✓ Supported integrations - ApiGateway - Cloudwatch events and cron - Dynamodb and S3 events - Kinesis streams
37. /settings.yml --- project: pycongalicia default-region: us-east-1 code-bucket: gordon-pycongalicia-39b948b8 apps: - gordon.contrib.lambdas - pyday apigateway: helloapi: description: My first API resources: /: methods: [GET, POST] integration: lambda: pyday.pyday /pyday/settings.yml --- lambdas: pyday: code: pyday handler: code.handler runtime: python Gordon design
38. /pyday/pyday/code.py import json def handler(event, context): data = "Hello World!" print(data) print("Received Event: " + json.dumps(event, indent=2)) return data
39. $ gordon build Loading project resources ✓ apigateway:helloapi Loading installed applications contrib_lambdas: ✓ lambdas:version pyday: ✓ lambdas:pyday Building project... 0001_p.json 0002_pr_r.json 0003_r.json
40. $ gordon apply Applying project... 0001_p.json (cloudformation) CREATE_COMPLETE waiting... | 0002_pr_r.json (custom) ✓ code/contrib_lambdas_version.zip (de19e97c) ✓ code/pyday_pyday.zip (c6f54de9) 0003_r.json (cloudformation) CREATE_COMPLETE waiting... | Project Outputs: LambdaPydayPyday arn:aws:lambda:us-east-1:105374107590:function:dev-pycongal icia-r-PydayPyday-M79TC8KW3O49:current ApigatewayHelloapi https://qootpywo86.execute-api.us-east-1.amazonaws.com/dev
41. Success stories ✓ 30K page views for $0.21 Examples ✓ github.com/jorgebastida/gordon/examples
42. Email: eloy.coto@acalustra.com Blog: http://acalustra.com Twitter: @eloycoto Graciñas!