Save 37% off PRO during our Black Friday Sale! »

Architecting for the Cloud

Architecting for the Cloud

An introduction to Amazon Web Services for developers. Best practices for development/deployment on multi server setups.

2ae345c92b5115d9d026ee752702fa56?s=128

Leonidas Tsementzis

April 26, 2012
Tweet

Transcript

  1. ARCHITECTING for the CLOUD leonidas tsementzis aka @goldstein

  2. # get social # awsuggr

  3. leonidas tsementzis aka @goldstein # who’s talking * software architect,

    engineer [all major web/mobile platforms] * devOps [enthusiast, not a real sysadmin] * entrepreneur [n00b]
  4. # format * the problem * development * deployment *

    failure * limitations * conclusion * questions
  5. # the problem * increasing/decreasing resources on the fly using

    auto scaling * availability * performance * multi server painless deployment
  6. :development:

  7. # your stack matters * the single most important aspect

    * cloud-ready open source libraries for major platforms * saves you a lot of development time * rapid changes * can lock you in
  8. # memory * avoid application level variables/ sessions * centralized

    storage: ✔ fast ✔ scalable ✔ efficient Amazon DynamoDB
  9. :storage:

  10. # storage - single server server

  11. # storage - multi server server 1 server farm server

    2 server 3 server 4 - scripts - static files
  12. # storage - multi server - S3 server 1 server

    farm server 2 server 3 server 4 - scripts - static files
  13. # storage application /local/address/site \\unc\path\site S3 API STORAGE MIDDLEWARE

  14. # storage * local filesystem * network storage * Amazon

    S3 * Rackspace CloudFiles * database (BLOB) * GridFS (MongoDB) * FTP, SFTP * Azure using a pluggable storage middleware, we can create storages like...:
  15. # storage ...and hopefully we don’t have to:

  16. # storage * avoid using HEAD/GET requests to check for

    existing files * store file list in memory instead * use S3 “PRELOAD_METADATA” ...but if we have to:
  17. :task queuing:

  18. # task queuing * image resizes * external api calls

    * low priority updates * intensive calculations * big data queues * preparing hot caches * indexing updates * logging use message/task queues for long running operations:
  19. # task queuing * organize tasks into different queues *

    organize queues into priority workers * scale workers using AWS auto scaling - send custom alerts using AWS CloudWatch API * it’s all about priorities Amazon SQS
  20. :database:

  21. # database * Amazon RDS does the trick if you’re

    on MySQL or Oracle * shard early mark down table dependencies from the start, work around this while you grow
  22. :deployment:

  23. # huh? * it’s your code * you know the

    dependencies * you know it’s breaking points * it’s your job to deal with deployment failures * continuous deployment? yes please!
  24. # requirements * 50+ deployments per day from n devs

    * secure * fast rollbacks on failure * zero downtime * dependency handling (restart services, migrate dbs etc.)
  25. # continuous deployment repo dev dev dev dev git pull

    master git push/pull 0.0.0.1 server farm 0.0.0.2 0.0.0.3 0.0.0.4 $: fab production deploy
  26. # where the magic happens

  27. pull from master -> run test suite (abort on failure)

    -> deploy/compress static files on S3 -> install new dependencies -> run db migration scripts -> cleanup -> rollback if something fails -> clone previous production for backup -> backup live db -> pre-compile less etc -> restart services if required
  28. # continuous deployment * master is always production safe use

    pull request for large teams * bootstrapped pre-configured AMIs * handle stale servers with care assumptions: tools:
  29. :failure:

  30. # failure “Design for failure and nothing will fail” “Everything

    fails, all the time” ~ Amazon CTO
  31. # failure * backup/restore strategy * bootstrapped AMIs * multi-AZ

    deployment
  32. :limitations:

  33. # limitations * disk I/O ✔ use multiple EBS in

    RAID config * database ✔ sharding ✔ multiple read-only ✔ clustering * ram ✔ memcache/redis replication
  34. # recap * the problem * development * deployment *

    failure * limitations * conclusion * questions
  35. :one more thing:

  36. :vendor lock-in: if you’re still following, there’s no such thing

    on AWS
  37. # vendor lock-in * S3 ✔ pluggable storages * EC2

    ✔ normal unix box * DynamoDB ✔ fully compatible NoSQL * RDS ✔ fully compatible MySQL/Oracle
  38. :conclusion:

  39. # conclusion * use best practices and you’ll be safe

    * your stack matters * Cloud != high availability * Cloud != high performance * Cloud != magic (but it’s close)
  40. # questions? challenges? ? @goldstein aka leonidas tsementzis leotsem [at]

    gmail.com
  41. # thank you @goldstein aka leonidas tsementzis leotsem [at] gmail.com

    !