Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scripting creating AWS based servers

Scripting creating AWS based servers

in the most Taco Bell way I know



January 27, 2014


  1. Scripting your servers @coldclimate CANDDi.com

  2. Put your pens down @coldclimate CANDDi.com

  3. There once was a man... @coldclimate CANDDi.com

  4. And then there was a document... @coldclimate CANDDi.com

  5. Which became a wiki... @coldclimate CANDDi.com

  6. Which finally because a script @coldclimate CANDDi.com

  7. Which mostly worked... @coldclimate CANDDi.com

  8. And then the madness had to stop @coldclimate CANDDi.com

  9. You wouldn’t treat code like this (hopefully) @coldclimate CANDDi.com

  10. Man => Document => Wiki => Script => Images @coldclimate

  11. Encoding Version controlling (automating) @coldclimate CANDDi.com

  12. Man => Document Removing the bus factor “The man we

    can’t fire” @coldclimate CANDDi.com
  13. Document => Wiki The fucker *might* stay up to date

    @coldclimate CANDDi.com
  14. Wiki => Script Because script in documentation is stupid @coldclimate

  15. @coldclimate CANDDi.com

  16. Why are you not talking about chef, puppet, ansible etc?

    @coldclimate CANDDi.com
  17. http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html @coldclimate CANDDi.com

  18. Build once or build twice? @coldclimate CANDDi.com

  19. Build once Raw OS Base Instance Script Working Instance @coldclimate

  20. Kick-off instance Install packages Configure them Install your software Hope

    it all works @coldclimate CANDDi.com
  21. Build Twice Raw OS Base Instance Script “Base Box” Make

    AMI “Base Box” “Instance Box” Script Base AMI @coldclimate CANDDi.com
  22. Build once or build twice? @coldclimate CANDDi.com

  23. Kick-off instance Install packages -------------------------------------------- Configure them Install your software

    @coldclimate CANDDi.com
  24. Kick-off instance Install packages -------------------------------------------- Configure them Install your software

    Changes rarely Changes frequently
  25. Kick-off instance Install packages -------------------------------------------- Configure them Install your software

    AMI Instance
  26. Stability @coldclimate CANDDi.com

  27. Speed @coldclimate CANDDi.com

  28. The game: always runs, no manual fixes @coldclimate CANDDi.com

  29. A script to built an AMI https://gist.github.com/coldclimate/5ff56c96dac116f3234e @coldclimate CANDDi.com

  30. ec2-run-instances --region 'eu-west-1' -g TEMP -k TEMP -f install.sh -t

    t1.micro ami-ce7b6fba @coldclimate CANDDi.com
  31. install.sh limited size (16kb) runs as root cock to debug

    @coldclimate CANDDi.com
  32. /var/log/cloud-init.log @coldclimate CANDDi.com

  33. set -ex @coldclimate CANDDi.com

  34. Add any sources Update the APT repository Install core packages

    @coldclimate CANDDi.com
  35. Add any sources @coldclimate CANDDi.com

  36. echo "deb http://archive.canonical.com/ precise partner" | sudo tee -a /etc/apt/sources.list

    @coldclimate CANDDi.com
  37. python-setuptools add-apt-repository -y ppa:ondrej/php5 @coldclimate CANDDi.com

  38. apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10 echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist

    10gen' | sudo tee /etc/apt/sources.list.d/10gen.list @coldclimate CANDDi.com
  39. Update the APT repository @coldclimate CANDDi.com

  40. apt-get update @coldclimate CANDDi.com

  41. Install core packages @coldclimate CANDDi.com

  42. apt-get -y install zip unzip xmlstarlet sysstat iftop ntp sysvbanner

    build- essential python-setuptools nginx nodejs @coldclimate CANDDi.com
  43. easy_install -U boto @coldclimate CANDDi.com

  44. wget https://github.com/nicolasff/phpredis/archive/2.2.4.zip -P /tmp/ mkdir -p /opt/code unzip /tmp/2.2.4.zip -d

    /opt/code/ cd /opt/code/phpredis-2.2.4 phpize ./configure make make install @coldclimate CANDDi.com
  45. banner “*** DONE BABY ***” @coldclimate CANDDi.com

  46. ec2-create-image -n<ami-name> instance_id @coldclimate CANDDi.com

  47. Making running instance @coldclimate CANDDi.com

  48. ec2-run-instances --region 'eu-west-1' -g sg-bb18b66c -k LIVE-PEM-KEY -p Instance_Profile -f

    install.sh -t m1.small --availability-zone eu-west-1a --disable-api-termination ami-XXXXXX @coldclimate CANDDi.com
  49. Roles http://docs.aws.amazon.com/cli/latest/reference/iam/create-role.html Instance Profiles http://docs.aws.amazon.com/cli/latest/reference/iam/create-instance-profile.html Adding a Role to an

    Instance Profile http://docs.aws.amazon.com/cli/latest/reference/iam/add-role-to-instance-profile.html @coldclimate CANDDi.com
  50. Download from S3 Setup directories Setup config Install code Start

    it up Register it @coldclimate CANDDi.com
  51. Download from S3 python <<PY import boto import sys from

    boto.s3.key import Key def get_wonka(bucket_name, file_name, destination_file): conn = boto.connect_s3() bucket = conn.get_bucket(bucket_name) k = Key(bucket) k.name = file_name destination = "/tmp/%s" % (destination_file) k.get_contents_to_filename(destination) get_wonka("BUCKET","SOURCEFILE","DESTINATION") PY @coldclimate CANDDi.com
  52. Setup directories Your script is running as root so remember

    to chown @coldclimate CANDDi.com
  53. Setup config Copy it from the download into place Restart

    the service @coldclimate CANDDi.com
  54. Install code https://github.com/jordansissel/fpm @coldclimate CANDDi.com

  55. Start it up Supervisord Ubuntu service Avoid own scripts @coldclimate

  56. Register it @coldclimate CANDDi.com

  57. Monitoring (zabbix) @coldclimate CANDDi.com

  58. Load Balancer (ec2 call) @coldclimate CANDDi.com

  59. Logging (papertrail) @coldclimate CANDDi.com

  60. DNS (R53 call) @coldclimate CANDDi.com

  61. And it works! @coldclimate CANDDi.com

  62. @coldclimate CANDDi.com

  63. Need to change the code or config? @coldclimate CANDDi.com

  64. JFDI and roll a fresh one @coldclimate CANDDi.com

  65. You need to change the base AMI @coldclimate CANDDi.com

  66. It’s simple, it just doesn’t feel it @coldclimate CANDDi.com

  67. Update the scripts until you can “launch clean” @coldclimate CANDDi.com

  68. Lastly @coldclimate CANDDi.com

  69. @coldclimate CANDDi.com

  70. speakerdeck. com/coldclimate/scripting-creating- aws-based-servers @coldclimate canddi.com/blog @coldclimate CANDDi.com