I get by with a little help from my friends (Hashidays AMS Day 2 Keynote 2018)

Transcript

1. I get by with a little help from my friends Abby Fuller, @abbyfuller Developer Relations, AWS AMSTERDAM AMSTERDAM
2. Hi, I’m Abby. I work in developer relations for AWS, and I talk about containers a lot.
3. Containers are great, but with them comes a fair amount of work: scheduling, resource usage, service discovery, deployments, monitoring, observability, logging…
5. Even when it is *literally* your job to keep up with the container space, it’s still overwhelming!
6. It seems like every day, there’s a new project …Or a blog …Or a new tool …Or a talk
7. So how are you supposed to keep up?
9. You can get by with a little help from your friends!
10. I mean this two (2) ways: 1. Through your tooling and processes (tools are friends!) 2. And also through the community (person friends!)
11. Let’s talk tools. In other words, how can you work smarter, not harder?
12. Safety first!
14. If you’re working with AWS, this starts with roles In other words, how you can, and your services perform actions securely? All types of IAM roles: • Service roles • Controls what the service can access on your behalf( for example, access to DynamoDB) • User roles • Represent a person, and what they can access. Forexample,a role “abby” with administrator access • Groups • logical groupings of IAMUsers with shared permissions
15. Sssh, I have secrets. Storing secret keys and passwords is worth a whole talk on its own. A few words of advice: • Rotate your keys, don’t share them between a) services, b) humans • Go for “principle of least access” • Where possible, store your keys somewhere else, and then control access (for example, store keys in KMS, and control access with IAM/service roles) • There are also third party tools out there (lots): for example, Vault
16. There are also tools to help with the security of your containers themselves Lean on tools to check your images for vulnerabilities. A couple of options, both paid and open source: • Aqua MicroScanner (community edition) • Aqua continuous image assurance • Docker Security Scan with Trusted Registry • Clair from CoreOS
18. Now that we’re thinking about safety, let’s go from the bottom up.
19. Good development starts locally • Standardized, easy to set up local environments help eliminate “well it worked on my machine” • Some common approaches to this: Vagrant and/or VirtualBox, EC2 box with local access • Any approach should have a few things in common: shared configuration between team members (i.e., running the environment uses the same versions of packages), and is as close as possible to the production environment.
20. Some tools that might help you out • Vagrant • AWS Cloud9 • Cloud development environment that integrates a shell, AWS resources, and an editor • If you’re using Kubernetes, you can run it locally for development purposes with MiniKube
21. From local environment to production, you probably interact with your containers through CLIs. Your CLI depends on your orchestration platform. We’ll look at two: Elastic Container Service (ECS) from Amazon, and Kubernetes.
22. CLIs for ECS (and Fargate) • aws-cli: the official OG. Open source, includes most AWS services. • ecs-cli: also official, but just for ECS. Supports docker compose files. Some good unofficial options: • Fargate CLI • Coldbrew CLI
23. (A few of the) CLIs for Kubernetes • kubectl: run commands against Kubernetes clusters. • kops: kubectl for clusters. Can use to manage production Kubernetes infrastructure in AWS. • kubeadm: bootstrap production-ready Kubernetes clusters, following best practices. Not production ready, but cool: • kubicorn: helps users manage their Kubernetes infrastructure
24. So you’re interacting with your clusters with CLIs- now what?
25. Automate!
26. Say it with me: no artisanal containers • Infrastructure as code: automate and template everything from your containers to your infrastructure. This makes everything repeatable and portable! • Containers can be built directly from repositories as part of your CI/CD process. • After that, automate your AWS!
27. A couple options for AWS automation Cloudformation: • Common language for describing, creating, and deleting AWS resources. • Created and maintained by Amazon. • Free to use, just pay for the resources created. Don’t like writing Cloudformation templates? There are samples and snippets here.
28. Looks something like this!
29. Terraform: • Created by Hashicorp (thanks y’all!) • Open source Either one works! Both focus on resource management. Cloudformation: planning and execution in one step. Meaning, just executes. Terraform: planning and execution in two steps (terraform plan, terraform apply) A less JSON-y option: Terraform!
30. Here’s a Terraform example
31. You’re all automated! You’re using CLIs! You thought about security! How do you know if everything is working?
32. Monitoring! Observability! These help you a) debug, and b) understand what’s happening in your system
33. Beyond just logs, you want actual, useable, helpful information about what’s happening with your systems.
34. If it doesn’t make sense to you at 3am, it’s probably not helping much: a memoir.
35. Just a few things to think about: • Reduce the noise • Page only on issues that require immediate, emergency attention • You need all of it- both monitoring AND observability (more than just logs!) • Make sure you’re asking and answering the right questions.
36. You don’t have to go it alone! Lots of tools out there to help you make sense of what’s going on
37. Here’s a caveat though: you have to use the RIGHT tool for the job.
38. Don’t do everything yourself when you don’t have to, but don’t be afraid to get fancy on your own if there isn’t a tool that works for you!
39. This might be things like tweaking your load balancer and autoscaling group settings for faster deployments
41. Or bringing a custom AMI to AWS
42. The easiest way to use ECS is with the ECS-optimized AMI, but life is too short to not have custom AMIs. A few steps to registering your own AMI with ECS: 0. Make sure your AMI has the right requirements! 1. Install ecs-agent 2. Install Docker daemon (if it’s not already installed) 3. Register instance with cluster Most of these options can be set in EC2 user-data (more on that in a sec)
43. With Amazon Linux, in /etc/ecs/ecs.config: ECS_ENABLE_TASK_IAM_ROLE=true ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true ECS_LOGFILE=/log/ecs-agent.log ECS_AVAILABLE_LOGGING_DRIVERS=["json- file","awslogs"] ECS_LOGLEVEL=info ECS_CLUSTER=default Full instructions here.
44. Or resorting to configuring everything with Bash in EC2 user-data
45. User-data is good for everything from starting services, to configuring your environment, to enabling options and flags that aren’t supported in the UI. Scripts run at boot, and you can modify them through the EC2 Console.
46. For example, I can pass in Docker flags that are unsupported by the AWS UI: $ echo ”WHATEVER_FLAG=hi” >> /etc/sysconfig/docker Here be dragons. And cruft.
48. OK, so remember when I said there were two ways you could get some help? Here’s #2
49. Your best resource is the community.
50. Kubernetes, AWS, Hashicorp services and tons of open source projects all have vibrant communities and resources here to help you.
52. Talk to peers, and colleagues, and internet friends. Write a blog, or give a talk. Come to conferences (just like this one).
54. If you don’t know how to get started, or you’re struggling: ask for help! I guarantee that someone has been in the same situation before.
55. Some resources to get you started Find a Kubernetes meetup Join the TGIK community call with Kris Nova and Joe Beda! Or a SIG. Try an AWS User Group Or try one of the approximately 12957329057 Slack channels out there for a community that works for you!
56. Thank you! @abbyfuller abbyfull@amazon.com AMSTERDAM