#VJMEJOHJOGSBTUSVDUVSF PO"84XJUI3VCZ !#SJTUPM$MPVE/BUJWF.FFUVQ

• Takayuki Watanabe • takanabe • @takanabe_w • Cookpad Inc. • Site Reliability Engineer 8IP

What is Cookpad ?

No content

https://cookpad.com Cookpad is the largest recipe sharing service in the world

https://cookpad.com Originally a recipe site for Japan

(MPCBMαʔϏεͷઆ໌

(MPCBMαʔϏεͷઆ໌ We are currently expanding regions

(MPCBMαʔϏεͷઆ໌ 21 Languages 67 Countries https://cookpad.com/us https://cookpad.com/id ɾ ɾ ɾ

(MPCBMαʔϏεͷઆ໌ Of course …

(MPCBMαʔϏεͷઆ໌ We have a site for the United Kingdom !! https://cookpad.com/uk

8BMLUIPVHIUIJTUBML • My experiences and opinions about organization growth • How we manage our infrastructure in Ruby DSL

Whobuilds infrastructures? Organization Story

*ODSFBTJOHPG&OHJOFFST ZFBS PGFOHJOFFST ɾ 'FXFOHJOFFSTBSFJO+BQBO ɾ PGVTFSTBSFTNBMM ɾ "SPVOEFOHJOFFST ɾ PGFOHJOFFSTBSFPVUTJEFPG+BQBO ɾ 5IFOVNCFSPGUFBNTBOETFSWJDFTJODSFBTF ɾ 6TFSTJODSFBTF ɾ "SPVOEFOHJOFFST ɾ .PTUPGFOHJOFFSTBSFJO+BQBO *KPJOFE$PPLQBE

*ODSFBTJOHPG&OHJOFFST ZFBS PGFOHJOFFST ɾ "SPVOEFOHJOFFST ɾ PGFOHJOFFSTBSFPVUTJEFPG+BQBO ɾ 5IFOVNCFSPGUFBNTBOETFSWJDFTJODSFBTF ɾ 6TFSTJODSFBTF ɾ "SPVOEFOHJOFFST ɾ .PTUPGFOHJOFFSTBSFJO+BQBO TUBHF TUBHF *KPJOFE$PPLQBE ɾ 'FXFOHJOFFSTBSFJO+BQBO ɾ PGVTFSTBSFTNBMM TUBHF

Selection of infrastructures and development approaches

4FMFDUJPOPGJOGSBTUSVDUVSFTBOEBQQSPBDIFT • Which platform do we use for production environment? • PaaS, IaaS, on premise or etc … • Which development approaches(software architecture)? • monolithic architecture, SOA, microservices or etc…

1MBUGPSNBQQSPBDIFTGPSTUBHF • Managed PaaS like Heroku • Enables developers to focus on development of service • We don’t need infra engineers • Monolithic Architecture • Communication is quite easy and comfortable

1MBUGPSNBQQSPBDIFTGPSTUBHF • Cloud Service Providers like AWS, GCP, Azure • Use Virtual Machines and managed service for production environment • Enable us to build more flexible infrastructures • Users gradually increase and scalabilities of capacity are important • Monolithic Architecture • Communication is still easy and comfortable

1MBUGPSNBQQSPBDIFTGPSTUBHF • Cloud Service Providers like AWS, GCP, Azure • Use Containers and managed service for production environment • stage 2 is a warmup period to move stage3 • Enable us to build more flexible infrastructures • Scalabilities of capacity are important • Microservice Architecture • Communication costs are high due to the number of engineers • Separation of authorities and responsibilities are necessary to scale out an organization

infrastructures for Cookpad

"84 Most servers for our services exist on AWS

3FBTPOTGPSVTJOH"84 • We use AWS since its early stage • AWS resources can be controlled via APIs • Pioneering cloud provider in this context • We use management tools written in Ruby for AWS • Declare AWS resources in Ruby DSL

8IZXFVTF3VCZ

SFGIUUQTTQFBLFSEFDLDPNB@NBUTVEBUIFSFDJQFGPSUIFXPSMETMBSHFTUSBJMTNPOPMJUI

SFGIUUQTTQFBLFSEFDLDPNB@NBUTVEBUIFSFDJQFGPSUIFXPSMETMBSHFTUSBJMTNPOPMJUI Most of our products are implemented in Ruby

SFGIUUQTTQFBLFSEFDLDPNB@NBUTVEBUIFSFDJQFGPSUIFXPSMETMBSHFTUSBJMTNPOPMJUI Site Reliability Engineers also use primarily in Ruby

Tools for our infrastructures

5PPMTGPSPVSJOGSBTUSVDUVSFT • AWS resource management • Other server resource management • Database management tools • CDN management tools • Server configuration management (provisioning) • Deployment tools

"84SFTPVSDFNBOBHFNFOU We codify AWS resources in Ruby DSL • Change history can be investigated from VCS(git, svn) • Idempotent • Current conditions of AWS resources should be synced to our codes • Don’t allow manual configuration changes to avoid chaos • If codes don’t have manual changes, they will be forcibly erased • Learning costs are low • Non-SRE engineer also can create PRs • Most of tools have: • dry-run feature to confirm changes before applying them • export feature to reflect current AWS condition to Ruby DSL

"84SFTPVSDFNBOBHFNFOU • We ride on codenize.tools (https://codenize.tools/) • mainly maintained by one of our SRE • terraform was not ready when we started to use AWS • stable enough • easy to use

"84SFTPVSDFNBOBHFNFOU Without any change tracking measures for the following resources, it tends to be linked to high operational costs • Route53 • Route Tables for Virtual Private Cloud • Identity and Access Management • Security Group • Elastic IP Addresses • Elastic Load Balancer • S3 Bucket Policy • CloudWatch Logs & Alarms

3PVUF • DNS service of AWS • Use Roadworker (https://github.com/codenize-tools/ roadworker) to define states of Route53 using Ruby DSL

hosted_zone "example.com." do rrset "example.com.", "A" do ttl 300 resource_records( "127.0.0.1", "127.0.0.2" ) end end 3PVUF (e.g) Declaration of “example.com” A record to Route53

71$3PVUF5BCMFT • Set of rules to determine where network traffic is directed • Use Mappru (https://github.com/codenize-tools/mappru) to define states of VPC Route Tables using Ruby DSL

vpc "vpc-12345678" do route_table "foo-rt" do subnets "subnet-12345678" route destination_cidr_block: "0.0.0.0/0", gateway_id: "igw-12345678" route destination_cidr_block: “192.168.100.101/32", network_interface_id: "eni-12345678" end route_table "bar-rt" do subnets "subnet-87654321" route destination_cidr_block: "192.168.100.102/32", network_interface_id: "eni-87654321" end # Undefined Route Table will be ignored end 71$3PVUF5BCMFT (e.g) Declaration of Route Tables for vpc-12345678

4FDVSJUZ(SPVQT • Security Groups is a virtual firewall that controls the traffic for one or more instances • Use Piculet (https://github.com/codenize-tools/piculet) to define states of Route53 using Ruby DSL

4FDVSJUZ(SPVQT ec2 "vpc-XXXXXXXX" do security_group "default" do description "default VPC security group" tags( "key1" => "value1", "key2" => "value2" ) ingress do permission :tcp, 22..22 do ip_ranges( "0.0.0.0/0", ) end permission :tcp, 80..80 do ip_ranges( "0.0.0.0/0" ) end permission :udp, 60000..61000 do ip_ranges( "0.0.0.0/0" ) end # ESP (IP Protocol number: 50) permission :"50" do ip_ranges( "0.0.0.0/0" ) end permission :any do groups( "any_other_group", "default" ) end end # Continue to the right codes # Continue from the left codes egress do permission :any do ip_ranges( "0.0.0.0/0" ) end end end security_group "any_other_group" do description "any_other_group" tags( "key1" => "value1", "key2" => "value2" ) egress do permission :any do ip_ranges( "0.0.0.0/0" ) end end end end (e.g) Declaration of Security Groups for vpc-XXXXXXXX

&MBTUJD*1"EESFTTFT • An Elastic IP address is a static IPv4 address designed for dynamic cloud computing • Use Eipmap (https://github.com/codenize-tools/eipmap) to define states of Elastic IP Addresses using Ruby DSL

domain "standard" do ip "54.256.256.1" ip "54.256.256.2", :instance_id=>"i-12345678" end domain "vpc" do ip "54.256.256.11", :network_interface_id=>"eni-12345678", :private_ip_address=>"10.0.1.1" ip "54.256.256.12", :network_interface_id=>"eni-12345678", :private_ip_address=>"10.0.1.2" ip "54.256.256.13" end &MBTUJD*1"EESFTTFT • (e.g) Declaration of Elastic IP Addresses

*EFOUJUZBOE"DDFTT.BOBHFNFOU • AWS Identity and Access Management is a web service that helps us securely control access to AWS resources • Use Miam (https://github.com/codenize-tools/miam) to define states of Elastic IP Addresses using Ruby DSL

user "takayuki-watanabe", :path=>"/infra/" do login_profile :password_reset_required=>false groups( "Admin" ) end group "Admin", :path => "/admin/" do policy "Admin" do {"Statement"=>[{"Effect"=>"Allow", "Action"=>"*", "Resource"=>"*"}]} end end *EFOUJUZBOE"DDFTT.BOBHFNFOU

&MBTUJD-PBE#BMBODJOH • Elastic Load Balancing distributes incoming application traffic across multiple targets, such as Amazon EC2 instances, containers, and IP addresses • Use Kelbim (https://github.com/codenize-tools/kelbim) to define states of Elastic IP Addresses using Ruby DSL

ec2 "vpc-XXXXXXXXX" do load_balancer "my-load-balancer", :internal => true do instances( "nyar", "yog" ) # or `any_instances` listeners do listener [:tcp, 80] => [:tcp, 80] listener [:https, 443] => [:http, 80] do app_cookie_stickiness "CookieName"=>"20" ssl_negotiation ["Protocol-TLSv1", "Protocol-SSLv3", "AES256-SHA", ...] server_certificate "my-cert" end end health_check do target "TCP:80" timeout 5 interval 30 healthy_threshold 10 unhealthy_threshold 2 end attributes do access_log :enabled => true, :s3_bucket_name => "any_bucket", :s3_bucket_prefix => nil, :emit_interval => 60 cross_zone_load_balancing :enabled => true connection_draining :enabled => false, :timeout => 300 end subnets( "subnet-XXXXXXXX" ) security_groups( "default" ) end end &MBTUJD-PBE#BMBODJOH • (e.g) Declaration of Elastic Load Balancing for vpc-XXXXXXXX

4#VDLFU1PMJDZ • Bucket policy and user policy are two of the access policy options available for you to grant permission to your Amazon S3 resources • Use Bukelatta (https://github.com/codenize-tools/kelbim) to define states of Elastic IP Addresses using Ruby DSL

bucket "foo-bucket" do { "Version"=>"2012-10-17", "Id"=>"AWSConsole-AccessLogs-Policy-XXX", "Statement"=> [ { "Sid"=>"AWSConsoleStmt-XXX", "Effect"=>"Allow", "Principal"=>{"AWS"=>"arn:aws:iam::XXX:root"}, "Action"=>"s3:PutObject", "Resource"=> "arn:aws:s3:::foo-bucket/AWSLogs/XXX/*" } ] } end 4#VDLFU1PMJDZ • (e.g) Declaration of S3 Bucket Policy for foo-bucket

$MPVE8BUDI-PHT"MBSN • Amazon CloudWatch offers cloud monitoring services for customers of AWS resources • Use Meteorlog (https://github.com/codenize-tools/ meteorlog) to define states of CloudWatch Logs using Ruby DSL • Use Radiosonde (https://github.com/codenize-tools/ radiosonde) to define states of CloudWatch Alarms using Ruby DSL

log_group "/var/log/messages" do log_stream "my-stream" metric_filter "MyAppAccessCount" do metric :name=>"EventCount", :namespace=>"YourNamespace", :value=>"1" end metric_filter "MyAppAccessCount2" do filter_pattern '[ip, user, username, timestamp, request, status_code, bytes > 1000]' metric :name=>"EventCount2", :namespace=>"YourNamespace2", :value=>"2" end end log_group "/var/log/maillog" do log_stream "my-stream2" metric_filter "MyAppAccessCount" do filter_pattern '[..., status_code, bytes]' metric :name=>"EventCount3", :namespace=>"YourNamespace", :value=>"1" end metric_filter "MyAppAccessCount2" do filter_pattern '[ip, user, username, timestamp, request = *html*, status_code = 4*, bytes]' metric :name=>"EventCount4", :namespace=>"YourNamespace2", :value=>"2" end end $MPVE8BUDI-PHT • (e.g) Declaration of CloudWatch Logs streams

alarm "alarm1" do namespace "AWS/EC2" metric_name "CPUUtilization" dimensions "InstanceId"=>"i-XXXXXXXX" period 300 statistic :average threshold ">=", 50.0 evaluation_periods 1 actions_enabled true alarm_actions [] ok_actions [] insufficient_data_actions ["arn:aws:sns:us-east-1:123456789012:my_topic"] end alarm "alarm2" do ... end $MPVE8BUDI"MBSNT • (e.g) Declaration of CloudWatch Alarms

Database management

.Z42-QSJWJMFHFT • The privileges granted to a MySQL account determine which operations the account can perform • Use Gratan (https://github.com/codenize-tools/gratan) to define states of MySQL access privileges using Ruby DSL

user "bob", "%" do on "*.*" do grant "USAGE" end on "test.*", expired: '2014/10/08', identified: "PASSWORD '*ABCDEF'" do grant "SELECT" grant "INSERT" end on /^foo\.prefix_/ do grant "SELECT" grant "INSERT" end end user "bob", ["localhost", "192.168.%"], expired: '2014/10/10' do on "*.*", with: 'GRANT OPTION' do grant "ALL PRIVILEGES" end end .Z42-QSJWJMFHFT • (e.g) Declaration of MySQL privileges

0OMJOFTDIFNBNJHSBUJPO • pt-online-schema-change (https://www.percona.com/doc/ percona-toolkit/3.0/pt-online-schema-change.html) performs online, non-blocking schema changes to a table • Use Departure (https://github.com/departurerb/departure) without needing to use a different DSL other than Rails' migrations DSL (under trial) • Departure uses pt-online-schema-change command-line tool of Percona Toolkit which runs MySQL alter table statements without downtime

CDN management

'BTUMZ • We use Fastly as our CDN • Use codily (https://github.com/sorah/codily) to define states of Fastly using Ruby DSL

service "foo" do response_object "method not allowed" do status "405" response "Method Not Allowed" content "405" content_type "text/plain" request_condition "request method is not GET, HEAD or FASTLYPURGE" do priority 10 statement '!(req.request == "GET" || req.request == "HEAD" || req.request == "FASTLYPURGE")' end end end # equals as follows: service "foo" do condition "request method is not GET, HEAD or FASTLYPURGE" do priority 10 statement '!(req.request == "GET" || req.request == "HEAD" || req.request == "FASTLYPURGE")' type "REQUEST" end response_object "method not allowed" do status "405" response "Method Not Allowed" content "405" content_type "text/plain" request_condition "request method is not GET, HEAD or FASTLYPURGE" end end 'BTUMZ • (e.g) Declaration of Fastly configurations

Server Configuration Management

4FSWFSDPOpHVSBUJPOT • Around a thousand EC2 instances are running on AWS • We used puppet previously as our configuration management • We want to use light tools like Ansible but also want to use Ruby DSL

IUUQTHJUIVCDPNJUBNBFLJUDIFOJUBNBF

*UBNBF • Configuration management tool inspired by Chef • An itamae (൘લ) is a cook in a Japanese kitchen • Chef-like Ruby DSL (but not compatible with Chef) • Simpler and lighter weight than Chef • Only recipes • Apply recipes to a local machine • Apply recipes to a remote machine over ssh • Idempotent

*UBNBF • (e.g) A sample recipe for nginx package 'nginx' do action :install end service 'nginx' do action [:enable, :start] end template "/path/to/dest" do action :create source "template.erb" variables(message: "World") end # template.erb Hello, <%= @message %>

Deployment tools

$BQJTUSBOP • Deploy Rails applications via Capistrano3 • Use Capistrano::BundleRsync (https://github.com/ sonots/capistrano-bundle_rsync) • Chat bot can invoke the deploy jobs via a deploy server

1SPCMFNTGPSUIFTFUPPMT • Tools explained in the previous slides work quite well for stage2 and a personal usage even though limited SRE can apply them to production environments • But if only SRE has privileges to use these tools, there might be problems in development scalabilities at stage3

1SPCMFNTGPSUIFTFUPPMT FYBNQMFT • Developers cannot update environment variables • SREs deploy them via Itamae • Developers cannot install software by themselves • SREs install them via Itamae • Developers cannot use new AWS resources soon • SREs deploy them via Codenize tools • SREs and Developers cannot work productively • frequent ops work might be requested to SREs and it becomes bottleneck of developments • Some part of authorities and responsibilities should be given them

What about containers?

Amazon ECS

%PDLFSDPOUBJOFSTPO&$4 • Use Amazon ECS • ECS allows us to easily run and manage Docker-enabled applications across a cluster of EC2 instances. • Use hako (https://github.com/eagletmt/hako) to deploy Docker containers onto ECS clusters • Some applications will use this container environment

%FQMPZNFOUqPXXJUI)BLP • Deploy containers via hako to ECS clusters and inject necessary data • Docker images are stored in ECR • Credentials are stored in Vault • Container app definitions are managed in yaml

%FQMPZNFOUqPXXJUI)BLP • (e.g) A sample hako app definition file scheduler: type: ecs region: ap-northeast-1 cluster: eagletmt desired_count: 2 task_role_arn: arn:aws:iam::012345678901:role/Hello deployment_configuration: maximum_percent: 200 minimum_healthy_percent: 50 app: image: ryotarai/hello-sinatra memory: 128 cpu: 256 links: - redis:redis env: $providers: - type: file path: hello.env PORT: '3000' MESSAGE: '#{username}-san' # Continue to the right codes # Continue from the right codes additional_containers: front: image_tag: hako-nginx memory: 32 cpu: 32 redis: image_tag: redis:3.0 cpu: 64 memory: 512 scripts: - <<: !include front.yml backend_port: 3000

%FQMPZNFOUGSPN4MBDL • Invoke deploy jobs defined on Rundeck via a chat bot • Use ruboty (https://github.com/r7kamura/ruboty) for chatops on Slack

*OUSPEVDUJPOPGDPOUBJOFSFOWJSPONFOUT • Developers can update environment variables • hako app yaml has environment variables for each application • Developers can install necessary software by themselves • Docker images include all software for each application • Developers can use new AWS resources soon • Many AWS resources are ready for use after deploying containers to our ECS clusters • SRE and Developers will become productively • Authority and responsibilities are given them

Conclusions

)VNCMFPQJOJPOT • Organization becomes big suddenly • Traditional development styles might not work suddenly and need to change them • There are technologies to support us and give more scalable environments • (e.g) Virtual Machine → Containers • (e.g) Monolithic architecture → Microservice architecture • But engineers cannot change their traditional workflows suddenly • Investigation and research at stage 2 is really important in terms of development scalabilities at stage 3 • At this moment, for container orchestrations, using kubernetes is better instead of ECS • Many players around containers join to kubernetes and develop eco systems (standing on the shoulders of giants)

3FDBQ • Selection of infrastructure platforms and approaches are important depend on the organization expansions • Writing infrastructures in Ruby DSL is pretty easy and works well • When organization becomes big, traditional workflow might not work

5IBOL:PV