CodeFest 2019. Леонид Руденко (JetBrains) — Terraform на примере кластера Selenoid

Тесты нескольких команд в JetBrains используют инфраструктуру на основе Selenoid. Поддерживать ее помогает Terraform.

Из доклада вы узнаете:
— что за инструмент Terraform;
— как Terraform может упростить развертывание и поддержку кластера Selenoid;
— в чем отличие от Ansible и стандартного configuration management Selenoid.

Terraform — достаточно универсальный инструмент. Найти ему применение можно не только в работе с Selenoid, но и в поддержке иной инфраструктуры.

Transcript

1. Terraform на примере кластера Selenoid Леонид Руденко Test Automation Engineer JetBrains
2. 5.5 years @ Yandex 3 years @ JetBrains
   Kotlin About me — 2
3. 3 What’s going on here —
4. What’s going on here — 4
5. What’s going on here — 5
6. https://youtu.be/wAKcBinMn6o https://youtu.be/4ZHQheFc4-8 https://youtu.be/G-TrW9SZRNg Selenoid — 6
7. Selenoid — Selenium Server OS 7
8. Selenoid — Selenoid OS 8
9. What’s going on here — 9
10. 1. Motivation 2. Terraform 101 3. Tips & Tricks for Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 10
11. 1. Motivation 2. Terraform 101 3. Tips & Tricks for Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 11
12. docker pull selenoid/firefox:latest docker pull selenoid/chrome:latest docker pull aandryashin/selenoid:1.0.0 Selenoid Quick Start — 12
13. docker pull selenoid/firefox:latest docker pull selenoid/chrome:latest docker pull aandryashin/selenoid:1.0.0 nano /etc/selenoid/browsers.json Selenoid Quick Start — 13
14. docker pull selenoid/firefox:latest docker pull selenoid/chrome:latest docker pull aandryashin/selenoid:1.0.0 nano /etc/selenoid/browsers.json docker run -d -p 4444:4444 \ -v /etc/selenoid:/etc/selenoid:ro \ -v /var/run/docker.sock:/var/run/docker.sock \ aandryashin/selenoid:1.0.0 Selenoid Quick Start — 14
15. Why Terraform? — Selenoid #1 Selenoid #2 GGR @Test Selenoid-UI #1 Selenoid-UI #2 68, 69, ... 47, 62, ... 68, 69, ... 47, 62, ... • 4 VMs • 9 containers • > 10 images 15
16. Terraform — 16
17. 1. Motivation 2. Terraform 101 3. Tips & Tricks for Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 17
18. • Configuration Terminology — 18
19. • Configuration • Provider Terminology — provider "docker" { host = "tcp://virtual-machine.company.net:2375" } 19
20. • Configuration • Provider Terminology — provider "vsphere" { user = "admin" password = "pa$$w0rd" vsphere_server = "vcenter-srv.company.net" allow_unverified_ssl = true } 20
21. • Configuration • Provider • Resource Terminology — resource "docker_container" "selenoid" { image = "aerokube/selenoid:1.9.0" command = ["-limit", "8", "-timeout", "2m0s"] ... } 21
22. • Configuration • Provider • Resource Terminology — resource "docker_container" "selenoid" { image = "aerokube/selenoid:1.9.0" command = ["-limit", "8", "-timeout", "2m0s"] ... } Resource name Resource type 22
23. Terminology — Resource 1 Provider 1 Configuration Provider 2 Resource 2 Resource 3 Resource 4 23
24. • https://www.terraform.io/downloads.html Installation — $ terraform version Terraform v0.11.11 $ 24
25. • Selenoid is running on http://localhost:4444 (Linux and Mac) • https://git.io/tf-task1 Task #1 — $ docker images | grep selenoid $ $ docker ps -a $ 25
26. Provider — provider "docker" { host = "unix:///var/run/docker.sock" // host = "tcp://virtual-machine.company.net:2375" } 26
27. Resource: image — resource "docker_image" "selenoid" { name = "aerokube/selenoid:1.9.0" } 27
28. Resource: container — resource "docker_container" "selenoid" { name = "selenoid" image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 28
29. Resource: container — resource "docker_container" "selenoid" { name = "selenoid" image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 29
30. Resource: container — resource "docker_container" "selenoid" { name = "selenoid" image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 30
31. Resource: container — resource "docker_container" "selenoid" { name = "selenoid" image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 31
32. Resource: container — resource "docker_container" "selenoid" { name = "selenoid" image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 32
33. Terraform init — $ terraform init Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "docker" (1.0.1)... Terraform has been successfully initialized! $ 33
34. Terraform init — $ terraform init Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "docker" (1.0.1)... Terraform has been successfully initialized! $ 34
35. Terraform apply — $ terraform apply docker_image.selenoid: Creating... docker_image.selenoid: Creation complete after 4s docker_container.selenoid: Creating... docker_container.selenoid: Creation complete after 1s Apply complete! Resources: 2 added, 0 changed, 0 destroyed. $ 35
36. Terraform apply — $ terraform apply ... Do you want to perform these actions? Terraform will perform the actions described above. Only ‘yes’ will be accepted to approve. Enter a value: 36
37. Terraform apply — $ terraform apply docker_image.selenoid: Creating... docker_image.selenoid: Creation complete after 4s docker_container.selenoid: Creating... docker_container.selenoid: Creation complete after 1s Apply complete! Resources: 2 added, 0 changed, 0 destroyed. $ 37
38. • http://localhost:4444/status Is Selenoid running? — 38
39. Terraform state file — $ ls selenoid.tf terraform.tfstate $ docker_image.selenoid: Creation complete after 4s docker_container.selenoid: Creating... docker_container.selenoid: Creation complete after 1s Apply complete! Resources: 2 added, 0 changed, 0 destroyed. $ 39
40. Terraform destroy — $ terraform destroy docker_container.selenoid: Destroying... docker_container.selenoid: Destruction complete after 1s docker_image.selenoid: Destroying... docker_image.selenoid: Destruction complete after 0s Destroy complete! Resources: 2 destroyed. $ 40
41. Terraform destroy — $ terraform destroy docker_container.selenoid: Destroying... docker_container.selenoid: Destruction complete after 1s docker_image.selenoid: Destroying... docker_image.selenoid: Destruction complete after 0s Destroy complete! Resources: 2 destroyed. $ 41
42. • http://localhost:4444/status Is Selenoid running? — 42
43. • Selenoid runs containers with Chrome and Firefox • https://git.io/tf-task2 Task #2 — 43
44. Variable — variable "browsers" { type = "list" default = [ "selenoid/chrome:69.0", "selenoid/firefox:62.0" ] } 44
45. Variable usage — resource "docker_image" "browser_images" { count = "${length(var.browsers)}" name = "${element(var.browsers, count.index)}" } 45
46. Variable usage — resource "docker_image" "browser_images" { count = "${length(var.browsers)}" name = "${element(var.browsers, count.index)}" } 46
47. Variable usage — resource "docker_image" "browser_images" { count = "${length(var.browsers)}" name = "${element(var.browsers, count.index)}" } 47
48. browsers.json — 48
49. Upload browsers.json — upload { content = "${file("browsers.json")}" file = "/etc/selenoid/browsers.json" } 49
50. Terraform apply — $ terraform apply docker_image.selenoid: Creating... docker_image.selenoid: Creation complete after 4s docker_container.selenoid: Creating... docker_container.selenoid: Creation complete after 1s Apply complete! Resources: 2 added, 0 changed, 0 destroyed. $ 50
51. • http://localhost:4444/status Is Selenoid running? — 51
52. • Add Opera browser with no Selenoid downtime • https://git.io/tf-task3 Task #3 — 52
53. Terraform apply — $ terraform apply Terraform will perform the following actions: -/+ docker_container.selenoid (new resource required) Plan: 1 to add, 0 to change, 1 to destroy. $ 53
54. Resource #1: Selenoid container with {} config Resource #2: file browsers.json + ??? Provisioner — 54
55. Mount config directory — upload { content = "{}" file = "/etc/selenoid/browsers.json" } volumes { host_path = "/Users/leonid.rudenko/selenoid/" container_path = "/etc/selenoid/" } 55
56. Mount config directory — upload { content = "{}" file = "/etc/selenoid/browsers.json" } volumes { host_path = "/Users/leonid.rudenko/selenoid/" container_path = "/etc/selenoid/" } 56
57. Mount config directory — upload { content = "{}" file = "/etc/selenoid/browsers.json" } volumes { host_path = "/Users/leonid.rudenko/selenoid/" container_path = "/etc/selenoid/" } 57
58. Local file resource and provisioner — resource "local_file" "browsers_json" { content = "${file("browsers.json")}" filename = "/Users/leonid.rudenko/selenoid/" provisioner "local_exec" { command = "docker kill –s HUP ${docker_container.selenoid.id}" } } 58
59. Local file resource and provisioner — resource "local_file" "browsers_json" { content = "${file("browsers.json")}" filename = "/Users/leonid.rudenko/selenoid/" provisioner "local_exec" { command = "docker kill –s HUP ${docker_container.selenoid.id}" } } 59
60. Local file resource and provisioner — resource "local_file" "browsers_json" { content = "${file("browsers.json")}" filename = "/Users/leonid.rudenko/selenoid/" provisioner "local_exec" { command = "docker kill –s HUP ${docker_container.selenoid.id}" } } 60
61. Local file resource and provisioner — resource "local_file" "browsers_json" { content = "${file("browsers.json")}" filename = "/Users/leonid.rudenko/selenoid/" provisioner "local_exec" { command = "docker kill –s HUP ${docker_container.selenoid.id}" } } 61
62. • http://localhost:4444/status Is Selenoid running? — 62
63. Terraform apply — $ terraform apply Terraform will perform the following actions: + docker_image.browser_images[2] -/+ local_file.browsers_json (new resource required) Plan: 2 to add, 0 to change, 1 to destroy. $ 63
64. • http://localhost:4444/status Is Selenoid running? — 64
65. 1. Motivation 2. Terraform 101 3. Tips & Tricks for Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 65
66. 1. Multiple Selenoids — 66
67. 1. Multiple Selenoids: create a module — 67
68. 1. Multiple Selenoids: create a module — • variables.tf 68
69. 1. Multiple Selenoids: create a module — • variables.tf • main.tf 69
70. 1. Multiple Selenoids: create a module — • variables.tf • main.tf • browsers.json "${file("${path.module}/browsers.json")}" 70
71. 1. Multiple Selenoids: module usage — module "selenoid" { source = "../modules/selenoid" hostname = "sel-1.***.net" selenoid_container_name = "sel1" } 71
72. 1. Multiple Selenoids: module usage — module "selenoid" { source = "../modules/selenoid" hostname = "sel-1.***.net" selenoid_container_name = "sel1" } $ terraform init - module.selenoid Getting source "../modules/selenoid" 72
73. 2. Grid Router — Selenoid #1 Selenoid #2 GGR @Test 73
74. 2. Grid Router — 74
75. resource "null_resource" "passwords" { triggers { passwords = "${sha1(file("./users.htpasswd"))}" } connection { host = "${var.hostname}" type = "ssh" user = "***" password = "***" } provisioner "file" { source = "./users.htpasswd" destination = "/home/jetbrains/users.htpasswd" } } 2. Grid Router — 75
76. resource "null_resource" "passwords" { triggers { passwords = "${sha1(file("./users.htpasswd"))}" } connection { host = "${var.hostname}" type = "ssh" user = "***" password = "***" } provisioner "file" { source = "./users.htpasswd" destination = "/home/jetbrains/users.htpasswd" } } 2. Grid Router — 76
77. resource "null_resource" "passwords" { triggers { passwords = "${sha1(file("./users.htpasswd"))}" } connection { host = "${var.hostname}" type = "ssh" user = "***" password = "***" } provisioner "file" { source = "./users.htpasswd" destination = "/home/jetbrains/users.htpasswd" } } 2. Grid Router — 77
78. resource "null_resource" "passwords" { triggers { passwords = "${sha1(file("./users.htpasswd"))}" } connection { host = "${var.hostname}" type = "ssh" user = "***" password = "***" } provisioner "file" { source = "./users.htpasswd" destination = "/home/jetbrains/users.htpasswd" } } 2. Grid Router — 78
79. resource "null_resource" "quota" { triggers { teamcity = "${sha1(file("./quota/teamcity.xml"))}" youtrack = "${sha1(file("./quota/youtrack.xml"))}" ... } connection { ... } provisioner "file" { source = "./quota" destination = "/home/jetbrains/" } provisioner "remote-exec" { inline = ["docker kill -s HUP ${docker_container.ggr.id}"] } } 2. Grid Router — 79
80. 3. Virtual machine parameters — 80
81. variable "vsphere_user" {} variable "vsphere_password" {} provider "vmware" { vcenter_server = "vcenter-srv.***.net" user = "${var.vsphere_user}" password = "${var.vsphere_password}" insecure_connection = true } 3. VMs in vSphere: non-official provider — 81
82. resource "vmware_virtual_machine" "sel-1" { image = "Shared-images/ubuntu-16.04-testing-39.32" ... cpus = 4 memory = 6144 } 3. VMs in vSphere — 82
83. resource "vmware_virtual_machine" "sel-1" { image = "Shared-images/ubuntu-16.04-testing-39.32" ... cpus = 4 memory = 6144 } 3. VMs in vSphere — 83
84. 3. Virtual machine parameters — • 10 containers, 4 CPU, 8GB RAM 84
85. 3. Virtual machine parameters — • 10 containers, 4 CPU, 8GB RAM • 8 containers, 4 CPU, 8GB RAM 85
86. 3. Virtual machine parameters — • 10 containers, 4 CPU, 8GB RAM • 8 containers, 4 CPU, 8GB RAM • 8 containers, 4 CPU, 6GB RAM 86
87. 1. Motivation 2. Terraform 101 3. Tips & Tricks for Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 87
88. 1. Selenoid configuration management — https://github.com/aerokube/cm/releases 88 ./cm selenoid start ./cm selenoid stop
89. 1. Selenoid cm: upgrade — 89 ./cm selenoid configure \ --browsers "chrome:X.0;firefox:Y.0" \ --version 1.9.0 --force ./cm selenoid start && ./cm selenoid stop
90. 1. Selenoid cm: drawbacks — • cm works with localhost (cm per machine, update binary) 90
91. 1. Selenoid cm: drawbacks — • cm works with localhost • bash script 91
92. 1. Selenoid cm: drawbacks — • cm works with localhost • bash script • old browser images are not removed 92
93. 1. Selenoid cm: drawbacks — • cm works with localhost • bash script • old browser images are not removed • can’t configure/upgrade ggr 93
94. 1. Selenoid cm: drawbacks — • cm works with localhost • bash script • old browser images are not removed • can’t configure/upgrade ggr • can’t help with monitoring 94
95. 1. Selenoid cm: drawbacks — • cm works with localhost • bash script • old browser images are not removed • can’t configure/upgrade ggr • can’t help with monitoring • no VM orchestration 95
96. 2. Ansible — 96 • configuration management tool, procedural & declarative
97. 2. Ansible — 97 • configuration management tool, procedural & declarative • ok, you can do everything with ansible :)
98. 2. Ansible — 98 • configuration management tool, procedural rather than declarative • ok, you can do everything with ansible :) • even VM orchestration
99. 2. Ansible — 99 • configuration management tool, procedural rather than declarative • ok, you can do everything with ansible :) • even VM orchestration • https://galaxy.ansible.com/iqoption/selenoid-docker
100. 2. Ansible — 100 • configuration management tool, procedural rather than declarative • ok, you can do everything with ansible :) • even VM orchestration • https://galaxy.ansible.com/iqoption/selenoid-docker • think of task idempotence (state=present or state=absent)
101. • Terraform basics • Real-life Selenoid cluster example • Take a look at your infrastructure "You see, I learned something today..." — 101
102. @LeonSabr Леонид Руденко Test Automation Engineer JetBrains Вопросы? leonid.rudenko@jetbrains.com leonid.a.rudenko@gmail.com