CodeFest 2019. Леонид Руденко (JetBrains) — Terraform на примере кластера Selenoid

16b6c87229eaf58768d25ed7b2bbbf52?s=47 CodeFest
April 06, 2019

CodeFest 2019. Леонид Руденко (JetBrains) — Terraform на примере кластера Selenoid

Тесты нескольких команд в JetBrains используют инфраструктуру на основе Selenoid. Поддерживать ее помогает Terraform.

Из доклада вы узнаете:
— что за инструмент Terraform;
— как Terraform может упростить развертывание и поддержку кластера Selenoid;
— в чем отличие от Ansible и стандартного configuration management Selenoid.

Terraform — достаточно универсальный инструмент. Найти ему применение можно не только в работе с Selenoid, но и в поддержке иной инфраструктуры.

16b6c87229eaf58768d25ed7b2bbbf52?s=128

CodeFest

April 06, 2019
Tweet

Transcript

  1. Terraform на примере кластера Selenoid Леонид Руденко Test Automation Engineer

    JetBrains
  2. 5.5 years @ Yandex 3 years @ JetBrains  Kotlin

    About me — 2
  3. 3 What’s going on here —

  4. What’s going on here — 4

  5. What’s going on here — 5

  6. https://youtu.be/wAKcBinMn6o https://youtu.be/4ZHQheFc4-8 https://youtu.be/G-TrW9SZRNg Selenoid — 6

  7. Selenoid — Selenium Server OS 7

  8. Selenoid — Selenoid OS 8

  9. What’s going on here — 9

  10. 1. Motivation 2. Terraform 101 3. Tips & Tricks for

    Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 10
  11. 1. Motivation 2. Terraform 101 3. Tips & Tricks for

    Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 11
  12. docker pull selenoid/firefox:latest docker pull selenoid/chrome:latest docker pull aandryashin/selenoid:1.0.0 Selenoid

    Quick Start — 12
  13. docker pull selenoid/firefox:latest docker pull selenoid/chrome:latest docker pull aandryashin/selenoid:1.0.0 nano

    /etc/selenoid/browsers.json Selenoid Quick Start — 13
  14. docker pull selenoid/firefox:latest docker pull selenoid/chrome:latest docker pull aandryashin/selenoid:1.0.0 nano

    /etc/selenoid/browsers.json docker run -d -p 4444:4444 \ -v /etc/selenoid:/etc/selenoid:ro \ -v /var/run/docker.sock:/var/run/docker.sock \ aandryashin/selenoid:1.0.0 Selenoid Quick Start — 14
  15. Why Terraform? — Selenoid #1 Selenoid #2 GGR @Test Selenoid-UI

    #1 Selenoid-UI #2 68, 69, ... 47, 62, ... 68, 69, ... 47, 62, ... • 4 VMs • 9 containers • > 10 images 15
  16. Terraform — 16

  17. 1. Motivation 2. Terraform 101 3. Tips & Tricks for

    Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 17
  18. • Configuration Terminology — 18

  19. • Configuration • Provider Terminology — provider "docker" { host

    = "tcp://virtual-machine.company.net:2375" } 19
  20. • Configuration • Provider Terminology — provider "vsphere" { user

    = "admin" password = "pa$$w0rd" vsphere_server = "vcenter-srv.company.net" allow_unverified_ssl = true } 20
  21. • Configuration • Provider • Resource Terminology — resource "docker_container"

    "selenoid" { image = "aerokube/selenoid:1.9.0" command = ["-limit", "8", "-timeout", "2m0s"] ... } 21
  22. • Configuration • Provider • Resource Terminology — resource "docker_container"

    "selenoid" { image = "aerokube/selenoid:1.9.0" command = ["-limit", "8", "-timeout", "2m0s"] ... } Resource name Resource type 22
  23. Terminology — Resource 1 Provider 1 Configuration Provider 2 Resource

    2 Resource 3 Resource 4 23
  24. • https://www.terraform.io/downloads.html Installation — $ terraform version Terraform v0.11.11 $

    24
  25. • Selenoid is running on http://localhost:4444 (Linux and Mac) •

    https://git.io/tf-task1 Task #1 — $ docker images | grep selenoid $ $ docker ps -a $ 25
  26. Provider — provider "docker" { host = "unix:///var/run/docker.sock" // host

    = "tcp://virtual-machine.company.net:2375" } 26
  27. Resource: image — resource "docker_image" "selenoid" { name = "aerokube/selenoid:1.9.0"

    } 27
  28. Resource: container — resource "docker_container" "selenoid" { name = "selenoid"

    image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 28
  29. Resource: container — resource "docker_container" "selenoid" { name = "selenoid"

    image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 29
  30. Resource: container — resource "docker_container" "selenoid" { name = "selenoid"

    image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 30
  31. Resource: container — resource "docker_container" "selenoid" { name = "selenoid"

    image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 31
  32. Resource: container — resource "docker_container" "selenoid" { name = "selenoid"

    image = "${docker_image.selenoid.latest}" command = ["-limit", "4", "-timeout", "2m0s"] ports { internal = 4444 external = 4444 } volumes { host_path = "/var/run/docker.sock" container_path = "/var/run/docker.sock" } upload { content = "{}" file = "/etc/selenoid/browsers.json" } } 32
  33. Terraform init — $ terraform init Initializing provider plugins... -

    Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "docker" (1.0.1)... Terraform has been successfully initialized! $ 33
  34. Terraform init — $ terraform init Initializing provider plugins... -

    Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "docker" (1.0.1)... Terraform has been successfully initialized! $ 34
  35. Terraform apply — $ terraform apply docker_image.selenoid: Creating... docker_image.selenoid: Creation

    complete after 4s docker_container.selenoid: Creating... docker_container.selenoid: Creation complete after 1s Apply complete! Resources: 2 added, 0 changed, 0 destroyed. $ 35
  36. Terraform apply — $ terraform apply ... Do you want

    to perform these actions? Terraform will perform the actions described above. Only ‘yes’ will be accepted to approve. Enter a value: 36
  37. Terraform apply — $ terraform apply docker_image.selenoid: Creating... docker_image.selenoid: Creation

    complete after 4s docker_container.selenoid: Creating... docker_container.selenoid: Creation complete after 1s Apply complete! Resources: 2 added, 0 changed, 0 destroyed. $ 37
  38. • http://localhost:4444/status Is Selenoid running? — 38

  39. Terraform state file — $ ls selenoid.tf terraform.tfstate $ docker_image.selenoid:

    Creation complete after 4s docker_container.selenoid: Creating... docker_container.selenoid: Creation complete after 1s Apply complete! Resources: 2 added, 0 changed, 0 destroyed. $ 39
  40. Terraform destroy — $ terraform destroy docker_container.selenoid: Destroying... docker_container.selenoid: Destruction

    complete after 1s docker_image.selenoid: Destroying... docker_image.selenoid: Destruction complete after 0s Destroy complete! Resources: 2 destroyed. $ 40
  41. Terraform destroy — $ terraform destroy docker_container.selenoid: Destroying... docker_container.selenoid: Destruction

    complete after 1s docker_image.selenoid: Destroying... docker_image.selenoid: Destruction complete after 0s Destroy complete! Resources: 2 destroyed. $ 41
  42. • http://localhost:4444/status Is Selenoid running? — 42

  43. • Selenoid runs containers with Chrome and Firefox • https://git.io/tf-task2

    Task #2 — 43
  44. Variable — variable "browsers" { type = "list" default =

    [ "selenoid/chrome:69.0", "selenoid/firefox:62.0" ] } 44
  45. Variable usage — resource "docker_image" "browser_images" { count = "${length(var.browsers)}"

    name = "${element(var.browsers, count.index)}" } 45
  46. Variable usage — resource "docker_image" "browser_images" { count = "${length(var.browsers)}"

    name = "${element(var.browsers, count.index)}" } 46
  47. Variable usage — resource "docker_image" "browser_images" { count = "${length(var.browsers)}"

    name = "${element(var.browsers, count.index)}" } 47
  48. browsers.json — 48

  49. Upload browsers.json — upload { content = "${file("browsers.json")}" file =

    "/etc/selenoid/browsers.json" } 49
  50. Terraform apply — $ terraform apply docker_image.selenoid: Creating... docker_image.selenoid: Creation

    complete after 4s docker_container.selenoid: Creating... docker_container.selenoid: Creation complete after 1s Apply complete! Resources: 2 added, 0 changed, 0 destroyed. $ 50
  51. • http://localhost:4444/status Is Selenoid running? — 51

  52. • Add Opera browser with no Selenoid downtime • https://git.io/tf-task3

    Task #3 — 52
  53. Terraform apply — $ terraform apply Terraform will perform the

    following actions: -/+ docker_container.selenoid (new resource required) Plan: 1 to add, 0 to change, 1 to destroy. $ 53
  54. Resource #1: Selenoid container with {} config Resource #2: file

    browsers.json + ??? Provisioner — 54
  55. Mount config directory — upload { content = "{}" file

    = "/etc/selenoid/browsers.json" } volumes { host_path = "/Users/leonid.rudenko/selenoid/" container_path = "/etc/selenoid/" } 55
  56. Mount config directory — upload { content = "{}" file

    = "/etc/selenoid/browsers.json" } volumes { host_path = "/Users/leonid.rudenko/selenoid/" container_path = "/etc/selenoid/" } 56
  57. Mount config directory — upload { content = "{}" file

    = "/etc/selenoid/browsers.json" } volumes { host_path = "/Users/leonid.rudenko/selenoid/" container_path = "/etc/selenoid/" } 57
  58. Local file resource and provisioner — resource "local_file" "browsers_json" {

    content = "${file("browsers.json")}" filename = "/Users/leonid.rudenko/selenoid/" provisioner "local_exec" { command = "docker kill –s HUP ${docker_container.selenoid.id}" } } 58
  59. Local file resource and provisioner — resource "local_file" "browsers_json" {

    content = "${file("browsers.json")}" filename = "/Users/leonid.rudenko/selenoid/" provisioner "local_exec" { command = "docker kill –s HUP ${docker_container.selenoid.id}" } } 59
  60. Local file resource and provisioner — resource "local_file" "browsers_json" {

    content = "${file("browsers.json")}" filename = "/Users/leonid.rudenko/selenoid/" provisioner "local_exec" { command = "docker kill –s HUP ${docker_container.selenoid.id}" } } 60
  61. Local file resource and provisioner — resource "local_file" "browsers_json" {

    content = "${file("browsers.json")}" filename = "/Users/leonid.rudenko/selenoid/" provisioner "local_exec" { command = "docker kill –s HUP ${docker_container.selenoid.id}" } } 61
  62. • http://localhost:4444/status Is Selenoid running? — 62

  63. Terraform apply — $ terraform apply Terraform will perform the

    following actions: + docker_image.browser_images[2] -/+ local_file.browsers_json (new resource required) Plan: 2 to add, 0 to change, 1 to destroy. $ 63
  64. • http://localhost:4444/status Is Selenoid running? — 64

  65. 1. Motivation 2. Terraform 101 3. Tips & Tricks for

    Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 65
  66. 1. Multiple Selenoids — 66

  67. 1. Multiple Selenoids: create a module — 67

  68. 1. Multiple Selenoids: create a module — • variables.tf 68

  69. 1. Multiple Selenoids: create a module — • variables.tf •

    main.tf 69
  70. 1. Multiple Selenoids: create a module — • variables.tf •

    main.tf • browsers.json "${file("${path.module}/browsers.json")}" 70
  71. 1. Multiple Selenoids: module usage — module "selenoid" { source

    = "../modules/selenoid" hostname = "sel-1.***.net" selenoid_container_name = "sel1" } 71
  72. 1. Multiple Selenoids: module usage — module "selenoid" { source

    = "../modules/selenoid" hostname = "sel-1.***.net" selenoid_container_name = "sel1" } $ terraform init - module.selenoid Getting source "../modules/selenoid" 72
  73. 2. Grid Router — Selenoid #1 Selenoid #2 GGR @Test

    73
  74. 2. Grid Router — 74

  75. resource "null_resource" "passwords" { triggers { passwords = "${sha1(file("./users.htpasswd"))}" }

    connection { host = "${var.hostname}" type = "ssh" user = "***" password = "***" } provisioner "file" { source = "./users.htpasswd" destination = "/home/jetbrains/users.htpasswd" } } 2. Grid Router — 75
  76. resource "null_resource" "passwords" { triggers { passwords = "${sha1(file("./users.htpasswd"))}" }

    connection { host = "${var.hostname}" type = "ssh" user = "***" password = "***" } provisioner "file" { source = "./users.htpasswd" destination = "/home/jetbrains/users.htpasswd" } } 2. Grid Router — 76
  77. resource "null_resource" "passwords" { triggers { passwords = "${sha1(file("./users.htpasswd"))}" }

    connection { host = "${var.hostname}" type = "ssh" user = "***" password = "***" } provisioner "file" { source = "./users.htpasswd" destination = "/home/jetbrains/users.htpasswd" } } 2. Grid Router — 77
  78. resource "null_resource" "passwords" { triggers { passwords = "${sha1(file("./users.htpasswd"))}" }

    connection { host = "${var.hostname}" type = "ssh" user = "***" password = "***" } provisioner "file" { source = "./users.htpasswd" destination = "/home/jetbrains/users.htpasswd" } } 2. Grid Router — 78
  79. resource "null_resource" "quota" { triggers { teamcity = "${sha1(file("./quota/teamcity.xml"))}" youtrack

    = "${sha1(file("./quota/youtrack.xml"))}" ... } connection { ... } provisioner "file" { source = "./quota" destination = "/home/jetbrains/" } provisioner "remote-exec" { inline = ["docker kill -s HUP ${docker_container.ggr.id}"] } } 2. Grid Router — 79
  80. 3. Virtual machine parameters — 80

  81. variable "vsphere_user" {} variable "vsphere_password" {} provider "vmware" { vcenter_server

    = "vcenter-srv.***.net" user = "${var.vsphere_user}" password = "${var.vsphere_password}" insecure_connection = true } 3. VMs in vSphere: non-official provider — 81
  82. resource "vmware_virtual_machine" "sel-1" { image = "Shared-images/ubuntu-16.04-testing-39.32" ... cpus =

    4 memory = 6144 } 3. VMs in vSphere — 82
  83. resource "vmware_virtual_machine" "sel-1" { image = "Shared-images/ubuntu-16.04-testing-39.32" ... cpus =

    4 memory = 6144 } 3. VMs in vSphere — 83
  84. 3. Virtual machine parameters — • 10 containers, 4 CPU,

    8GB RAM 84
  85. 3. Virtual machine parameters — • 10 containers, 4 CPU,

    8GB RAM • 8 containers, 4 CPU, 8GB RAM 85
  86. 3. Virtual machine parameters — • 10 containers, 4 CPU,

    8GB RAM • 8 containers, 4 CPU, 8GB RAM • 8 containers, 4 CPU, 6GB RAM 86
  87. 1. Motivation 2. Terraform 101 3. Tips & Tricks for

    Selenoid cluster 4. Alternatives: cm & ansible What’s going on here — 87
  88. 1. Selenoid configuration management — https://github.com/aerokube/cm/releases 88 ./cm selenoid start

    ./cm selenoid stop
  89. 1. Selenoid cm: upgrade — 89 ./cm selenoid configure \

    --browsers "chrome:X.0;firefox:Y.0" \ --version 1.9.0 --force ./cm selenoid start && ./cm selenoid stop
  90. 1. Selenoid cm: drawbacks — • cm works with localhost

    (cm per machine, update binary) 90
  91. 1. Selenoid cm: drawbacks — • cm works with localhost

    • bash script 91
  92. 1. Selenoid cm: drawbacks — • cm works with localhost

    • bash script • old browser images are not removed 92
  93. 1. Selenoid cm: drawbacks — • cm works with localhost

    • bash script • old browser images are not removed • can’t configure/upgrade ggr 93
  94. 1. Selenoid cm: drawbacks — • cm works with localhost

    • bash script • old browser images are not removed • can’t configure/upgrade ggr • can’t help with monitoring 94
  95. 1. Selenoid cm: drawbacks — • cm works with localhost

    • bash script • old browser images are not removed • can’t configure/upgrade ggr • can’t help with monitoring • no VM orchestration 95
  96. 2. Ansible — 96 • configuration management tool, procedural &

    declarative
  97. 2. Ansible — 97 • configuration management tool, procedural &

    declarative • ok, you can do everything with ansible :)
  98. 2. Ansible — 98 • configuration management tool, procedural rather

    than declarative • ok, you can do everything with ansible :) • even VM orchestration
  99. 2. Ansible — 99 • configuration management tool, procedural rather

    than declarative • ok, you can do everything with ansible :) • even VM orchestration • https://galaxy.ansible.com/iqoption/selenoid-docker
  100. 2. Ansible — 100 • configuration management tool, procedural rather

    than declarative • ok, you can do everything with ansible :) • even VM orchestration • https://galaxy.ansible.com/iqoption/selenoid-docker • think of task idempotence (state=present or state=absent)
  101. • Terraform basics • Real-life Selenoid cluster example • Take

    a look at your infrastructure "You see, I learned something today..." — 101
  102. @LeonSabr Леонид Руденко Test Automation Engineer JetBrains Вопросы? leonid.rudenko@jetbrains.com leonid.a.rudenko@gmail.com