Taming AEM deployments

Continuous delivery model sounds great to almost everyone, but is not that easy to implement when real life comes into play. On the one hand, AEM architecture feels like a good fit for this strategy (everything is content after all, right?), but on the other hand we all know how challenging even individual deployment can get. Content management system itself is often just a fraction of your concern - you have to build it first, test it, stitch all the elements up and then figure out how to deliver your code in a repeatable and controlled way to production.

Throughout the last couple of years we’ve been trying really hard at Cognifide to make continuous release and delivery possible for AEM. It was a bumpy and twisty road, but we finally made it. During my talk I’d like to show you how we build AEM platforms these days using open source tools, including:

- Terraform to define and manage infrastructure as code
- Chef to describe deployments in an approachable way, which enforces consistency & repeatability across environments
- Consul that helps us discover services around and mitigate failures
- Jenkins & GoCD to orchestrate delivery model

Transcript

1. APACHE SLING & FRIENDS TECH MEETUP BERLIN, 25-27 SEPTEMBER 2017 Taming AEM deployments Jakub Wądołowski, Cognifide
2. The road to efficient AEM delivery 2 https://flic.kr/p/z8g25R ▪ Starts on your computer ▪ Ends in production ▪ Includes everything in between
3. 3 Development phase
4. Local environment 4 https://flic.kr/p/b93YGx ▪ Everyone runs the same ▪ It contains the entire technology stack ▪ Start over any time you need it
5. Solution 5
6. What’s in the box? 6 https://flic.kr/p/cMBbTu ▪ AEM Author & AEM Publish ▪ Preinstalled packages ▪ Dispatcher ▪ Solr Master & Slave ▪ knot.x
7. Vagrant in practice (1/3) 7 # Install prerequisites: # * Vagrant # * VirtualBox # * ChefDK $ git clone git@example.org:xyz/vagrant.git $ cd project-vagrant $ rake init $ rake up
8. Vagrant in practice (2/3) 8 ▪ http://localhost:4502 (AEM Author) ▪ http://localhost:4503 (AEM Publish) ▪ https://example.vagrant (Dispatcher) ▪ http://localhost:8983 (Solr) ▪ etc
9. Vagrant in practice (3/3) 9 $ rake provision:publish ... Chef Client finished, 8/196 resources updated in 02 minutes 02 seconds $ rake provision:httpd ... Chef Client finished, 4/41 resources updated in 19 seconds
10. Vagrant talk 10 https://goo.gl/hu216e
11. 11 Continuous integration
12. Continuous integration 12 https://flic.kr/p/nCGank ▪ Code integration ▪ Tests ▪ Release
13. Git flow 13
14. Use case (1/4) 14 ▪ 17 Chef cookbook repos ▪ 2 CI related repos ▪ 3 config repos ▪ 4 app repos ▪ 2 infrastructure repos https://flic.kr/p/bF3Nwm
15. Use case (2/4) 15 ▪ Each repository has its own lifecycle ▪ Simple Git flow https://flic.kr/p/peZfvA
16. Use case (3/4) 16 $ tree -L 1 . ├── aem ├── domain ├── knotx-autocomplete ├── knotx-download ├── knotx-filters ├── knotx-index ├── knotx-search ├── pom.xml └── solr 8 directories, 1 file https://flic.kr/p/fnPQuz
17. Use case (3/4) 17 $ tree -L 1 . ├── aem ├── domain ├── knotx-autocomplete ├── knotx-download ├── knotx-filters ├── knotx-index ├── knotx-search ├── pom.xml └── solr 8 directories, 1 file https://flic.kr/p/fnPQuz
18. Use case (3/4) 18 $ tree -L 1 . ├── aem ├── domain ├── knotx-autocomplete ├── knotx-download ├── knotx-filters ├── knotx-index ├── knotx-search ├── pom.xml └── solr 8 directories, 1 file https://flic.kr/p/fnPQuz
19. Use case (3/4) 19 $ tree -L 1 . ├── aem ├── domain ├── knotx-autocomplete ├── knotx-download ├── knotx-filters ├── knotx-index ├── knotx-search ├── solr └── pom.xml 8 directories, 1 file https://flic.kr/p/fnPQuz
20. Use case (3/4) 20 $ tree -L 1 . ├── aem ├── domain ├── knotx-autocomplete ├── knotx-download ├── knotx-filters ├── knotx-index ├── knotx-search ├── solr └── pom.xml 8 directories, 1 file https://flic.kr/p/fnPQuz
21. Use case (4/4) 21 ▪ Compatibility matrix is no longer needed ▪ End-to-end testing ▪ One release contains multiple artifacts
22. Pipelines 22 https://flic.kr/p/qG26gA ▪ Variety of choices ▪ Jenkins ▪ GoCD ▪ Bamboo ▪ etc ▪ Pipeline as code (Jenkinsfile)
23. 23 node('master') { stage('Checkout') { dir('app') { git url: 'git://example.com/aem-project.git', branch: 'master' } dir('tests') { git url: 'git://example.com/tests.git', branch: 'master' } } stage('Build') { node('aem') { dir('app/aem') { sh 'mvn clean install' } } } stage('Test') { dir('tests') { sh './gradlew clean contractTests' } } }
24. 24 node('master') { stage('Checkout') { dir('app') { git url: 'git://example.com/aem-project.git', branch: 'master' } dir('tests') { git url: 'git://example.com/tests.git', branch: 'master' } } stage('Build') { node('aem') { dir('app/aem') { sh 'mvn clean install' } } } stage('Test') { dir('tests') { sh './gradlew clean contractTests' } } }
25. 25 node('master') { stage('Checkout') { dir('app') { git url: 'git://example.com/aem-project.git', branch: 'master' } dir('tests') { git url: 'git://example.com/tests.git', branch: 'master' } } stage('Build') { node('aem') { dir('app/aem') { sh 'mvn clean install' } } } stage('Test') { dir('tests') { sh './gradlew clean contractTests' } } }
26. 26 node('master') { stage('Checkout') { dir('app') { git url: 'git://example.com/aem-project.git', branch: 'master' } dir('tests') { git url: 'git://example.com/tests.git', branch: 'master' } } stage('Build') { node('aem') { dir('app/aem') { sh 'mvn clean install' } } } stage('Test') { dir('tests') { sh './gradlew clean contractTests' } } }
27. Build / Deploy / Test 27
29. Build / Deploy / Test 29
30. Build / Deploy / Test 30
31. Build / Deploy / Test 31
32. Build / Deploy / Test 32
33. Build / Deploy / Test 33
34. Build / Deploy / Test 34
35. Build / Deploy / Test 35
36. Build / Deploy / Test 36
37. Build / Deploy / Test 37
38. Build / Deploy / Test 38
39. Build / Deploy / Test 39
40. Pipeline improvements 40 ▪ Dynamically defined stages ▪ Local & global configuration files ▪ Custom shared library https://flic.kr/p/juzu1W
41. 41 #!groovy @Library('cognifide-library@master') import com.cognifide.jenkinslib.Pipeline def pipeline = new Pipeline(this) node('master') { withMaven(maven: 'Maven 3') { stage('Quality Gate: Sonarqube') { sonarqube pipeline.CFG.sonarqube } stage('Build') { // ... } stage('Clear dispatcher cache') { knifeExec query: "chef_environment:${pipeline.CFG.chef.environmentName} AND tags:dispatcher" } } }
42. 42 #!groovy @Library('cognifide-library@master') import com.cognifide.jenkinslib.Pipeline def pipeline = new Pipeline(this) node('master') { withMaven(maven: 'Maven 3') { stage('Quality Gate: Sonarqube') { sonarqube pipeline.CFG.sonarqube } stage('Build') { // ... } stage('Clear dispatcher cache') { knifeExec query: "chef_environment:${pipeline.CFG.chef.environmentName} AND tags:dispatcher" } } }
43. 43 #!groovy @Library('cognifide-library@master') import com.cognifide.jenkinslib.Pipeline def pipeline = new Pipeline(this) node('master') { withMaven(maven: 'Maven 3') { stage('Quality Gate: Sonarqube') { sonarqube pipeline.CFG.sonarqube } stage('Build') { // ... } stage('Clear dispatcher cache') { knifeExec query: "chef_environment:${pipeline.CFG.chef.environmentName} AND tags:dispatcher" } } }
44. 44 #!groovy @Library('cognifide-library@master') import com.cognifide.jenkinslib.Pipeline def pipeline = new Pipeline(this) node('master') { withMaven(maven: 'Maven 3') { stage('Quality Gate: Sonarqube') { sonarqube pipeline.CFG.sonarqube } stage('Build') { // ... } stage('Clear dispatcher cache') { knifeExec query: "chef_environment:${pipeline.CFG.chef.environmentName} AND tags:dispatcher" } } }
45. 45 #!groovy @Library('cognifide-library@master') import com.cognifide.jenkinslib.Pipeline def pipeline = new Pipeline(this) node('master') { withMaven(maven: 'Maven 3') { stage('Quality Gate: Sonarqube') { sonarqube pipeline.CFG.sonarqube } stage('Build') { // ... } stage('Clear dispatcher cache') { knifeExec query: "chef_environment:${pipeline.CFG.chef.environmentName} AND tags:dispatcher" } } }
46. 46 #!groovy @Library('cognifide-library@master') import com.cognifide.jenkinslib.Pipeline def pipeline = new Pipeline(this) node('master') { withMaven(maven: 'Maven 3') { stage('Quality Gate: Sonarqube') { sonarqube pipeline.CFG.sonarqube } stage('Build') { // ... } stage('Clear dispatcher cache') { knifeExec query: "chef_environment:${pipeline.CFG.chef.environmentName} AND tags:dispatcher" } } }
47. 47 notifications: enabled: true recipients: - 'dev-team@example.com' deploy: profiles: aem-author: 'dev-author' aem-publish1: 'dev-publish1' aem-publish2: 'dev-publish2' tests: bobcat: profiles: - 'dev' - 'grid' mvnParams: '-Dfork.count=4' chef: environmentName: 'dev'
48. Testing 48 ▪ Unit tests ▪ Bobcat ▪ github.com/Cognifide/bobcat ▪ Zalenium & Docker ▪ github.com/zalando/zalenium ▪ AET ▪ github.com/Cognifide/aet https://flic.kr/p/juzgT9
49. Release 49 https://flic.kr/p/EQbNYd ▪ Release != deployment ▪ Unnoticeable process ▪ Artifact store is a must
50. 50 Infrastructure
51. Solution 51
52. Infrastructure setup 52 ▪ Multiple AWS environments ▪ Terraform modules ▪ Bootstrap process https://flic.kr/p/qCk82v
53. Service discovery 53 ▪ Interconnectivity ▪ Hardcoded IPs are not an option ▪ Adapts to changes by itself https://flic.kr/p/qr4XXW
54. Solution 54
55. Consul features 55 ▪ DNS-based service discovery ▪ Load balancing ▪ Health checks
56. Consul talk 56 https://goo.gl/aZjKsR
57. 57 Automated deployments
58. Solution 58
59. Chef 59 ▪ Configuration management ▪ Each service has its own cookbook ▪ Deployment steps written in DSL https://flic.kr/p/W25Lmv
60. Chef example 60 ▪ Install a package ▪ Restart AEM ▪ Remove default flush agent https://flic.kr/p/9qsZDC
61. 61 cq_package 'Author: Core app' do username 'admin' password 'admin' instance 'http://localhost:4502' source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip' http_user 'basic_auth_user' http_pass 'basic_auth_password' action :deploy end
62. 62 cq_package 'Author: Core app' do username 'admin' password 'admin' instance 'http://localhost:4502' source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip' http_user 'basic_auth_user' http_pass 'basic_auth_password' action :deploy end
63. 63 cq_package 'Author: Core app' do username 'admin' password 'admin' instance 'http://localhost:4502' source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip' http_user 'basic_auth_user' http_pass 'basic_auth_password' action :deploy end
64. 64 cq_package 'Author: Core app' do username 'admin' password 'admin' instance 'http://localhost:4502' source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip' http_user 'basic_auth_user' http_pass 'basic_auth_password' action :deploy end
65. 65 cq_package 'Author: Core app' do username node['cq']['author']['credentials']['login'] password node['cq']['author']['credentials']['password'] instance "http://localhost:#{node['cq']['author']['port']}" source "https://artifacts.example.com/xyz/#{ver_dir}/"\ "xyz-#{ver}-full.zip" http_user node['xyz-webapp']['nexus']['user'] http_pass node['xyz-webapp']['nexus']['password'] action :deploy end
66. 66 cq_package 'Author: Core app' do username 'admin' password 'admin' instance 'http://localhost:4502' source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip' http_user 'basic_auth_user' http_pass 'basic_auth_password' action :deploy notifies :restart, 'service[cq62-author]', :immediately end service 'cq62-author' do action :nothing end
67. 67 cq_package 'Author: Core app' do username 'admin' password 'admin' instance 'http://localhost:4502' source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip' http_user 'basic_auth_user' http_pass 'basic_auth_password' action :deploy notifies :restart, 'service[cq62-author]', :immediately end service 'cq62-author' do action :nothing end
68. 68 cq_package 'Author: Core app' do username 'admin' password 'admin' instance 'http://localhost:4502' source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip' http_user 'basic_auth_user' http_pass 'basic_auth_password' action :deploy notifies :restart, 'service[cq62-author]', :immediately end service 'cq62-author' do action :nothing end
69. 69 cq_package 'Author: Core app' do username 'admin' password 'admin' instance 'http://localhost:4502' source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip' http_user 'basic_auth_user' http_pass 'basic_auth_password' action :deploy notifies :restart, 'service[cq62-author]', :immediately end service 'cq62-author' do action :nothing end
70. 70 cq_jcr 'Author: /etc/replication/agents.author/flush' do path '/etc/replication/agents.author/flush' username 'admin' password 'admin' instance 'http://localhost:4502' action :delete end
71. 71 cq_jcr 'Author: /etc/replication/agents.author/flush' do path '/etc/replication/agents.author/flush' username 'admin' password 'admin' instance 'http://localhost:4502' action :delete end
72. 72 cq_jcr 'Author: /etc/replication/agents.author/flush' do path '/etc/replication/agents.author/flush' username 'admin' password 'admin' instance 'http://localhost:4502' action :delete end
73. 73 cq_jcr 'Author: /etc/replication/agents.author/flush' do path '/etc/replication/agents.author/flush' username 'admin' password 'admin' instance 'http://localhost:4502' action :delete end
74. 74 $ chef-client # ... Chef Client finished, 39/652 resources updated in 09 minutes 46 seconds $ chef-client # ... Chef Client finished, 0/652 resources updated in 01 minutes 58 seconds
75. 75 $ chef-client # ... Chef Client finished, 39/652 resources updated in 09 minutes 46 seconds $ chef-client # ... Chef Client finished, 0/652 resources updated in 01 minutes 58 seconds
76. 76 $ grep cq_package recipes/author*.rb | wc -l 70 $ grep cq_jcr recipes/author*.rb | wc -l 20 $ grep cq_package recipes/publish*.rb | wc -l 67 $ grep cq_jcr recipes/publish*.rb | wc -l 11
77. What’s currently available? 77 ▪ cq_package ▪ cq_osgi_config ▪ cq_osgi_bundle ▪ cq_osgi_component ▪ cq_user ▪ cq_jcr ▪ groovy_script ▪ apm_script ▪ … https://flic.kr/p/qhRjAF
78. AEM cookbook 78 github.com/jwadolowski/cookbook-cq
79. 79 Orchestration
80. Orchestration 80 ▪ Deployment pipelines ▪ Driven by “knife ssh” command https://flic.kr/p/eYhy4y
82. 82 $ knife ssh \ 'chef_environment:xyz-prod AND tags:aem AND tags:author' \ 'sudo -u apache /bin/touch /path/to/maintenance/maintenance.lock'
83. 83 $ knife ssh \ 'chef_environment:xyz-prod AND tags:aem AND tags:author' \ 'sudo -u apache /bin/touch /path/to/maintenance/maintenance.lock'
84. 84 $ knife ssh \ 'chef_environment:xyz-prod AND tags:aem AND tags:author' \ 'sudo -u apache /bin/touch /path/to/maintenance/maintenance.lock'
86. 86 $ knife ssh \ 'chef_environment:xyz-prod AND tags:dispatcher AND tags:us-east-1a' \ 'sudo -u apache /bin/touch /path/to/maintenance/deployment.lock'
87. 87 $ knife ssh \ 'chef_environment:xyz-prod AND tags:dispatcher AND tags:us-east-1a' \ 'sudo -u apache /bin/touch /path/to/maintenance/deployment.lock'
88. 88 $ knife ssh \ 'chef_environment:xyz-prod AND tags:dispatcher AND tags:us-east-1a' \ 'sudo -u apache /bin/touch /path/to/maintenance/deployment.lock'
90. 90 $ knife ssh \ 'chef_environment:xyz-prod AND ( (tags:dispatcher AND tags:us-east-1a) OR (tags:aem AND tags:publish AND tags:us-east-1a) )' \ 'sudo /usr/bin/chef-client'
91. 91 $ knife ssh \ 'chef_environment:xyz-prod AND ( (tags:dispatcher AND tags:us-east-1a) OR (tags:aem AND tags:publish AND tags:us-east-1a) )' \ 'sudo /usr/bin/chef-client'
92. 92 $ knife ssh \ 'chef_environment:xyz-prod AND ( (tags:dispatcher AND tags:us-east-1a) OR (tags:aem AND tags:publish AND tags:us-east-1a) )' \ 'sudo /usr/bin/chef-client'
93. 93 $ knife ssh \ 'chef_environment:xyz-prod AND tags:dispatcher AND tags:us-east-1a' \ 'sudo -u apache /usr/bin/rm -f /path/to/maintenance/deployment.lock'
98. Lessons learned 98 ▪ Fail fast ▪ Enforce correct exit codes ▪ How to resume failed pipeline? https://flic.kr/p/jzAyNF
99. 99 Danke schön!