ARM Wrestling, with the Elastic Stack

Azure Sydney user group presentation on building out an ARM template for deploying Elasticsearch and Kibana to Azure.

Transcript

1. 1 Russ Cam - @forloop ARM wrestling, with the Elastic Stack
2. 2 .NET Elasticsearch clients
3. 3 MSI Installers Microsoft Ignite
4. 4 Azure Marketplace
5. 5 The Elastic Stack Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store, Index, & Analyze Ingest Logstash Beats + Alerting Monitoring Graph ML
6. 6 Elasticsearch Overview Node 1 Node 2 Node3 Index
7. 7 Azure Marketplace & ARM
8. 8 Marketplace demo
9. 9 Marketplace UI { "handler": "Microsoft.Compute.Multi Vm", "version": "0.1.0- preview", "parameters": { "basics": [], "steps": [], "outputs": {} } }
10. 10 Marketplace UI and ARM Template
11. 11 Marketplace UI Issues • Only deploy into new Resource Group • Cross property validation • Available elements • No Integrated Contact Details
12. 12 Azure Resource Manager • Consistent API to provision resources ‒ Resource Provider Namespaces ‒ Microsoft.Storage ‒ Microsoft.Compute ‒ Microsoft.Network • Resource Groups ‒ Resources deployed into one Resource Group ‒ Resources can exist in different regions RESOURCE GROUP
13. 13 Imperative Deployment 1 Create resource group 2 Create vnet 3 etc…
14. 14 Declarative Deployment with templates ": "https://../deploymentTemplate.json#", ": "1.0.0.0", "resources": [{ "name" "my-vnet" "type" "Microsoft.Network/virtualNetworks" "location" "australiasoutheast" "apiVersion" "2015-06-15" "properties": { "addressSpace": { "addressPrefixes": ["192.168.0.0/16"] } } }]
15. 15 Declarative Deployment with templates Create resource group 2 Create deployment
16. 16 Elastic ARM Template https://github.com/elastic/azure-marketplace 0
17. 17 Deployments • Transactional(-ish) ‒ Deployment succeeds or fails in its entirety ‒ No rollback of resources created before point of failure • Deployment modes ‒ Incremental ‒ Complete
18. 18 ARM Quickstart Templates https://azure.microsoft.com/en-us/documentation/templates Web Based Deploy
19. 19 Publishing Portal https://publish.windowsazure.com/
20. 20 Publisher Portal Issues • Approval for Production ‒ Sometimes days, sometimes weeks • Different Best Practices ‒ Quickstart Templates vs. Marketplace Templates • Communication ‒ Ubuntu base image removed from Azure in some regions ‒ Existing resource selector removed from Marketplace
21. 21 Template Development
22. 22 Template Sections "parameters": {}, "variables": {}, "resources": [] "outputs": {}
23. 23 Parameters "parameters": { "vnetname" : { "type" : "string" } }, "[parameters('vnetname')]"
24. 24 Variables and Functions "variables": { "vnetname" : "[concat(parameters('vprefix'), '-', uniqueString(resourceGroup().id, deployment().name))]" }, "[variables('vnetname')]"
25. 25 Linked Templates "variables": "resources": [{ "name": "create-network", "type": "Microsoft.Resources/deployments", "apiVersion": "2016-02-01", "properties": { "mode": "incremental", "templateLink": { "uri": "linked-templates/create-network.json" }, "parameters": {}, }],
26. 26 Outputs "variables": {}, "resources": [], "outputs": { "fqdn": { "value": "[concat('http://', reference(resourceId('Microsoft.Network/publicIPAddresses', 'es-external-lb-ip'),providers('Microsoft.Network', 'publicIPAddresses').apiVersions[0]).dnsSettings.fqdn, ':9200')]", "type": "string" } }
27. 27 Outputs • Reference outputs in outer templates "[reference('create-network').outputs.fqdn.value]" • Resources themselves have outputs "[reference('<resource-id>', '<api-version>').<value>]"
28. 28 Template Outputs
29. 29 Branching "parameters": { "userinput" : { "type" : "string", "allowedValues" : ["option1","option2"] } }, "variables": { "options" : { "option1" : { "addressPrefixes": ["192.168.0.0/16"] }, "option2" : { "addressPrefixes": ["10.0.0.0/24"] }, }, "option" : "[variables('options')[parameters('userinput')]]" }, "[variables('option')]"
30. 30 Choosing Linked Templates "parameters": { "vNet" : { "type" : "string", "allowedValues" : ["new","existing"] } } "variables": { "networkTemplateOptions" : { "new" : "nested/new-network.json", "existing" : "nested/existing-network.json" }, "networkTemplate" : "[variables('networkTemplateOptions')[parameters('vNet')]]" }, "templateLink": { "uri" : "[variables('networkTemplate')]" }
31. 31 Looping copyIndex() "copy": { "name": "storagecopy", "count": 3 }
32. 32 Dependencies copyIndex() "copy": { "name": "storagecopy", "count": 3 } "dependsOn": ["storagecopy"]
33. 33 Deploying your Software onto Virtual Machines
34. 34 To bake or not to bake Using VM Extensions ✔ Faster Deployments ✔ Attaching disks to VMs is very fast ✘ Require baking for each version ✘ Upload to Azure ✘ Certification Process ✔ Faster Updates ✔ Leverage VM images available ✔ Adding new features a breeze ✘ Slower Deployments ✘ Desired State Configuration on each deploy
35. 35 Custom Script VM Extension "publisher" "Microsoft.OSTCExtensions" "type" "CustomScriptForLinux" "settings": { "fileUris": "[concat(variables('templateBaseUrl'), 'scripts/elastic-install.sh')] }, "protectedSettings": { "commandToExecute": "bash elastic-install.sh" }
36. 36 "Managing" waagent Scripts need to be Idempotent Package Manager Failures Custom Script VM Extension Issues
37. 37 Newer Custom Script VM Extension "publisher" "Microsoft.Azure.Extensions", "type" "CustomScript", "typeHandlerVersion" "2.0",
38. 38 Other VM Extensions https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-extensions-features/
39. 39 ARM Template Issues • Imperatively Declarative • Preflight Validation Checks • Manual Mapping • Still missing functions ‒ Max of two integers (now exists, but other useful ones missing) [div(add(add(a,b),(int(replace(string(sub(a,b)), "-", "")))), 2)]
40. 40 Recommendations
41. 41 Separation of Concerns
42. 42 Logging • Default built in log periodically flushes ‒ Duplicate log messages ‒ Out of order log messages • Custom Log function ‒ Order log messages with execution timestamp ‒ Easily see where failure occurs ‒ Diagnose slowest parts In-built and bespoke
43. 43 Web Deploy https://portal.azure.com/#create/Microsoft.Template/uri/{encoded-template-uri} Template Parameters
44. 44 Automated Testing • azure group template validate • UI Definition Outputs === ARM Template Inputs • azure group deployment create • CasperJS with Phantom • msportalfx-test ‒ https://www.npmjs.com/package/msportalfx-test
45. 45 Summary
46. 46 Elastic Stack Options ✔ Managed by Elastic ✔ Upgrades and Scaling ✔ Commercial Plugins ✔ Dedicated SLA based Support ✔ Automatic Backups ✘ Not on Azure…yet ✔ Customizable ✔ Automatable ✔ Reproducible ✔ Data Locality ✘ Management ✘ Administration ✔ Easy onboarding ✔ Discoverable through Azure Portal ✔ Contact Details ✘ Proof of Concepts ✘ Limited Features ✘ Updated
47. 47 Resources • ARM template ‒ https://github.com/elastic/azure-marketplace • ARM template example scenarios ‒ https://github.com/elastic/azure-marketplace-examples • Elastic ‒ https://elastic.co
48. 48 Thank you Questions?