Running Elasticsearch on Azure

Transcript

1. 1 Martijn Laarman - @mpdreamz Russ Cam - @forloop Running Elasticsearch On Azure
2. 2 Microsoft team at
3. 3 NEST - .NET Elasticsearch client
4. 4 MSI Installers Microsoft Ignite
5. 5 Azure Marketplace
6. 6 The Elastic Stack Elastic Cloud Security Monitoring Alerting Graph X-Pack Kibana User Interface Elasticsearch Store, Index, & Analyze Ingest Logstash Beats +
7. 7 Elasticsearch Overview Node 1 Node 2 Node3 Index
8. 8 Azure Marketplace & ARM
9. 9 Marketplace demo
10. 10 Marketplace UI { "handler": "Microsoft.Compute.Multi Vm", "version": "0.1.0- preview", "parameters": { "basics": [], "steps": [], "outputs": {} } }
11. 11 Marketplace UI and ARM Template
12. 12 Marketplace UI Issues • Only deploy into new Resource Group • Cross property validation • Available elements • No Integrated Contact Details
13. 13 Publishing Portal https://publish.windowsazure.com/
14. 14 Publisher Portal Issues • Approval for Production ‒ Sometimes days, sometimes weeks • Different Best Practices ‒ Quickstart Templates vs. Marketplace Templates • Communication ‒ Ubuntu base image removed from Azure in some regions ‒ Existing resource selector removed from Marketplace
15. 15 Azure Resource Manager • Consistent API to provision resources ‒ Resource Provider Namespaces ‒ Microsoft.Storage ‒ Microsoft.Compute ‒ Microsoft.Network • Resource Groups ‒ Resources deployed into one Resource Group ‒ Resources can exist in different regions RESOURCE GROUP
16. 16 Imperative Deployment 1 Create resource group 2 Create vnet 3 etc…
17. 17 Declarative Deployment ": "https://../deploymentTemplate.json#", ": "1.0.0.0", "resources": [{ "name" "my-vnet" "type" "Microsoft.Network/virtualNetworks" "location" "australiasoutheast" "apiVersion" "2015-06-15" "properties": { "addressSpace": { "addressPrefixes": ["192.168.0.0/16"] } } }]
18. 18 Declarative Deployment Create resource group 2 Create deployment
19. 19 Elastic ARM Template https://github.com/elastic/azure-marketplace 0
20. 20 Deployment • Transactional(-ish) ‒ Deployment succeeds or fails in its entirety ‒ No rollback of resources created before point of failure • Incremental ‒ Preexisting resources not in template untouched • Complete ‒ Delete preexisting resources not in template
21. 21 ARM Quickstart Templates https://azure.microsoft.com/en-us/documentation/templates Web Based Deploy
22. 22 Template Development
23. 23 Template Sections "parameters": {}, "variables": {}, "resources": [] "outputs": {}
24. 24 Parameters "parameters": { "vnetname" : { "type" : "string" } }, "[parameters('vnetname')]"
25. 25 Variables and Functions "variables": { "vnetname" : "[concat(parameters('vprefix'), '-', uniqueString(resourceGroup().id, deployment().name))]" }, "[variables('vnetname')]"
26. 26 Linked Templates "variables": "resources": [{ "name": "create-network", "type": "Microsoft.Resources/deployments", "apiVersion": "2016-02-01", "properties": { "mode": "incremental", "templateLink": { "uri": "linked-templates/create-network.json" }, "parameters": {}, }],
27. 27 Outputs "variables": {}, "resources": [], "outputs": { "fqdn": { "value": "[concat('http://', reference(resourceId('Microsoft.Network/publicIPAddresses', 'es-external-lb-ip'),providers('Microsoft.Network', 'publicIPAddresses').apiVersions[0]).dnsSettings.fqdn, ':9200')]", "type": "string" } }
28. 28 Outputs • Reference outputs in outer templates "[reference('create-network').outputs.fqdn.value]" • Resources themselves have outputs "[reference('<resource-id>', '<api-version>').<value>]"
29. 29 Template Outputs
30. 30 Branching "parameters": { "userinput" : { "type" : "string", "allowedValues" : ["option1","option2"] } }, "variables": { "options" : { "option1" : { "addressPrefixes": ["192.168.0.0/16"] }, "option2" : { "addressPrefixes": ["10.0.0.0/24"] }, }, "option" : [variables('options')[parameters('userinput')]] }, "[variables('option')]"
31. 31 Choosing Linked Templates "parameters": { "vNet" : { "type" : "string", "allowedValues" : ["new","existing"] } } "variables": { "networkTemplateOptions" : { "new" : "nested/new-network.json", "existing" : "nested/existing-network.json" }, "networkTemplate" : [variables('networkTemplateOptions')[parameters('vNet')]] }, "templateLink": { "uri" : "[variables('networkTemplate')]" }
32. 32 Looping copyIndex() "copy": { "name": "storagecopy", "count": 3 }
33. 33 Dependencies copyIndex() "copy": { "name": "storagecopy", "count": 3 } "dependsOn": ["storagecopy"]
34. 34 Deploying your Software onto Virtual Machines
35. 35 To bake or not to bake Using VM Extensions ✔ Faster Deployments ✔ Attaching disks to VMs is very fast ✘ Require baking for each version ✘ Upload to Azure ✘ Certification Process ✔ Faster Updates ✔ Leverage VM images available ✔ Adding new features a breeze ✘ Slower Deployments ✘ Desired State Configuration on each deploy
36. 36 Custom Script VM Extension "publisher" "Microsoft.OSTCExtensions" "type" "CustomScriptForLinux" "settings": { "fileUris": "[concat(variables('templateBaseUrl'), 'scripts/elastic-install.sh')] }, "protectedSettings": { "commandToExecute": "bash elastic-install.sh" }
37. 37 "Managing" waagent Scripts need to be Idempotent Package Manager Failures Custom Script VM Extension Issues
38. 38 Newer Custom Script VM Extension "publisher" "Microsoft.Azure.Extensions", "type" "CustomScript", "typeHandlerVersion" "2.0",
39. 39 Other VM Extensions https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-extensions-features/
40. 40 ARM Template Issues • Imperatively Declarative • Preflight Validation Checks • Manual Mapping • Missing functions div(add(add(a,b),(int(replace(string(sub(a,b)), "-", "")))), 2)
41. 41 Recommendations
42. 42 Logging • Default built in log periodically flushes ‒ Duplicate log messages ‒ Out of order log messages • Custom Log function ‒ Order log messages with execution timestamp ‒ Easily see where failure occurs ‒ Diagnose slowest parts In-built and bespoke
43. 43 Separation of Concerns
44. 44 Web Deploy https://portal.azure.com/#create/Microsoft.Template/uri/{encoded-template-uri} Template Parameters
45. 45 Automated Testing • azure group template validate • UI Definition Outputs === ARM Template Inputs • azure group deployment create • CasperJS with Phantom • msportalfx-test ‒ https://www.npmjs.com/package/msportalfx-test
46. 46 Summary
47. 47 Elastic Stack Options ✔ Managed by Elastic ✔ Upgrades and Scaling ✔ Commercial Plugins ✔ Dedicated SLA based Support ✔ Automatic Backups ✘ Not on Azure…yet ✔ Customizable ✔ Automatable ✔ Reproducible ✔ Data Locality ✘ Management ✘ Administration ✔ Easy onboarding ✔ Discoverable through Azure Portal ✔ Contact Details ✘ Proof of Concepts ✘ Limited Features ✘ Updated
48. 48 Thank you! Questions?