Provisioning your cloud with .NET

Compare approaches to provisioning cloud resources
Introduces pulumi as a way provisioning resources using .NET

Transcript

1. Infrastructure as Code Is it really? Provision your cloud with .NET core Shahid Iqbal | Freelance Azure consultant @shahiddev
2. @shahiddev Q&A go to slido.com event #Y860 Photo by Tyler Nix on Unsplash
4. @shahiddev Q&A go to slido.com event #Y860 Agenda Popular common options for provisioning cloud infrastructure Limitations of the common approaches Pulumi Title slide photo by Oscar Nord on Unsplash
5. @shahiddev Q&A go to slido.com event #Y860 Caveats Focus on Azure & .NET but principles are same on other platforms and languages Talk is inspired by my experience dealing with Cloud infrastructure whilst working with smaller teams I don’t work for Pulumi – i.e. not a sales pitch ☺
6. @shahiddev Q&A go to slido.com event #Y860 Who am I Freelance Azure consultant specialising in Azure, Kubernetes & Cloud native technologies. Over a decade of experience as a developer (mostly .NET) Microsoft MVP Co-organise meetup in the UK (Milton Keynes .NET) https://linkedin.shahid.dev shahid@headforcloud.com https://blog.headforcloud.com
7. @shahiddev Q&A go to slido.com event #Y860
8. @shahiddev Q&A go to slido.com event #Y860 Portal pros • Discoverability • Wizards/options • Some people more comfortable with GUIs
9. @shahiddev Q&A go to slido.com event #Y860 Portal cons • Not easy to repeat consistently • Options can be buried deep in submenus • Provisioning lots of resources takes time (not easy to run in parallel) • You have to resolve the resource dependencies
10. @shahiddev Q&A go to slido.com event #Y860 Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. https://en.wikipedia.org/wiki/Infrastructure_as_code
11. @shahiddev Q&A go to slido.com event #Y860 Cloud provider solutions Azure Resource Manager Google Cloud Deployment Manager
12. @shahiddev Q&A go to slido.com event #Y860 { "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "storageAccountType": { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } } }, "variables": { "storageAccountName": "[concat('store', uniquestring(resourceGroup().id))]" }, "resources": [ { "type": "Microsoft.Storage/storageAccounts", "apiVersion": "2019-04-01", "name": "[variables('storageAccountName')]", "location": "[parameters('location')]", "sku": { "name": "[parameters('storageAccountType')]" }, "kind": "StorageV2", "properties": {} } ], "outputs": { "storageAccountName": { "type": "string", "value": "[variables('storageAccountName')]" } } }
13. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "storageAccountType": { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } } },
14. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "storageAccountType": { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } } },
15. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "storageAccountType": { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } }
16. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "storageAccountType": { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } }
17. @shahiddev Q&A go to slido.com event #Y860 "variables": { "storageAccountName": "[concat('store', uniquestring(resourceGroup().id))]" },
18. @shahiddev Q&A go to slido.com event #Y860 "resources": [ { "type": "Microsoft.Storage/storageAccounts", "apiVersion": "2019-04-01", "name": "[variables('storageAccountName')]", "location": "[parameters('location')]", "sku": { "name": "[parameters('storageAccountType')]" }, "kind": "StorageV2", "properties": {} } ],
19. @shahiddev Q&A go to slido.com event #Y860 "resources": [ { "type": "Microsoft.Storage/storageAccounts", "apiVersion": "2019-04-01", "name": "[variables('storageAccountName')]", "location": "[parameters('location')]", "sku": { "name": "[parameters('storageAccountType')]" }, "kind": "StorageV2", "properties": {} } ],
20. @shahiddev Q&A go to slido.com event #Y860 "outputs": { "storageAccountName": { "type": "string", "value": "[variables('storageAccountName')]" } }
21. @shahiddev Q&A go to slido.com event #Y860 az group deployment create --resource-group <rg-name> --template-file <path-to-template> Azure CLI
22. @shahiddev Q&A go to slido.com event #Y860 Pros of Cloud templating options • Can be automated • Idempotent • Declarative – desired state • Easier to provision multiple resources • Resource dependency (mostly) automatically resolved
23. @shahiddev Q&A go to slido.com event #Y860 Cons of cloud templating • Verbose • Need to learn domain specific language • Specific to each cloud provider • Closed source
24. @shahiddev Q&A go to slido.com event #Y860 Terraform • Tool from Hashicorp • Uses their own markup language (HCL) • Multi-cloud/platform • Open source • Written in Go
25. @shahiddev Q&A go to slido.com event #Y860 resource "azurerm_resource_group" "example" { name = "example-resources" location = "West Europe" } resource "azurerm_storage_account" "example" { name = "storageaccountname" resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } }
26. @shahiddev Q&A go to slido.com event #Y860 resource "azurerm_resource_group" "example" { name = "example-resources" location = "West Europe" } resource "azurerm_storage_account" "example" { name = "storageaccountname" resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } }
27. @shahiddev Q&A go to slido.com event #Y860 Cons with Terraform • Special language used by Terraform only • Support in Terraform lags behind cloud provider • State management
28. @shahiddev Q&A go to slido.com event #Y860 More complex deployment scenarios • Conditional logic • Loops • Invoke 3rd party code
29. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "production": { "type": "string", "allowedValues": [ "Yes", "No" ], "metadata": { "description": “Should be in production or not." } } } "resources": [ { "condition": "[equals(parameters('production'), 'Yes')]", "name": "myApp", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "properties": { "serverFarmId": "hostingPlanName" } } ] ARM template snippet: Conditional logic
30. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "production": { "type": "string", "allowedValues": [ "Yes", "No" ], "metadata": { "description": “Should be in production or not." } } } "resources": [ { "condition": "[equals(parameters('production'), 'Yes')]", "name": "myApp", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "properties": { "serverFarmId": "hostingPlanName" } } ARM template snippet: Conditional logic
31. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "production": { "type": "string", "allowedValues": [ "Yes", "No" ], "metadata": { "description": “Should be in production or not." } } } "resources": [ { "condition": "[equals(parameters('production'), 'Yes')]", "name": "myApp", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "properties": { "serverFarmId": "hostingPlanName" } } ] ARM template snippet: Conditional logic
32. @shahiddev Q&A go to slido.com event #Y860 variable "IsProd" { type = bool, description = "if true this is production" } resource "azurerm_storage_account" "example" { name = "storageaccountname" count = var.IsProd ? 1 : 0 resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } Terraform snippet: Conditional logic
33. @shahiddev Q&A go to slido.com event #Y860 variable "IsProd" { type = bool, description = "if true this is production" } resource "azurerm_storage_account" "example" { name = "storageaccountname" count = var.IsProd ? 1 : 0 resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } Terraform snippet: Conditional logic
34. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "purpose": { "type": "array", "defaultValue": [ "Production", "Development" ] } }, "resources": [ { "name": "[concat('myApp-', parameters('purpose')[copyIndex()])]", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "copy": { "name": "websitescopy", "count": "[length(parameters('purpose'))]" }, "properties": { "serverFarmId": "hostingPlanName" } } ] ARM template snippet: Loops
35. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "purpose": { "type": "array", "defaultValue": [ "Production", "Development" ] } }, "resources": [ { "name": "[concat('myApp-', parameters('purpose')[copyIndex()])]", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "copy": { "name": "websitescopy", "count": "[length(parameters('purpose'))]" }, "properties": { "serverFarmId": "hostingPlanName" } } ] ARM template snippet: Loops
36. @shahiddev Q&A go to slido.com event #Y860 variable "environments" { description = "set of environments" type = list(string) default = ["dev", "staging", "prod"] } resource "azurerm_storage_account" "example" { count = length(var.environments) name = var.environments[count.index] resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } Terraform snippet: Loops
37. @shahiddev Q&A go to slido.com event #Y860 variable "environments" { description = "set of environments" type = list(string) default = ["dev", "staging", "prod"] } resource "azurerm_storage_account" "example" { count = length(var.environments) name = var.environments[count.index] resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } Terraform snippet: Loops
38. @shahiddev Q&A go to slido.com event #Y860 Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. https://en.wikipedia.org/wiki/Infrastructure_as_code
39. @shahiddev Q&A go to slido.com event #Y860 Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools.
40. @shahiddev Q&A go to slido.com event #Y860 Infrastructure as Code Software
41. @shahiddev Q&A go to slido.com event #Y860 Pulumi • Open source* • Uses modern programming languages • Supports many platforms • Declarative *Except for the Pulumi console
42. @shahiddev Q&A go to slido.com event #Y860 Supported languages
43. @shahiddev Q&A go to slido.com event #Y860 Supported Platforms
44. @shahiddev Q&A go to slido.com event #Y860
45. @shahiddev Q&A go to slido.com event #Y860
46. @shahiddev Q&A go to slido.com event #Y860
47. @shahiddev Q&A go to slido.com event #Y860 class Program { static Task<int> Main() { return Deployment.RunAsync(() => { // Create an Azure Resource Group var resourceGroup = new ResourceGroup("resourceGroup"); // Create an Azure Storage Account var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); // Export the connection string for the storage account return new Dictionary<string, object?> { { "connectionString", storageAccount.PrimaryConnectionString }, }; }); } }
48. @shahiddev Q&A go to slido.com event #Y860 class Program { static Task<int> Main() { return Deployment.RunAsync(() => { // Create an Azure Resource Group var resourceGroup = new ResourceGroup("resourceGroup"); // Create an Azure Storage Account var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); // Export the connection string for the storage account return new Dictionary<string, object?> { { "connectionString", storageAccount.PrimaryConnectionString }, }; }); }
49. @shahiddev Q&A go to slido.com event #Y860 class Program { static Task<int> Main() { return Deployment.RunAsync(() => { // Create an Azure Resource Group var resourceGroup = new ResourceGroup("resourceGroup"); // Create an Azure Storage Account var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); // Export the connection string for the storage account return new Dictionary<string, object?> { { "connectionString", storageAccount.PrimaryConnectionString }, }; }); }
50. @shahiddev Q&A go to slido.com event #Y860
51. @shahiddev Q&A go to slido.com event #Y860
52. @shahiddev Q&A go to slido.com event #Y860
53. @shahiddev Q&A go to slido.com event #Y860
54. @shahiddev Q&A go to slido.com event #Y860
55. @shahiddev Q&A go to slido.com event #Y860
56. @shahiddev Q&A go to slido.com event #Y860 Pulumi components https://www.pulumi.com/docs/intro/concepts/how-pulumi-works/ Local disk Cloud storage Pulumi console - $
57. @shahiddev Q&A go to slido.com event #Y860
58. @shahiddev Q&A go to slido.com event #Y860 Invoking 3rd party APIs Templating and Terraform options usually require dropping into a script inside the template. • Not easy to test • Relies on 3rd party tools to be available on the machine running the template • Yet another language (bash/PowerShell)
59. @shahiddev Q&A go to slido.com event #Y860 Azure storage static website hosting Run static websites/SPAs from storage account Feature can’t be enabled from Azure Resource Manager API • ARM templates/Terraform/Pulumi cannot enable this easily • Need to invoke script/Azure CLI to enable feature Can we use Azure storage .NET SDK? (spoiler alert – Yes!) https://github.com/pulumi/examples/tree/master/azure-cs-static-website
60. @shahiddev Q&A go to slido.com event #Y860 class Program { static Task<int> Main() { return Deployment.RunAsync(() => { // Create an Azure Resource Group var resourceGroup = new ResourceGroup("mystaticsite"); // Create an Azure Storage Account var storageAccount = new Account("mysite", new AccountArgs { ResourceGroupName = resourceGroup.Name, EnableHttpsTrafficOnly = true, AccountReplicationType = "LRS", AccountTier = "Standard", AccountKind = "StorageV2", AccessTier = "Hot", });
61. @shahiddev Q&A go to slido.com event #Y860 // The code in the Apply method must be idempotent. if (!Deployment.Instance.IsDryRun) storageAccount.PrimaryBlobConnectionString.Apply(async v => await EnableStaticSites(v) ); // Upload the files var files = new[]{"index.html", "404.html"}; foreach (var file in files) { var uploadedFile = new Blob(file, new BlobArgs { Name = file, StorageAccountName = storageAccount.Name, StorageContainerName = "$web", Type = "block", Source = $"./wwwroot/{file}", ContentType = "text/html", }); }
62. @shahiddev Q&A go to slido.com event #Y860 // The code in the Apply method must be idempotent. if (!Deployment.Instance.IsDryRun) storageAccount.PrimaryBlobConnectionString.Apply(async v => await EnableStaticSites(v) ); // Upload the files var files = new[]{"index.html", "404.html"}; foreach (var file in files) { var uploadedFile = new Blob(file, new BlobArgs { Name = file, StorageAccountName = storageAccount.Name, StorageContainerName = "$web", Type = "block", Source = $"./wwwroot/{file}", ContentType = "text/html", }); }
63. @shahiddev Q&A go to slido.com event #Y860 static async Task EnableStaticSites(string connectionString) { CloudStorageAccount sa = CloudStorageAccount.Parse(connectionString); var blobClient = sa.CreateCloudBlobClient(); ServiceProperties blobServiceProperties = new ServiceProperties(); blobServiceProperties.StaticWebsite = new StaticWebsiteProperties { Enabled = true, IndexDocument = "index.html", ErrorDocument404Path = "404.html" }; await blobClient.SetServicePropertiesAsync(blobServiceProperties); }
64. @shahiddev Q&A go to slido.com event #Y860 static async Task EnableStaticSites(string connectionString) { CloudStorageAccount sa = CloudStorageAccount.Parse(connectionString); var blobClient = sa.CreateCloudBlobClient(); ServiceProperties blobServiceProperties = new ServiceProperties(); blobServiceProperties.StaticWebsite = new StaticWebsiteProperties { Enabled = true, IndexDocument = "index.html", ErrorDocument404Path = "404.html" }; await blobClient.SetServicePropertiesAsync(blobServiceProperties); }
65. @shahiddev Q&A go to slido.com event #Y860 // The code in the Apply method must be idempotent. if (!Deployment.Instance.IsDryRun) storageAccount.PrimaryBlobConnectionString.Apply(async v => await EnableStaticSites(v) ); // Upload the files var files = new[]{"index.html", "404.html"}; foreach (var file in files) { var uploadedFile = new Blob(file, new BlobArgs { Name = file, StorageAccountName = storageAccount.Name, StorageContainerName = "$web", Type = "block", Source = $"./wwwroot/{file}", ContentType = "text/html", }); }
66. @shahiddev Q&A go to slido.com event #Y860 Changes: Type Name Operation + azure:core:ResourceGroup mystaticsite created + azure:storage:Account mysite created + pulumi:pulumi:Stack azure-cs-static-website-dev created + azure:storage:Blob index.html created + azure:storage:Blob 404.html created Resources: + created 5 Duration: 28s
67. @shahiddev Q&A go to slido.com event #Y860 Primary region Application gateway Web apps
68. @shahiddev Q&A go to slido.com event #Y860 Primary region Application gateway Web apps Secondary region Application gateway Web apps Traffic manager SQL Server Failover group Geo-replication High availability configuration
69. @shahiddev Q&A go to slido.com event #Y860 Steps for single region • Provision App service plan and n-web apps, capturing app urls • Provision Application gateway and configure routes to backend web apps (using the app urls). • Adding SSL certificates • Configure security headers • Configure healthchecks • Configure SQL server and create database, capturing server and db details
70. @shahiddev Q&A go to slido.com event #Y860 Steps for multi-region deployment • Provision into primary and secondary regions, capturing app gateway addresses • Add SQL geo-replication • Configure SQL failover group and capture the failover group connection string • Add Traffic manager and wire up backends to app gateways
71. @shahiddev Q&A go to slido.com event #Y860 Multi-region deployment code Coming soon
72. @shahiddev Q&A go to slido.com event #Y860 Challenges for smaller teams • No dedicated person/team to manage cloud resources • Complex templates or duplicated code • Often several steps that need to be coordinated • Need for team to understand not only their own app code but also cloud platform + templating language
73. @shahiddev Q&A go to slido.com event #Y860 Where Pulumi can help • Likelihood that .NET (or other languages) are more understood by team • Pulumi console means everyone sees the same picture ($)
74. @shahiddev Q&A go to slido.com event #Y860 Re-usable components • Create a stack for a group of resources that you can deploy together • Can be packaged into a Nuget package to use in my org • Compose more complex deployments by re-using stacks whilst ensuring everyone is using consistent configuration
75. @shahiddev Q&A go to slido.com event #Y860 Policy as Code Allows you to define policies which “intercept” deployments and will prevent certain things from being deployed. Better option than trying to abstract the Pulumi api and hide certain options from teams.
76. @shahiddev Q&A go to slido.com event #Y860 Working with existing resources • Flexible approach to working with existing resources • Co-exist along side previously deployed resources – no interference • Adopt existing resources into Pulumi (doesn’t generate the .NET code!) • Re-write/generate Pulumi code from existing resources • Tool to generate Pulumi code from Terraform – Tf2pulumi https://www.pulumi.com/docs/guides/adopting/
77. @shahiddev Q&A go to slido.com event #Y860 Pulumi pros • Using languages your teams are familiar with already • With the power of modern langaguages and 3rd SDKs can achieve most things directly in the code • Easy to get started • Free to use and OSS (unless you want the optional console)
78. @shahiddev Q&A go to slido.com event #Y860 Pulumi cons • Fewer platforms supported vs Terraform • A lag between feature release and support in Pulumi • Azure provider is using Terraform provider – dependency on competitor product • Still need to learn/discover the cloud provider resource API (not specific to Pulumi) • Some errors not apparent until Pulumi up • Can get into bad state – especially if you cancel mid-way* *it does warn you to not cancel mid-way!
79. @shahiddev Q&A go to slido.com event #Y860 What about SDKs for the Cloud? • Abstract REST calls behind code • Imperative • Not necessarily idempotent • Difficult to reason about current state vs new state
80. @shahiddev Q&A go to slido.com event #Y860 Transpilers • Take general purpose language and convert to cloud templating • AWS CDK – generates Cloud Formation templates • Farmer – F# code generates ARM templates.
81. @shahiddev Q&A go to slido.com event #Y860 Portals Cloud templating
82. @shahiddev Q&A go to slido.com event #Y860 Learn more https://github.com/pulumi/examples https://slack.pulumi.com/ https://blog.headforcloud.com
83. @shahiddev https://linkedin.shahid.dev shahid@headforcloud.com https://blog.headforcloud.com Photo by Pete Pedroza on Unsplash