Provisioning your cloud with .NET

Provisioning your cloud with .NET

Compare approaches to provisioning cloud resources
Introduces pulumi as a way provisioning resources using .NET

3afba4357cdeb0055d2d6cc5e2216d8b?s=128

Shahid Iqbal

March 13, 2020
Tweet

Transcript

  1. Infrastructure as Code Is it really? Provision your cloud with

    .NET core Shahid Iqbal | Freelance Azure consultant @shahiddev
  2. @shahiddev Q&A go to slido.com event #Y860 Photo by Tyler

    Nix on Unsplash
  3. None
  4. @shahiddev Q&A go to slido.com event #Y860 Agenda Popular common

    options for provisioning cloud infrastructure Limitations of the common approaches Pulumi Title slide photo by Oscar Nord on Unsplash
  5. @shahiddev Q&A go to slido.com event #Y860 Caveats Focus on

    Azure & .NET but principles are same on other platforms and languages Talk is inspired by my experience dealing with Cloud infrastructure whilst working with smaller teams I don’t work for Pulumi – i.e. not a sales pitch ☺
  6. @shahiddev Q&A go to slido.com event #Y860 Who am I

    Freelance Azure consultant specialising in Azure, Kubernetes & Cloud native technologies. Over a decade of experience as a developer (mostly .NET) Microsoft MVP Co-organise meetup in the UK (Milton Keynes .NET) https://linkedin.shahid.dev shahid@headforcloud.com https://blog.headforcloud.com
  7. @shahiddev Q&A go to slido.com event #Y860

  8. @shahiddev Q&A go to slido.com event #Y860 Portal pros •

    Discoverability • Wizards/options • Some people more comfortable with GUIs
  9. @shahiddev Q&A go to slido.com event #Y860 Portal cons •

    Not easy to repeat consistently • Options can be buried deep in submenus • Provisioning lots of resources takes time (not easy to run in parallel) • You have to resolve the resource dependencies
  10. @shahiddev Q&A go to slido.com event #Y860 Infrastructure as code

    (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. https://en.wikipedia.org/wiki/Infrastructure_as_code
  11. @shahiddev Q&A go to slido.com event #Y860 Cloud provider solutions

    Azure Resource Manager Google Cloud Deployment Manager
  12. @shahiddev Q&A go to slido.com event #Y860 { "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

    "contentVersion": "1.0.0.0", "parameters": { "storageAccountType": { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } } }, "variables": { "storageAccountName": "[concat('store', uniquestring(resourceGroup().id))]" }, "resources": [ { "type": "Microsoft.Storage/storageAccounts", "apiVersion": "2019-04-01", "name": "[variables('storageAccountName')]", "location": "[parameters('location')]", "sku": { "name": "[parameters('storageAccountType')]" }, "kind": "StorageV2", "properties": {} } ], "outputs": { "storageAccountName": { "type": "string", "value": "[variables('storageAccountName')]" } } }
  13. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "storageAccountType":

    { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } } },
  14. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "storageAccountType":

    { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } } },
  15. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "storageAccountType":

    { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } }
  16. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "storageAccountType":

    { "type": "string", "defaultValue": "Standard_LRS", "allowedValues": [ "Standard_LRS", "Standard_GRS", "Standard_ZRS", "Premium_LRS" ], "metadata": { "description": "Storage Account type" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } }
  17. @shahiddev Q&A go to slido.com event #Y860 "variables": { "storageAccountName":

    "[concat('store', uniquestring(resourceGroup().id))]" },
  18. @shahiddev Q&A go to slido.com event #Y860 "resources": [ {

    "type": "Microsoft.Storage/storageAccounts", "apiVersion": "2019-04-01", "name": "[variables('storageAccountName')]", "location": "[parameters('location')]", "sku": { "name": "[parameters('storageAccountType')]" }, "kind": "StorageV2", "properties": {} } ],
  19. @shahiddev Q&A go to slido.com event #Y860 "resources": [ {

    "type": "Microsoft.Storage/storageAccounts", "apiVersion": "2019-04-01", "name": "[variables('storageAccountName')]", "location": "[parameters('location')]", "sku": { "name": "[parameters('storageAccountType')]" }, "kind": "StorageV2", "properties": {} } ],
  20. @shahiddev Q&A go to slido.com event #Y860 "outputs": { "storageAccountName":

    { "type": "string", "value": "[variables('storageAccountName')]" } }
  21. @shahiddev Q&A go to slido.com event #Y860 az group deployment

    create --resource-group <rg-name> --template-file <path-to-template> Azure CLI
  22. @shahiddev Q&A go to slido.com event #Y860 Pros of Cloud

    templating options • Can be automated • Idempotent • Declarative – desired state • Easier to provision multiple resources • Resource dependency (mostly) automatically resolved
  23. @shahiddev Q&A go to slido.com event #Y860 Cons of cloud

    templating • Verbose • Need to learn domain specific language • Specific to each cloud provider • Closed source
  24. @shahiddev Q&A go to slido.com event #Y860 Terraform • Tool

    from Hashicorp • Uses their own markup language (HCL) • Multi-cloud/platform • Open source • Written in Go
  25. @shahiddev Q&A go to slido.com event #Y860 resource "azurerm_resource_group" "example"

    { name = "example-resources" location = "West Europe" } resource "azurerm_storage_account" "example" { name = "storageaccountname" resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } }
  26. @shahiddev Q&A go to slido.com event #Y860 resource "azurerm_resource_group" "example"

    { name = "example-resources" location = "West Europe" } resource "azurerm_storage_account" "example" { name = "storageaccountname" resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } }
  27. @shahiddev Q&A go to slido.com event #Y860 Cons with Terraform

    • Special language used by Terraform only • Support in Terraform lags behind cloud provider • State management
  28. @shahiddev Q&A go to slido.com event #Y860 More complex deployment

    scenarios • Conditional logic • Loops • Invoke 3rd party code
  29. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "production":

    { "type": "string", "allowedValues": [ "Yes", "No" ], "metadata": { "description": “Should be in production or not." } } } "resources": [ { "condition": "[equals(parameters('production'), 'Yes')]", "name": "myApp", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "properties": { "serverFarmId": "hostingPlanName" } } ] ARM template snippet: Conditional logic
  30. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "production":

    { "type": "string", "allowedValues": [ "Yes", "No" ], "metadata": { "description": “Should be in production or not." } } } "resources": [ { "condition": "[equals(parameters('production'), 'Yes')]", "name": "myApp", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "properties": { "serverFarmId": "hostingPlanName" } } ARM template snippet: Conditional logic
  31. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "production":

    { "type": "string", "allowedValues": [ "Yes", "No" ], "metadata": { "description": “Should be in production or not." } } } "resources": [ { "condition": "[equals(parameters('production'), 'Yes')]", "name": "myApp", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "properties": { "serverFarmId": "hostingPlanName" } } ] ARM template snippet: Conditional logic
  32. @shahiddev Q&A go to slido.com event #Y860 variable "IsProd" {

    type = bool, description = "if true this is production" } resource "azurerm_storage_account" "example" { name = "storageaccountname" count = var.IsProd ? 1 : 0 resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } Terraform snippet: Conditional logic
  33. @shahiddev Q&A go to slido.com event #Y860 variable "IsProd" {

    type = bool, description = "if true this is production" } resource "azurerm_storage_account" "example" { name = "storageaccountname" count = var.IsProd ? 1 : 0 resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } Terraform snippet: Conditional logic
  34. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "purpose":

    { "type": "array", "defaultValue": [ "Production", "Development" ] } }, "resources": [ { "name": "[concat('myApp-', parameters('purpose')[copyIndex()])]", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "copy": { "name": "websitescopy", "count": "[length(parameters('purpose'))]" }, "properties": { "serverFarmId": "hostingPlanName" } } ] ARM template snippet: Loops
  35. @shahiddev Q&A go to slido.com event #Y860 "parameters": { "purpose":

    { "type": "array", "defaultValue": [ "Production", "Development" ] } }, "resources": [ { "name": "[concat('myApp-', parameters('purpose')[copyIndex()])]", "type": "Microsoft.Web/sites", "location": "West Europe", "apiVersion": "2015-08-01", "copy": { "name": "websitescopy", "count": "[length(parameters('purpose'))]" }, "properties": { "serverFarmId": "hostingPlanName" } } ] ARM template snippet: Loops
  36. @shahiddev Q&A go to slido.com event #Y860 variable "environments" {

    description = "set of environments" type = list(string) default = ["dev", "staging", "prod"] } resource "azurerm_storage_account" "example" { count = length(var.environments) name = var.environments[count.index] resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } Terraform snippet: Loops
  37. @shahiddev Q&A go to slido.com event #Y860 variable "environments" {

    description = "set of environments" type = list(string) default = ["dev", "staging", "prod"] } resource "azurerm_storage_account" "example" { count = length(var.environments) name = var.environments[count.index] resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "GRS" } Terraform snippet: Loops
  38. @shahiddev Q&A go to slido.com event #Y860 Infrastructure as code

    (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. https://en.wikipedia.org/wiki/Infrastructure_as_code
  39. @shahiddev Q&A go to slido.com event #Y860 Infrastructure as code

    (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools.
  40. @shahiddev Q&A go to slido.com event #Y860 Infrastructure as Code

    Software
  41. @shahiddev Q&A go to slido.com event #Y860 Pulumi • Open

    source* • Uses modern programming languages • Supports many platforms • Declarative *Except for the Pulumi console
  42. @shahiddev Q&A go to slido.com event #Y860 Supported languages

  43. @shahiddev Q&A go to slido.com event #Y860 Supported Platforms

  44. @shahiddev Q&A go to slido.com event #Y860

  45. @shahiddev Q&A go to slido.com event #Y860

  46. @shahiddev Q&A go to slido.com event #Y860

  47. @shahiddev Q&A go to slido.com event #Y860 class Program {

    static Task<int> Main() { return Deployment.RunAsync(() => { // Create an Azure Resource Group var resourceGroup = new ResourceGroup("resourceGroup"); // Create an Azure Storage Account var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); // Export the connection string for the storage account return new Dictionary<string, object?> { { "connectionString", storageAccount.PrimaryConnectionString }, }; }); } }
  48. @shahiddev Q&A go to slido.com event #Y860 class Program {

    static Task<int> Main() { return Deployment.RunAsync(() => { // Create an Azure Resource Group var resourceGroup = new ResourceGroup("resourceGroup"); // Create an Azure Storage Account var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); // Export the connection string for the storage account return new Dictionary<string, object?> { { "connectionString", storageAccount.PrimaryConnectionString }, }; }); }
  49. @shahiddev Q&A go to slido.com event #Y860 class Program {

    static Task<int> Main() { return Deployment.RunAsync(() => { // Create an Azure Resource Group var resourceGroup = new ResourceGroup("resourceGroup"); // Create an Azure Storage Account var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); // Export the connection string for the storage account return new Dictionary<string, object?> { { "connectionString", storageAccount.PrimaryConnectionString }, }; }); }
  50. @shahiddev Q&A go to slido.com event #Y860

  51. @shahiddev Q&A go to slido.com event #Y860

  52. @shahiddev Q&A go to slido.com event #Y860

  53. @shahiddev Q&A go to slido.com event #Y860

  54. @shahiddev Q&A go to slido.com event #Y860

  55. @shahiddev Q&A go to slido.com event #Y860

  56. @shahiddev Q&A go to slido.com event #Y860 Pulumi components https://www.pulumi.com/docs/intro/concepts/how-pulumi-works/

    Local disk Cloud storage Pulumi console - $
  57. @shahiddev Q&A go to slido.com event #Y860

  58. @shahiddev Q&A go to slido.com event #Y860 Invoking 3rd party

    APIs Templating and Terraform options usually require dropping into a script inside the template. • Not easy to test • Relies on 3rd party tools to be available on the machine running the template • Yet another language (bash/PowerShell)
  59. @shahiddev Q&A go to slido.com event #Y860 Azure storage static

    website hosting Run static websites/SPAs from storage account Feature can’t be enabled from Azure Resource Manager API • ARM templates/Terraform/Pulumi cannot enable this easily • Need to invoke script/Azure CLI to enable feature Can we use Azure storage .NET SDK? (spoiler alert – Yes!) https://github.com/pulumi/examples/tree/master/azure-cs-static-website
  60. @shahiddev Q&A go to slido.com event #Y860 class Program {

    static Task<int> Main() { return Deployment.RunAsync(() => { // Create an Azure Resource Group var resourceGroup = new ResourceGroup("mystaticsite"); // Create an Azure Storage Account var storageAccount = new Account("mysite", new AccountArgs { ResourceGroupName = resourceGroup.Name, EnableHttpsTrafficOnly = true, AccountReplicationType = "LRS", AccountTier = "Standard", AccountKind = "StorageV2", AccessTier = "Hot", });
  61. @shahiddev Q&A go to slido.com event #Y860 // The code

    in the Apply method must be idempotent. if (!Deployment.Instance.IsDryRun) storageAccount.PrimaryBlobConnectionString.Apply(async v => await EnableStaticSites(v) ); // Upload the files var files = new[]{"index.html", "404.html"}; foreach (var file in files) { var uploadedFile = new Blob(file, new BlobArgs { Name = file, StorageAccountName = storageAccount.Name, StorageContainerName = "$web", Type = "block", Source = $"./wwwroot/{file}", ContentType = "text/html", }); }
  62. @shahiddev Q&A go to slido.com event #Y860 // The code

    in the Apply method must be idempotent. if (!Deployment.Instance.IsDryRun) storageAccount.PrimaryBlobConnectionString.Apply(async v => await EnableStaticSites(v) ); // Upload the files var files = new[]{"index.html", "404.html"}; foreach (var file in files) { var uploadedFile = new Blob(file, new BlobArgs { Name = file, StorageAccountName = storageAccount.Name, StorageContainerName = "$web", Type = "block", Source = $"./wwwroot/{file}", ContentType = "text/html", }); }
  63. @shahiddev Q&A go to slido.com event #Y860 static async Task

    EnableStaticSites(string connectionString) { CloudStorageAccount sa = CloudStorageAccount.Parse(connectionString); var blobClient = sa.CreateCloudBlobClient(); ServiceProperties blobServiceProperties = new ServiceProperties(); blobServiceProperties.StaticWebsite = new StaticWebsiteProperties { Enabled = true, IndexDocument = "index.html", ErrorDocument404Path = "404.html" }; await blobClient.SetServicePropertiesAsync(blobServiceProperties); }
  64. @shahiddev Q&A go to slido.com event #Y860 static async Task

    EnableStaticSites(string connectionString) { CloudStorageAccount sa = CloudStorageAccount.Parse(connectionString); var blobClient = sa.CreateCloudBlobClient(); ServiceProperties blobServiceProperties = new ServiceProperties(); blobServiceProperties.StaticWebsite = new StaticWebsiteProperties { Enabled = true, IndexDocument = "index.html", ErrorDocument404Path = "404.html" }; await blobClient.SetServicePropertiesAsync(blobServiceProperties); }
  65. @shahiddev Q&A go to slido.com event #Y860 // The code

    in the Apply method must be idempotent. if (!Deployment.Instance.IsDryRun) storageAccount.PrimaryBlobConnectionString.Apply(async v => await EnableStaticSites(v) ); // Upload the files var files = new[]{"index.html", "404.html"}; foreach (var file in files) { var uploadedFile = new Blob(file, new BlobArgs { Name = file, StorageAccountName = storageAccount.Name, StorageContainerName = "$web", Type = "block", Source = $"./wwwroot/{file}", ContentType = "text/html", }); }
  66. @shahiddev Q&A go to slido.com event #Y860 Changes: Type Name

    Operation + azure:core:ResourceGroup mystaticsite created + azure:storage:Account mysite created + pulumi:pulumi:Stack azure-cs-static-website-dev created + azure:storage:Blob index.html created + azure:storage:Blob 404.html created Resources: + created 5 Duration: 28s
  67. @shahiddev Q&A go to slido.com event #Y860 Primary region Application

    gateway Web apps
  68. @shahiddev Q&A go to slido.com event #Y860 Primary region Application

    gateway Web apps Secondary region Application gateway Web apps Traffic manager SQL Server Failover group Geo-replication High availability configuration
  69. @shahiddev Q&A go to slido.com event #Y860 Steps for single

    region • Provision App service plan and n-web apps, capturing app urls • Provision Application gateway and configure routes to backend web apps (using the app urls). • Adding SSL certificates • Configure security headers • Configure healthchecks • Configure SQL server and create database, capturing server and db details
  70. @shahiddev Q&A go to slido.com event #Y860 Steps for multi-region

    deployment • Provision into primary and secondary regions, capturing app gateway addresses • Add SQL geo-replication • Configure SQL failover group and capture the failover group connection string • Add Traffic manager and wire up backends to app gateways
  71. @shahiddev Q&A go to slido.com event #Y860 Multi-region deployment code

    Coming soon
  72. @shahiddev Q&A go to slido.com event #Y860 Challenges for smaller

    teams • No dedicated person/team to manage cloud resources • Complex templates or duplicated code • Often several steps that need to be coordinated • Need for team to understand not only their own app code but also cloud platform + templating language
  73. @shahiddev Q&A go to slido.com event #Y860 Where Pulumi can

    help • Likelihood that .NET (or other languages) are more understood by team • Pulumi console means everyone sees the same picture ($)
  74. @shahiddev Q&A go to slido.com event #Y860 Re-usable components •

    Create a stack for a group of resources that you can deploy together • Can be packaged into a Nuget package to use in my org • Compose more complex deployments by re-using stacks whilst ensuring everyone is using consistent configuration
  75. @shahiddev Q&A go to slido.com event #Y860 Policy as Code

    Allows you to define policies which “intercept” deployments and will prevent certain things from being deployed. Better option than trying to abstract the Pulumi api and hide certain options from teams.
  76. @shahiddev Q&A go to slido.com event #Y860 Working with existing

    resources • Flexible approach to working with existing resources • Co-exist along side previously deployed resources – no interference • Adopt existing resources into Pulumi (doesn’t generate the .NET code!) • Re-write/generate Pulumi code from existing resources • Tool to generate Pulumi code from Terraform – Tf2pulumi https://www.pulumi.com/docs/guides/adopting/
  77. @shahiddev Q&A go to slido.com event #Y860 Pulumi pros •

    Using languages your teams are familiar with already • With the power of modern langaguages and 3rd SDKs can achieve most things directly in the code • Easy to get started • Free to use and OSS (unless you want the optional console)
  78. @shahiddev Q&A go to slido.com event #Y860 Pulumi cons •

    Fewer platforms supported vs Terraform • A lag between feature release and support in Pulumi • Azure provider is using Terraform provider – dependency on competitor product • Still need to learn/discover the cloud provider resource API (not specific to Pulumi) • Some errors not apparent until Pulumi up • Can get into bad state – especially if you cancel mid-way* *it does warn you to not cancel mid-way!
  79. @shahiddev Q&A go to slido.com event #Y860 What about SDKs

    for the Cloud? • Abstract REST calls behind code • Imperative • Not necessarily idempotent • Difficult to reason about current state vs new state
  80. @shahiddev Q&A go to slido.com event #Y860 Transpilers • Take

    general purpose language and convert to cloud templating • AWS CDK – generates Cloud Formation templates • Farmer – F# code generates ARM templates.
  81. @shahiddev Q&A go to slido.com event #Y860 Portals Cloud templating

  82. @shahiddev Q&A go to slido.com event #Y860 Learn more https://github.com/pulumi/examples

    https://slack.pulumi.com/ https://blog.headforcloud.com
  83. @shahiddev https://linkedin.shahid.dev shahid@headforcloud.com https://blog.headforcloud.com Photo by Pete Pedroza on Unsplash