Scaling Infrastructure with Pulumi

TechMeetup Conference 2023 - Ostrava (Czech Republic)

Every company goes through a journey of growing their services and the infrastructure that needs to support it. The right tools are needed to support this journey. By leveraging regular programming languages, Pulumi supports you at every step in this growth trajectory.

In this talk, Ringo will sketch how Pulumi supports your team in integrating the infrastructure setup at every step of the way.

Some of the questions which will be answered during the talk:

* why did Pulumi choose regular programming languages?
* how can this still be declarative?
* am I limited to a single programming language for all my infrastructure code?
* how can I reuse my infrastructure code over different teams?
* how can I set up secure & compliant infrastructure?
* what if I want to build a custom developer platform?

This talk will show you solutions ranging from the smallest Pulumi infrastructure program up to a fully integrated self-service platform setup.

Transcript

1. TechMeetup Conference 2023 Ostrava 3 November 2023 Ringo De Smet Mastodon: @ringods@hachyderm.io Twitter/X: @ringods
2. 2 On-Premises Datacenters (Server-Centric) Lift-and-Shift Migrations to Cloud (Server-Centric) D ocker AW S Lam bda TODAY Cloud Engineering AW S EC 2 (IaaS) R estful A P Is TIPPING POINT: Legacy tools and processes were not designed for modern applications and cloud services, stifling innovation. G itH ub Web-Scale IT (Server-Centric) Modern Cloud-Native Apps (Service-Centric) D evO ps Infrastructure-as-Code Cloud-as-Software Managing Servers Focus on Building Infrastructure & Operating IT Managing Services Focus on Composing & Consuming Cloud Services Server Config Management M icroservices Kubernetes A zure G C P 2010 2000 2023 The Rise of Modern Cloud Cloud complexity is escalating
3. 3 Yesterday’s tools and practices don’t scale in the modern cloud era Legacy Tools Built for the speed and scale of yesterday Modern Tools Built for high speed and massive scale
4. 4 Getting Started A first full environ- ment Growing your portfolio Any protection ? Growing your company How does Pulumi support you in your growth?
5. 5 1. Getting Started Understanding the (power of) the product
6. 6 Multi-Cloud Infrastructure as Code BUILD, DEPLOY, AND MANAGE ANYWHERE WITH A STANDARD WORKFLOW Infra Providers Cloud Native Clouds Core Features: • Any Cloud, Any Language • State Management • Secrets Management • Guaranteed Preview Plans • CI/CD Integrations • Webhooks • REST API • Automation API • Dashboards and Reports
7. 7 Pulumi - High Level View • Pulumi Architecture • Language Host • CLI & Engine • Providers • State Backend •Pulumi Service •Cloud Storage (S3, …)
8. 8 Pulumi - High Level View • Automate your infrastructure setup with 1 or more Pulumi projects • A project contains • a program: this is your “template” from which you can create stacks • the stacks: each stack is an “instance” of the program • One project can use info from another project as input
9. 9 How Pulumi Works - first run Pulumi CLI & Engine $ pulumi up
10. 10 How Pulumi Works - first run Language Host Pulumi CLI & Engine $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup
11. 11 How Pulumi Works - first run Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CREATE SG
12. 12 How Pulumi Works - first run Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CREATE SG web-sg: … Actual SG info
13. 13 How Pulumi Works - second run Pulumi CLI & Engine $ pulumi up web-sg: …
14. 14 How Pulumi Works - second run Language Host Pulumi CLI & Engine $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup web-sg: …
15. 15 How Pulumi Works - second run Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CHECK SG web-sg: …
16. 16 How Pulumi Works - multiple resources Pulumi CLI & Engine $ pulumi up
17. 17 How Pulumi Works - multiple resources Language Host Pulumi CLI & Engine $ pulumi up runtime: nodejs Start SG & Instance
18. 18 How Pulumi Works - multiple resources Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start SG & Instance provider: aws Start CREATE SG & CREATE Instance
19. 19 How Pulumi Works - dependencies Pulumi CLI & Engine $ pulumi up
20. 20 How Pulumi Works - dependencies Language Host Pulumi CLI & Engine $ pulumi up runtime: nodejs Start SG & Instance
21. 21 How Pulumi Works - dependencies Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start SG & Instance provider: aws Start CREATE SG CREATE Instance Wait
22. 22 2. Your first full environment The journey from network to services
23. 23 Pulumi at Scale org/network/dev org/network/qa org/network/prod org/cluster/dev org/cluster/qa org/cluster/prod org services svc1-pro d org services svc2-pro d org services svc3-pro d org services svc1-qa org services svc2-qa org services svc3-qa org services svc1-dev org services svc2-dev org services svc3-dev network cluster services dev qa prod
24. 24 Pulumi Stacks • Pulumi.<stack>.yaml • Configuration for an “instance” of your project • Input values specific to your instance • Namespaces • get/require • Structured configuration
25. 25 Pulumi Stacks • Stack Outputs
26. 26 Pulumi Stacks • Stack References • Can use values from other stacks • Info retrieved from the state backend
27. 27 Pulumi at Scale org/network/dev org/network/qa org/network/prod org/cluster/dev org/cluster/qa org/cluster/prod org services svc1-pro d org services svc2-pro d org services svc3-pro d org services svc1-qa org services svc2-qa org services svc3-qa org services svc1-dev org services svc2-dev org services svc3-dev network cluster services dev qa prod vpc-out-sg service-sg clust-sg-rule svc2-service-rule svc2-out-rule
28. 28 3. Growing your portfolio A tail of service (over)load
29. 29 Pulumi - ComponentResource Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start Network + children provider: aws Start CREATE …
30. 30 Pulumi - Multi Language Component Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start org.nw.Network provider: aws Start CREATE … MLC (e.g. org) Start Nw resources
31. 31 4. Any protection? Security, auditing & certification
32. 32 Policy as Code What is it? • Enforce security, compliance and best practices • Use general purpose programming languages* • Write expressive policies • Enforce gated deployments • Meet business requirements
33. 33 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types
34. 34 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types
35. 35 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types
36. 36 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types
37. 37 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types
38. 38 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types
39. 39 Policy as Code Data protection import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], });
40. 40 Policy as Code Data protection import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], });
41. 41 Policy as Code Data protection import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], });
42. 42 Policy as Code Cost optimization import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-cost", { policies: [ { name: "expire-old-logs", description: "Expire old logs.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { bucket.lifecycleRules?.forEach(rule => { if (! rule.enabled || !rule.expiration || !rule.expiration.days || rule.expiration.days < 45) { reportViolation("Bucket logs must an expiration days < 45."); } }); }), }, ], });
43. 43 Policy as Code Reliability import * as k8s from "@pulumi/kubernetes"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("k8s-reliability", { policies: [ { name: "minimum-replica-count", description: "Checks that Kubernetes Deployments and ReplicaSets have at least three replicas.", enforcementLevel: "mandatory", validateResource: [ validateResourceOfType(k8s.apps.v1.Deployment, (deployment, _, reportViolation) => { if (!deployment.spec || !deployment.spec.replicas || deployment.spec.replicas < 3) { reportViolation("Kubernetes Deployments should have at least three replicas."); } }), validateResourceOfType(k8s.apps.v1.ReplicaSet, (replicaSet, _, reportViolation) => { if (!replicaSet.spec || !replicaSet.spec.replicas || replicaSet.spec.replicas < 3) { reportViolation("Kubernetes ReplicaSet should have at least three replicas."); } }), ], }, ], });
44. 44 Policy as Code Best Practices Learn, codify, share and automatically reapply. import * as k8s from "@pulumi/kubernetes"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("k8s-best", { policies: [ { name: "pods-are-prohibited", description: "Checks that Kubernetes Pods are not being used directly.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(k8s.core.v1.Pod, (_, __, reportViolation) => { reportViolation("Kubernetes Pods should not be used directly. " + "Instead, you may want to use a Deployment, ReplicaSet or Job."); }), }, ], });
45. 45 Policy Remediations Don’t just issue policy violations, fix them automatically • Automatically remediate configuration violations at deployment time • Add `transform` method to policy pack and Pulumi IaC will enforce it • Available in the open source IaC with organization-wide enforcement for Pulumi Business Critical customers Examples • Apply tags to resources automatically • Remove public Internet access from firewalls and gateways • Apply encryption and backups to storage New
46. 46 CrossGuard What is it? • Author & Publish Policy Packs • Policy Pack versioning • Applies to any runtime (mix and match) • Policy packs and stack grouping
47. 47 CrossGuard Policy validation • Fetch Policy packs from the Pulumi Service (SaaS & Self-Hosted) • Check resource properties (graph) • Evaluation before deployment Benefits • Catch problem before they reach production • Educate developers on best practices
48. 48 CrossGuard Policy Pack configuration • Policy pack version selection • Per pack and per rule override
49. 49 Compliance-Ready Policies 100s of new out of the box policies • PCI DSS, ISO 27001, SOC 2, CIS • All Pulumi Languages and Clouds supported • Build custom policy packs. Choose cloud provider X service X severity X compliance framework New
50. 50 5. Growing your company More teams & self-service
52. 52 Pulumi Developer Portal Out of the box golden path to production for developers • Private organization templates • Simple user interface to browse templates and deploy them through a new project wizard • Deploy via Pulumi CLI or Pulumi Deployments • Integrate with Git for instant project creation • Pulumi ESC for secrets/config • Extensible with Pulumi Cloud REST API New
53. 53 Pulumi Backstage • Integrate Pulumi into CNCF Backstage Portals ◦ Deploy infrastructure with Pulumi programs ◦ Use private templates from Pulumi Developer Portal • Open Source • Available in Backstage Marketplace https://backstage.io/plugins New
54. 54 Pulumi ESC ENVIRONMENTS, SECRETS, AND CONFIG MADE SECURE AND EASY New
55. 55 Pulumi ESC Architecture EXTENSIBLE WITH DOZENS OF SOURCES AND TARGETS ESC Environments (Hierarchies of Config/Secrets) 🔐Native Secrets Third party secrets: (and more…) Sources Native CLI Execution environments: (and more…) Targets # shopping-service-production imports: - aws-production - shopping-service # aws-production config: aws: provider: aws-oidc inputs: roleArn: …arn… sessionName: aws-prod # shopping-service config: stripeKey: value: ***** secret: true dbPassword: provider: workers-kv inputs: Auth Tokens RBAC Identity Managed or OSS Encryption Audit Logs Web UX Policy Versioning Sync (future) New
56. 56 Pulumi IaC Under the Hood Example # shopping-service-production imports: - aws-production - shopping-service values: dbPassword: ABCDEF # aws-production values: aws: provider: aws-oidc inputs: roleArn: …arn… sessionName: aws-prod # shopping-service values: stripeKey: value: ***** secret: true dbPassword: provider: 1password inputs: # Pulumi.prod.yaml environment: - shopping-service-production $ esc run acme/shopping-service-production -- aws s3 ls import * as esc from ‘pulumi-esc’; const env = new esc.Environment(...); const config = env.open("shopping-service-production"); env.watch("shopping-service-production", config => …); Consuming from: CLI SDK AWS 1password
57. 57 Closing thoughts
58. 58 Migrate from Terraform to Pulumi HCL $ git checkout <your-hcl> $ pulumi convert --from terraform --language typescript --out pulumi Blog State $ pulumi stack init $ pulumi import --from terraform ./terraform.tfstate Blog New
59. 59 Finding help!
60. 60 Pulumi AI!
61. 61 Thank you! https://www.pulumi.com