Slide 1

Slide 1 text

TechMeetup Conference 2023 Ostrava 3 November 2023 Ringo De Smet Mastodon: @[email protected] Twitter/X: @ringods

Slide 2

Slide 2 text

2 On-Premises Datacenters (Server-Centric) Lift-and-Shift Migrations to Cloud (Server-Centric) D ocker AW S Lam bda TODAY Cloud Engineering AW S EC 2 (IaaS) R estful A P Is TIPPING POINT: Legacy tools and processes were not designed for modern applications and cloud services, stifling innovation. G itH ub Web-Scale IT (Server-Centric) Modern Cloud-Native Apps (Service-Centric) D evO ps Infrastructure-as-Code Cloud-as-Software Managing Servers Focus on Building Infrastructure & Operating IT Managing Services Focus on Composing & Consuming Cloud Services Server Config Management M icroservices Kubernetes A zure G C P 2010 2000 2023 The Rise of Modern Cloud Cloud complexity is escalating

Slide 3

Slide 3 text

3 Yesterday’s tools and practices don’t scale in the modern cloud era Legacy Tools Built for the speed and scale of yesterday Modern Tools Built for high speed and massive scale

Slide 4

Slide 4 text

4 Getting Started A first full environ- ment Growing your portfolio Any protection ? Growing your company How does Pulumi support you in your growth?

Slide 5

Slide 5 text

5 1. Getting Started Understanding the (power of) the product

Slide 6

Slide 6 text

6 Multi-Cloud Infrastructure as Code BUILD, DEPLOY, AND MANAGE ANYWHERE WITH A STANDARD WORKFLOW Infra Providers Cloud Native Clouds Core Features: ● Any Cloud, Any Language ● State Management ● Secrets Management ● Guaranteed Preview Plans ● CI/CD Integrations ● Webhooks ● REST API ● Automation API ● Dashboards and Reports

Slide 7

Slide 7 text

7 Pulumi - High Level View • Pulumi Architecture • Language Host • CLI & Engine • Providers • State Backend •Pulumi Service •Cloud Storage (S3, …)

Slide 8

Slide 8 text

8 Pulumi - High Level View • Automate your infrastructure setup with 1 or more Pulumi projects • A project contains • a program: this is your “template” from which you can create stacks • the stacks: each stack is an “instance” of the program • One project can use info from another project as input

Slide 9

Slide 9 text

9 How Pulumi Works - first run Pulumi CLI & Engine $ pulumi up

Slide 10

Slide 10 text

10 How Pulumi Works - first run Language Host Pulumi CLI & Engine $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup

Slide 11

Slide 11 text

11 How Pulumi Works - first run Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CREATE SG

Slide 12

Slide 12 text

12 How Pulumi Works - first run Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CREATE SG web-sg: … Actual SG info

Slide 13

Slide 13 text

13 How Pulumi Works - second run Pulumi CLI & Engine $ pulumi up web-sg: …

Slide 14

Slide 14 text

14 How Pulumi Works - second run Language Host Pulumi CLI & Engine $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup web-sg: …

Slide 15

Slide 15 text

15 How Pulumi Works - second run Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CHECK SG web-sg: …

Slide 16

Slide 16 text

16 How Pulumi Works - multiple resources Pulumi CLI & Engine $ pulumi up

Slide 17

Slide 17 text

17 How Pulumi Works - multiple resources Language Host Pulumi CLI & Engine $ pulumi up runtime: nodejs Start SG & Instance

Slide 18

Slide 18 text

18 How Pulumi Works - multiple resources Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start SG & Instance provider: aws Start CREATE SG & CREATE Instance

Slide 19

Slide 19 text

19 How Pulumi Works - dependencies Pulumi CLI & Engine $ pulumi up

Slide 20

Slide 20 text

20 How Pulumi Works - dependencies Language Host Pulumi CLI & Engine $ pulumi up runtime: nodejs Start SG & Instance

Slide 21

Slide 21 text

21 How Pulumi Works - dependencies Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start SG & Instance provider: aws Start CREATE SG CREATE Instance Wait

Slide 22

Slide 22 text

22 2. Your first full environment The journey from network to services

Slide 23

Slide 23 text

23 Pulumi at Scale org/network/dev org/network/qa org/network/prod org/cluster/dev org/cluster/qa org/cluster/prod org services svc1-pro d org services svc2-pro d org services svc3-pro d org services svc1-qa org services svc2-qa org services svc3-qa org services svc1-dev org services svc2-dev org services svc3-dev network cluster services dev qa prod

Slide 24

Slide 24 text

24 Pulumi Stacks • Pulumi..yaml • Configuration for an “instance” of your project • Input values specific to your instance • Namespaces • get/require • Structured configuration

Slide 25

Slide 25 text

25 Pulumi Stacks • Stack Outputs

Slide 26

Slide 26 text

26 Pulumi Stacks • Stack References • Can use values from other stacks • Info retrieved from the state backend

Slide 27

Slide 27 text

27 Pulumi at Scale org/network/dev org/network/qa org/network/prod org/cluster/dev org/cluster/qa org/cluster/prod org services svc1-pro d org services svc2-pro d org services svc3-pro d org services svc1-qa org services svc2-qa org services svc3-qa org services svc1-dev org services svc2-dev org services svc3-dev network cluster services dev qa prod vpc-out-sg service-sg clust-sg-rule svc2-service-rule svc2-out-rule

Slide 28

Slide 28 text

28 3. Growing your portfolio A tail of service (over)load

Slide 29

Slide 29 text

29 Pulumi - ComponentResource Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start Network + children provider: aws Start CREATE …

Slide 30

Slide 30 text

30 Pulumi - Multi Language Component Language Host Pulumi CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start org.nw.Network provider: aws Start CREATE … MLC (e.g. org) Start Nw resources

Slide 31

Slide 31 text

31 4. Any protection? Security, auditing & certification

Slide 32

Slide 32 text

32 Policy as Code What is it? • Enforce security, compliance and best practices • Use general purpose programming languages* • Write expressive policies • Enforce gated deployments • Meet business requirements

Slide 33

Slide 33 text

33 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types

Slide 34

Slide 34 text

34 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types

Slide 35

Slide 35 text

35 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types

Slide 36

Slide 36 text

36 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types

Slide 37

Slide 37 text

37 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types

Slide 38

Slide 38 text

38 Policy as Code import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types

Slide 39

Slide 39 text

39 Policy as Code Data protection import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], });

Slide 40

Slide 40 text

40 Policy as Code Data protection import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], });

Slide 41

Slide 41 text

41 Policy as Code Data protection import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], });

Slide 42

Slide 42 text

42 Policy as Code Cost optimization import * as aws from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-cost", { policies: [ { name: "expire-old-logs", description: "Expire old logs.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { bucket.lifecycleRules?.forEach(rule => { if (! rule.enabled || !rule.expiration || !rule.expiration.days || rule.expiration.days < 45) { reportViolation("Bucket logs must an expiration days < 45."); } }); }), }, ], });

Slide 43

Slide 43 text

43 Policy as Code Reliability import * as k8s from "@pulumi/kubernetes"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("k8s-reliability", { policies: [ { name: "minimum-replica-count", description: "Checks that Kubernetes Deployments and ReplicaSets have at least three replicas.", enforcementLevel: "mandatory", validateResource: [ validateResourceOfType(k8s.apps.v1.Deployment, (deployment, _, reportViolation) => { if (!deployment.spec || !deployment.spec.replicas || deployment.spec.replicas < 3) { reportViolation("Kubernetes Deployments should have at least three replicas."); } }), validateResourceOfType(k8s.apps.v1.ReplicaSet, (replicaSet, _, reportViolation) => { if (!replicaSet.spec || !replicaSet.spec.replicas || replicaSet.spec.replicas < 3) { reportViolation("Kubernetes ReplicaSet should have at least three replicas."); } }), ], }, ], });

Slide 44

Slide 44 text

44 Policy as Code Best Practices Learn, codify, share and automatically reapply. import * as k8s from "@pulumi/kubernetes"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("k8s-best", { policies: [ { name: "pods-are-prohibited", description: "Checks that Kubernetes Pods are not being used directly.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(k8s.core.v1.Pod, (_, __, reportViolation) => { reportViolation("Kubernetes Pods should not be used directly. " + "Instead, you may want to use a Deployment, ReplicaSet or Job."); }), }, ], });

Slide 45

Slide 45 text

45 Policy Remediations Don’t just issue policy violations, fix them automatically ● Automatically remediate configuration violations at deployment time ● Add `transform` method to policy pack and Pulumi IaC will enforce it ● Available in the open source IaC with organization-wide enforcement for Pulumi Business Critical customers Examples ● Apply tags to resources automatically ● Remove public Internet access from firewalls and gateways ● Apply encryption and backups to storage New

Slide 46

Slide 46 text

46 CrossGuard What is it? • Author & Publish Policy Packs • Policy Pack versioning • Applies to any runtime (mix and match) • Policy packs and stack grouping

Slide 47

Slide 47 text

47 CrossGuard Policy validation • Fetch Policy packs from the Pulumi Service (SaaS & Self-Hosted) • Check resource properties (graph) • Evaluation before deployment Benefits • Catch problem before they reach production • Educate developers on best practices

Slide 48

Slide 48 text

48 CrossGuard Policy Pack configuration • Policy pack version selection • Per pack and per rule override

Slide 49

Slide 49 text

49 Compliance-Ready Policies 100s of new out of the box policies ● PCI DSS, ISO 27001, SOC 2, CIS ● All Pulumi Languages and Clouds supported ● Build custom policy packs. Choose cloud provider X service X severity X compliance framework New

Slide 50

Slide 50 text

50 5. Growing your company More teams & self-service

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

52 Pulumi Developer Portal Out of the box golden path to production for developers ● Private organization templates ● Simple user interface to browse templates and deploy them through a new project wizard ● Deploy via Pulumi CLI or Pulumi Deployments ● Integrate with Git for instant project creation ● Pulumi ESC for secrets/config ● Extensible with Pulumi Cloud REST API New

Slide 53

Slide 53 text

53 Pulumi Backstage ● Integrate Pulumi into CNCF Backstage Portals ○ Deploy infrastructure with Pulumi programs ○ Use private templates from Pulumi Developer Portal ● Open Source ● Available in Backstage Marketplace https://backstage.io/plugins New

Slide 54

Slide 54 text

54 Pulumi ESC ENVIRONMENTS, SECRETS, AND CONFIG MADE SECURE AND EASY New

Slide 55

Slide 55 text

55 Pulumi ESC Architecture EXTENSIBLE WITH DOZENS OF SOURCES AND TARGETS ESC Environments (Hierarchies of Config/Secrets) 🔐Native Secrets Third party secrets: (and more…) Sources Native CLI Execution environments: (and more…) Targets # shopping-service-production imports: - aws-production - shopping-service # aws-production config: aws: provider: aws-oidc inputs: roleArn: …arn… sessionName: aws-prod # shopping-service config: stripeKey: value: ***** secret: true dbPassword: provider: workers-kv inputs: Auth Tokens RBAC Identity Managed or OSS Encryption Audit Logs Web UX Policy Versioning Sync (future) New

Slide 56

Slide 56 text

56 Pulumi IaC Under the Hood Example # shopping-service-production imports: - aws-production - shopping-service values: dbPassword: ABCDEF # aws-production values: aws: provider: aws-oidc inputs: roleArn: …arn… sessionName: aws-prod # shopping-service values: stripeKey: value: ***** secret: true dbPassword: provider: 1password inputs: # Pulumi.prod.yaml environment: - shopping-service-production $ esc run acme/shopping-service-production -- aws s3 ls import * as esc from ‘pulumi-esc’; const env = new esc.Environment(...); const config = env.open("shopping-service-production"); env.watch("shopping-service-production", config => …); Consuming from: CLI SDK AWS 1password

Slide 57

Slide 57 text

57 Closing thoughts

Slide 58

Slide 58 text

58 Migrate from Terraform to Pulumi HCL $ git checkout $ pulumi convert --from terraform --language typescript --out pulumi Blog State $ pulumi stack init $ pulumi import --from terraform ./terraform.tfstate Blog New

Slide 59

Slide 59 text

59 Finding help!

Slide 60

Slide 60 text

60 Pulumi AI!

Slide 61

Slide 61 text

61 Thank you! https://www.pulumi.com