VMworld 2016 INF8631 - VMware Certificate Management for Mere Mortals

Presented at VMworld 2016
Recording at http://vmware.mediasite.com/mediasite/Play/c221061b371b41dcb37774f0d734c58e1d?catalog=dbf1ec28-2557-4dd3-a381-e5fe4ceabc40

SSL certificates have become essential in securing access to our data centers and infrastructure. As VMware vSphere has evolved, so too have the processes, applications, and management of SSL certificates. Gone are the days of simply relying on self-signed certificates and never having to validate, renew, or really manage those certificates. In this session, we will discuss and demonstrate the new VMware Certificate Authority to show how to deploy and manage SSL certificates in vSphere 6. We'll also discuss upgrade considerations and how to evaluate the different deployment models for the VMware Certificate Authority in order to decide which is best for the user's organization.

Transcript

1. VMware Certificate Management for Mere Mortals
   Adam Eckerle, VMware, Inc
   Ryan Johnson, VMware, Inc
   INF8631
   #INF8631
   

   View Slide

2. • This presentation may contain product features that are currently under development.
   • This overview of new technology represents no commitment from VMware to deliver these
   features in any generally available product.
   • Features are subject to change, and must not be included in contracts, purchase orders, or
   sales agreements of any kind.
   • Technical feasibility and market demand will affect final delivery.
   • Pricing and packaging for any new technologies or features discussed or presented have not
   been determined.
   Disclaimer
   CONFIDENTIAL 2
   

   View Slide

3. View Slide

4. Certificate Lifecycle Management
   4
   VMware
   Certificate
   Authority
   VMCA
   VMware
   Endpoint
   Certificate
   Store
   VECS
   Located on:
   Embedded Deployment and
   Platform Services Controller
   Located on:
   Embedded Deployment and
   vCenter Server Node
   

   View Slide

5. VMware Certificate Authority (VMCA)
   5
   Root CA
   • During installation, VMCA
   automatically creates a root CA
   certificate.
   • This certificate is capable of issuing
   other certificates.
   • All solutions and endpoint
   certificates are created and trusted
   through to this certificate.
   Issuer CA
   • Can replace the default root CA
   certificate created during
   installation.
   • Requires a CSR issued from VMCA
   to be used by an enterprise or 3rd
   party CA to generate a new issuing
   certificate.
   • Requires replacement of all issued
   default certificates after
   implementation.
   

   View Slide

6. VMware Endpoint Certificate Store (VECS)
   § Repository for Certificates and Private Keys
   § Mandatory Component
   § Key Stores:
   – Machine SSL Certificates
   – Trusted Roots
   – Solution Users Certificates
   § Generally managed via Certificate Manager
   § vecs-cli available for more advanced operations or automation
   § Does Not Manage Single Sign-On Certificates
   6
   

   View Slide

7. VECS
   VMCA
   VMware Endpoint Certificate Store (VECS)
   7
   Signed
   VMCA
   Certificate
   Machine SSL
   Certificate
   

   View Slide

8. VMware vSphere 6.0 Certificate Types
   § ESXi Certificates
   § Machine SSL Certificate
   § Solution User Certificates
   § Single Sign-On Certificates
   8
   

   View Slide

9. ESXi Certificates
   9
   § Post-install, ESXi always has an auto-generated
   certificate
   § VMCA will provision a signed certificate when
   host is joined to vCenter (default mode)
   § Custom certificates can be use if desired (custom
   mode)
   § ESXi certificates are stored locally on each host in
   the /etc/vmware/ssl
   § VMCA issued certificates can be renewed via the
   vSphere Web Client or PowerCLI
   

   View Slide

10. ESXi Certificates
    10
    Example:
    function refreshcerts {
    process {
    $hostid = Get-VMHost $vmhost | Get-View
    $hostParam = New-Object VMware.Vim.ManagedObjectReference[] (1)
    $hostParam[0] = New-Object VMware.Vim.ManagedObjectReference
    $hostParam[0].value = $hostid.moref.value
    $hostParam[0].type = 'HostSystem'
    $_this = Get-View -Id 'CertificateManager-certificateManager'
    $_this.CertMgrRefreshCertificates_Task($hostParam)
    }
    }
    

    View Slide

11. Machine SSL Certificates
    § Creates a server-side SSL socket
    § Server verification and secure communication
    e.g. HTTPS or LDAPS
    § Each node has its own Machine SSL Certificate.
    i.e. Embedded Deployment; Management Node; or Platform Services
    Controller
    § All services use a Machine SSL Certificate for endpoint
    encryption.
    § All services communicate through the reverse proxy
    § Traffic does not go to the services themselves
    e.g. The vpxd service uses the MACHINE_SSL_CERT to expose its endpoint.
    11
    

    View Slide

12. Solution User Certificates
    Located in VECS:
    § machine – Used by component manager, license server, and
    the logging service
    § vpxd – vCenter service daemon (vpxd) store on management
    nodes and embedded deployments. vpxd uses the solution
    user certificate to authenticate to vCenter Single Sign-On
    § vpxd-extensions – Includes the Auto Deploy service, inventory
    service, and other services that are not part of other solution
    users
    § vsphere-webclient – Includes the vSphere Web Client and
    some additional services such as the performance chart
    service
    12
    

    View Slide

13. Solution User Certificates
    § Encapsulates one or more vCenter Server
    services
    § Certificate authenticated by vCenter Single
    Sign-On and issued a SAML token to
    authenticate to other solution user and services
    § Each solution user must be authenticated to
    vCenter Single Sign-On
    § Re-authentication occurs after a reboot and
    after a timeout
    § The timeout configurable in the vSphere Web
    Client and defaults to 2592000 seconds (30
    days) 13
    30 DAYS
    

    View Slide

14. Single Sign-On Certificates
    VMware Directory Service SSL Certificate –
    § With custom certificates you may need to replace this
    SSL certificate explicitly.
    VMware vCenter Single Sign-On Signing Certificate –
    § Security Token Service (STS) – an identity provider that
    issues, validates, and renews SAML tokens that are used
    for authentication throughout vSphere
    § By default, the STS signing certificate is generated by
    VMCA
    § Manually refresh STS certificate via vSphere Web
    Client when the certificate expires or changes
    14
    

    View Slide

15. Single Sign-On Certificates
    § Not stored in VECS
    § Not managed with certificate management tools
    § Changes are not necessary, but in special
    situations, you can replace these certificates
    15
    

    View Slide

16. Certificate Replacement Options for vCenter
    16
    VMCA Default
    ••VMCA provides
    the Root
    certificate
    ••All vSphere
    certificates chain
    to VMCA
    ••Regenerate
    certificates on
    demand easily
    VMCA Enterprise
    ••Replace VMCA CA
    cert with a
    subordinate CA
    certificate from the
    Enterprise PKI
    ••Upon removal of
    the old VMCA CA
    certificate, all old
    certificates will be
    regenerated
    Custom
    ••Disable VMCA as
    CA
    ••Provision your
    own custom
    certificates for
    each solution user
    and endpoint
    ••More complicated
    For highly security
    conscious
    customers only
    Hybrid
    ••Replacement of
    the Machine_SSL
    certs
    ••VMCA for Hosts
    and Solution Users
    ••Very popular with
    high security
    customers
    ••Recommended
    

    View Slide

17. VMware vSphere 6.0 Certificate Manager
    17
    Appliance Deployment
    /usr/lib/vmware-vmca/
    bin/certificate-manager
    Windows Deployment
    :\Program Files\VMware\
    vCenter Server\vmcad\
    certificate-manager
    

    View Slide

18. Common Certificate Manager Use Cases
    18
    VMCA
    as Root CA
    (Default or
    Option 4)
    VMCA
    as Enterprise
    CA
    Subordinate
    (Option 2)
    Custom CA
    (Option 1 & 5)
    Hybrid
    (Option 1 & 6)
    

    View Slide

19. VMCA as Root CA
    19
    VMware KB 2108294
    

    View Slide

20. VMCA as Enterprise CA Subordinate
    Private Key Algorithm: RSA with 2048 bits.
    Standard: X.509 v3
    Format: PEM (PKCS8 and PKCS1) with a header of ---BEGIN CERTIFICATE---
    Recommended Signature Algorithms: SHA256, SHA384, or SHA512
    § Does NOT support wildcard cards or SubjectAltName
    § You CANNOT create subsidiary CAs of VMCA
    § No explicit limit to the length of the certificate chain
    § Synchronize time for all nodes in environment
    20
    

    View Slide

21. But…
    21
    

    View Slide

22. Hybrid Approach
    22
    

    View Slide

23. Custom certificates for
    the Web Client
    VMCA for everything
    else (User Solutions,
    ESX hosts)
    Hybrid Approach Concepts
    23
    Operations
    Security
    

    View Slide

24. Hybrid Mode: 3rd Party Cert for Web Client access
    24
    What many security concerned companies are using for their vSphere environments
    3rd Party
    Certificate
    Authority
    DC1.lab.local
    VCSA
    vCenter Server
    https://vcsa.lab.local
    https://esxi-a.lab.local
    SSL Certificate
    issued by
    DC1.lab.local
    Certificate
    Authority
    SSL Certificate
    issued by
    vcsa.lab.local VMCA
    

    View Slide

25. Implementing Hybrid Mode (High Level)
    25
    1. Use Option 1 to:
    • Replace Machine_SSL cert
    on all PSCs in SSO Domain
    • Replace Machine_SSL cert
    on all vCenter Servers in
    SSO domain
    2. Use option 6 to:
    • Replace everything else on
    PSC & vCenter Server
    nodes
    3. Use Web Client to replace
    certs on ESX hosts
    

    View Slide

26. ESXi Certificate Management Modes
    26
    VMware ESXi 6.0
    VMCA
    Authority
    Mode
    Custom
    Mode
    Thumbprint
    Mode
    

    View Slide

27. 27
    Default Value = vmca
    Possible Values = vmca | custom |
    thumbprint
    Search for certmgmt
    

    View Slide

28. VMCA Authority Mode
    § The default mode
    § Post-install ESXi always has an auto-generated certificate
    § ESXi certificates are stored locally on each host in the /etc/vmware/ssl
    § VMCA provisions the host a signed certificate when added to vCenter
    Server
    § Host certificates include the full chain to VMCA
    § ESXi certificates can be renewed via the vSphere Web Client or
    PowerCLI
    28
    vpxd.certmgmt.mode = vmca
    

    View Slide

29. Custom Mode
    § Replacement is the same as vSphere 5.5
    – ESXi Shell
    – HTTPS GET/PUT
    § vifs will wrap these operations.
    § Custom / 3rd Party certificates
    – Must change vpxd.certmgmt.mode to custom or risk replacement by VMCA
    – Must update TRUSTED_ROOTS store in VECS on vCenter with the custom root
    certificates to ensure trust relationship – use the vecs-cli entry create
    command
    29
    vpxd.certmgmt.mode = custom
    

    View Slide

30. Thumbprint Mode
    § Legacy mode
    § Fallback option for vSphere 6.0
    § May be used to retains vSphere 5.5 certificates during an upgrade
    § DO NOT use this mode unless encountering issues with vmca or custom
    mode
    § vCenter 6.0 and later services may not work correctly in thumbprint mode
    § Switching from thumbprint to vmca mode requires extensive planning
    30
    vpxd.certmgmt.mode = thumbprint
    

    View Slide

31. Walkthrough
    

    View Slide

32. Detailed Process – Certificate Manager
    32
    

    View Slide

33. Detailed Process – Generate CSR
    33
    

    View Slide

34. Detailed Process – Copy CSR Files
    34
    

    View Slide

35. Detailed Process – Copy Certificate
    35
    

    View Slide

36. Detailed Process – Submit CSR
    36
    

    View Slide

37. Detailed Process – Download Certificate & Chain
    37
    

    View Slide

38. Detailed Process – Export Root Certificate
    38
    

    View Slide

39. Detailed Process – Upload Root & New Certificate
    39
    

    View Slide

40. Detailed Process – Replace MACHINE_SSL Certificate
    40
    

    View Slide

41. Detailed Process – Certificate Replacement Completed
    41
    

    View Slide

42. Detailed Process – Validation
    42
    

    View Slide

43. Detailed Process – ESXi Hosts
    43
    

    View Slide

44. Detailed Process – Script for Entire Clusters
    44
    

    View Slide

45. Upgrades and Operational
    Considerations
    VMware vSphere 6.0 Certificate Management
    

    View Slide

46. Deployment Considerations
    § VMCA as Enterprise CA Subordinate
    – Perform the signing certificate replacement on all Platform Services
    Controllers to ensure trusted certificates for all vCenter Server 6.0
    installations
    • The ‘24 Hour Rule’
    – Signing certificate must have a valid date of 24 hours prior before renewing
    host certificates or adding new hosts to vCenter Server
    – Plan for this aging period when configuring an existing environment
    – Replace certificates prior to putting a new environment into production
    – 6.0 U2 adds vpxd.certmgmt.certs.minutesBefore parameter to mitigate
    this design characteristic
    46
    

    View Slide

47. Managing Certificates
    • Supports replacing certificates
    • No CRL enforcement against PKI for vCenter Server and ESXi hosts
    • If you suspect that one of your certificates has been compromised,
    revoke and replace all existing certificates, including the VMCA root
    certificate
    • If you do not remove revoked certificates, a man-in-the-middle attack
    might enable compromise through impersonation with the account's
    credentials.
    47
    

    View Slide

48. Upgrades & Auto Deploy
    § Host Upgrades and VMCA Signed Certificates
    – Upgrade process replaces self-signed certificates with VMCA-signed certificates
    – vCenter then monitors certificates and displays details vSphere Web Client
    § Host Upgrades and Custom Certificates
    – Custom certificates are retained – even if expired or invalid
    – Change vxd.certmgmt.mode to custom to ensure certificates are not replaced
    accidentally
    § Update Manager
    – Not compatible with the Machine SSL certificate template in vSphere 6.0.
    § Use the vSphere 5.5 certificate template for Update Manager 6.0
    48
    

    View Slide

49. VMware Validated Designs
    

    View Slide

50. CONFIDENTIAL 50
    

    View Slide

51. Resources
    • VMworld 2015 Presentation, scripts, videos
    – http://tenthrityam.org/INF4529
    • Mike Foley’s Blog
    – http://www.vmware.com/go/hybridvmca
    • Hands On Labs – HOL-SDC-1610
    • VMware vCenter Server Appliance 6.0 Poster
    – https://blogs.vmware.com/vsphere
    • Platform Services Controller 6.0 Topology Decision Tree
    – https://vmware.com/go/psctree
    • VMware Validated Designs
    – http://vmware.com/go/vvd
    • Twitter - @eck79, @tenthirtyam
    51
    

    View Slide

52. View Slide

53. View Slide