Upgrade to Pro — share decks privately, control downloads, hide ads and more …

VMworld 2016 INF8631 - VMware Certificate Management for Mere Mortals

2b60f038e196add9756714ba505b3189?s=47 Ryan Johnson
September 01, 2016

VMworld 2016 INF8631 - VMware Certificate Management for Mere Mortals

Presented at VMworld 2016
Recording at http://vmware.mediasite.com/mediasite/Play/c221061b371b41dcb37774f0d734c58e1d?catalog=dbf1ec28-2557-4dd3-a381-e5fe4ceabc40

SSL certificates have become essential in securing access to our data centers and infrastructure. As VMware vSphere has evolved, so too have the processes, applications, and management of SSL certificates. Gone are the days of simply relying on self-signed certificates and never having to validate, renew, or really manage those certificates. In this session, we will discuss and demonstrate the new VMware Certificate Authority to show how to deploy and manage SSL certificates in vSphere 6. We'll also discuss upgrade considerations and how to evaluate the different deployment models for the VMware Certificate Authority in order to decide which is best for the user's organization.


Ryan Johnson

September 01, 2016


  1. VMware Certificate Management for Mere Mortals Adam Eckerle, VMware, Inc

    Ryan Johnson, VMware, Inc INF8631 #INF8631
  2. • This presentation may contain product features that are currently

    under development. • This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. • Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Technical feasibility and market demand will affect final delivery. • Pricing and packaging for any new technologies or features discussed or presented have not been determined. Disclaimer CONFIDENTIAL 2
  3. 3

  4. Certificate Lifecycle Management 4 VMware Certificate Authority VMCA VMware Endpoint

    Certificate Store VECS Located on: Embedded Deployment and Platform Services Controller Located on: Embedded Deployment and vCenter Server Node
  5. VMware Certificate Authority (VMCA) 5 Root CA • During installation,

    VMCA automatically creates a root CA certificate. • This certificate is capable of issuing other certificates. • All solutions and endpoint certificates are created and trusted through to this certificate. Issuer CA • Can replace the default root CA certificate created during installation. • Requires a CSR issued from VMCA to be used by an enterprise or 3rd party CA to generate a new issuing certificate. • Requires replacement of all issued default certificates after implementation.
  6. VMware Endpoint Certificate Store (VECS) § Repository for Certificates and

    Private Keys § Mandatory Component § Key Stores: – Machine SSL Certificates – Trusted Roots – Solution Users Certificates § Generally managed via Certificate Manager § vecs-cli available for more advanced operations or automation § Does Not Manage Single Sign-On Certificates 6
  7. VECS VMCA VMware Endpoint Certificate Store (VECS) 7 Signed VMCA

    Certificate Machine SSL Certificate
  8. VMware vSphere 6.0 Certificate Types § ESXi Certificates § Machine

    SSL Certificate § Solution User Certificates § Single Sign-On Certificates 8
  9. ESXi Certificates 9 § Post-install, ESXi always has an auto-generated

    certificate § VMCA will provision a signed certificate when host is joined to vCenter (default mode) § Custom certificates can be use if desired (custom mode) § ESXi certificates are stored locally on each host in the /etc/vmware/ssl § VMCA issued certificates can be renewed via the vSphere Web Client or PowerCLI
  10. ESXi Certificates 10 Example: function refreshcerts { process { $hostid

    = Get-VMHost $vmhost | Get-View $hostParam = New-Object VMware.Vim.ManagedObjectReference[] (1) $hostParam[0] = New-Object VMware.Vim.ManagedObjectReference $hostParam[0].value = $hostid.moref.value $hostParam[0].type = 'HostSystem' $_this = Get-View -Id 'CertificateManager-certificateManager' $_this.CertMgrRefreshCertificates_Task($hostParam) } }
  11. Machine SSL Certificates § Creates a server-side SSL socket §

    Server verification and secure communication e.g. HTTPS or LDAPS § Each node has its own Machine SSL Certificate. i.e. Embedded Deployment; Management Node; or Platform Services Controller § All services use a Machine SSL Certificate for endpoint encryption. § All services communicate through the reverse proxy § Traffic does not go to the services themselves e.g. The vpxd service uses the MACHINE_SSL_CERT to expose its endpoint. 11
  12. Solution User Certificates Located in VECS: § machine – Used

    by component manager, license server, and the logging service § vpxd – vCenter service daemon (vpxd) store on management nodes and embedded deployments. vpxd uses the solution user certificate to authenticate to vCenter Single Sign-On § vpxd-extensions – Includes the Auto Deploy service, inventory service, and other services that are not part of other solution users § vsphere-webclient – Includes the vSphere Web Client and some additional services such as the performance chart service 12
  13. Solution User Certificates § Encapsulates one or more vCenter Server

    services § Certificate authenticated by vCenter Single Sign-On and issued a SAML token to authenticate to other solution user and services § Each solution user must be authenticated to vCenter Single Sign-On § Re-authentication occurs after a reboot and after a timeout § The timeout configurable in the vSphere Web Client and defaults to 2592000 seconds (30 days) 13 30 DAYS
  14. Single Sign-On Certificates VMware Directory Service SSL Certificate – §

    With custom certificates you may need to replace this SSL certificate explicitly. VMware vCenter Single Sign-On Signing Certificate – § Security Token Service (STS) – an identity provider that issues, validates, and renews SAML tokens that are used for authentication throughout vSphere § By default, the STS signing certificate is generated by VMCA § Manually refresh STS certificate via vSphere Web Client when the certificate expires or changes 14
  15. Single Sign-On Certificates § Not stored in VECS § Not

    managed with certificate management tools § Changes are not necessary, but in special situations, you can replace these certificates 15
  16. Certificate Replacement Options for vCenter 16 VMCA Default ••VMCA provides

    the Root certificate ••All vSphere certificates chain to VMCA ••Regenerate certificates on demand easily VMCA Enterprise ••Replace VMCA CA cert with a subordinate CA certificate from the Enterprise PKI ••Upon removal of the old VMCA CA certificate, all old certificates will be regenerated Custom ••Disable VMCA as CA ••Provision your own custom certificates for each solution user and endpoint ••More complicated For highly security conscious customers only Hybrid ••Replacement of the Machine_SSL certs ••VMCA for Hosts and Solution Users ••Very popular with high security customers ••Recommended
  17. VMware vSphere 6.0 Certificate Manager 17 Appliance Deployment /usr/lib/vmware-vmca/ bin/certificate-manager

    Windows Deployment <Drive>:\Program Files\VMware\ vCenter Server\vmcad\ certificate-manager
  18. Common Certificate Manager Use Cases 18 VMCA as Root CA

    (Default or Option 4) VMCA as Enterprise CA Subordinate (Option 2) Custom CA (Option 1 & 5) Hybrid (Option 1 & 6)
  19. VMCA as Root CA 19 VMware KB 2108294

  20. VMCA as Enterprise CA Subordinate Private Key Algorithm: RSA with

    2048 bits. Standard: X.509 v3 Format: PEM (PKCS8 and PKCS1) with a header of ---BEGIN CERTIFICATE--- Recommended Signature Algorithms: SHA256, SHA384, or SHA512 § Does NOT support wildcard cards or SubjectAltName § You CANNOT create subsidiary CAs of VMCA § No explicit limit to the length of the certificate chain § Synchronize time for all nodes in environment 20
  21. But… 21

  22. Hybrid Approach 22

  23. Custom certificates for the Web Client VMCA for everything else

    (User Solutions, ESX hosts) Hybrid Approach Concepts 23 Operations Security
  24. Hybrid Mode: 3rd Party Cert for Web Client access 24

    What many security concerned companies are using for their vSphere environments 3rd Party Certificate Authority DC1.lab.local VCSA vCenter Server https://vcsa.lab.local https://esxi-a.lab.local SSL Certificate issued by DC1.lab.local Certificate Authority SSL Certificate issued by vcsa.lab.local VMCA
  25. Implementing Hybrid Mode (High Level) 25 1. Use Option 1

    to: • Replace Machine_SSL cert on all PSCs in SSO Domain • Replace Machine_SSL cert on all vCenter Servers in SSO domain 2. Use option 6 to: • Replace everything else on PSC & vCenter Server nodes 3. Use Web Client to replace certs on ESX hosts
  26. ESXi Certificate Management Modes 26 VMware ESXi 6.0 VMCA Authority

    Mode Custom Mode Thumbprint Mode
  27. 27 Default Value = vmca Possible Values = vmca |

    custom | thumbprint Search for certmgmt
  28. VMCA Authority Mode § The default mode § Post-install ESXi

    always has an auto-generated certificate § ESXi certificates are stored locally on each host in the /etc/vmware/ssl § VMCA provisions the host a signed certificate when added to vCenter Server § Host certificates include the full chain to VMCA § ESXi certificates can be renewed via the vSphere Web Client or PowerCLI 28 vpxd.certmgmt.mode = vmca
  29. Custom Mode § Replacement is the same as vSphere 5.5

    – ESXi Shell – HTTPS GET/PUT § vifs will wrap these operations. § Custom / 3rd Party certificates – Must change vpxd.certmgmt.mode to custom or risk replacement by VMCA – Must update TRUSTED_ROOTS store in VECS on vCenter with the custom root certificates to ensure trust relationship – use the vecs-cli entry create command 29 vpxd.certmgmt.mode = custom
  30. Thumbprint Mode § Legacy mode § Fallback option for vSphere

    6.0 § May be used to retains vSphere 5.5 certificates during an upgrade § DO NOT use this mode unless encountering issues with vmca or custom mode § vCenter 6.0 and later services may not work correctly in thumbprint mode § Switching from thumbprint to vmca mode requires extensive planning 30 vpxd.certmgmt.mode = thumbprint
  31. Walkthrough

  32. Detailed Process – Certificate Manager 32

  33. Detailed Process – Generate CSR 33

  34. Detailed Process – Copy CSR Files 34

  35. Detailed Process – Copy Certificate 35

  36. Detailed Process – Submit CSR 36

  37. Detailed Process – Download Certificate & Chain 37

  38. Detailed Process – Export Root Certificate 38

  39. Detailed Process – Upload Root & New Certificate 39

  40. Detailed Process – Replace MACHINE_SSL Certificate 40

  41. Detailed Process – Certificate Replacement Completed 41

  42. Detailed Process – Validation 42

  43. Detailed Process – ESXi Hosts 43

  44. Detailed Process – Script for Entire Clusters 44

  45. Upgrades and Operational Considerations VMware vSphere 6.0 Certificate Management

  46. Deployment Considerations § VMCA as Enterprise CA Subordinate – Perform

    the signing certificate replacement on all Platform Services Controllers to ensure trusted certificates for all vCenter Server 6.0 installations • The ‘24 Hour Rule’ – Signing certificate must have a valid date of 24 hours prior before renewing host certificates or adding new hosts to vCenter Server – Plan for this aging period when configuring an existing environment – Replace certificates prior to putting a new environment into production – 6.0 U2 adds vpxd.certmgmt.certs.minutesBefore parameter to mitigate this design characteristic 46
  47. Managing Certificates • Supports replacing certificates • No CRL enforcement

    against PKI for vCenter Server and ESXi hosts • If you suspect that one of your certificates has been compromised, revoke and replace all existing certificates, including the VMCA root certificate • If you do not remove revoked certificates, a man-in-the-middle attack might enable compromise through impersonation with the account's credentials. 47
  48. Upgrades & Auto Deploy § Host Upgrades and VMCA Signed

    Certificates – Upgrade process replaces self-signed certificates with VMCA-signed certificates – vCenter then monitors certificates and displays details vSphere Web Client § Host Upgrades and Custom Certificates – Custom certificates are retained – even if expired or invalid – Change vxd.certmgmt.mode to custom to ensure certificates are not replaced accidentally § Update Manager – Not compatible with the Machine SSL certificate template in vSphere 6.0. § Use the vSphere 5.5 certificate template for Update Manager 6.0 48
  49. VMware Validated Designs


  51. Resources • VMworld 2015 Presentation, scripts, videos – http://tenthrityam.org/INF4529 •

    Mike Foley’s Blog – http://www.vmware.com/go/hybridvmca • Hands On Labs – HOL-SDC-1610 • VMware vCenter Server Appliance 6.0 Poster – https://blogs.vmware.com/vsphere • Platform Services Controller 6.0 Topology Decision Tree – https://vmware.com/go/psctree • VMware Validated Designs – http://vmware.com/go/vvd • Twitter - @eck79, @tenthirtyam 51
  52. None
  53. None