VMworld 2016 INF8631 - VMware Certificate Management for Mere Mortals

Presented at VMworld 2016
Recording at http://vmware.mediasite.com/mediasite/Play/c221061b371b41dcb37774f0d734c58e1d?catalog=dbf1ec28-2557-4dd3-a381-e5fe4ceabc40

SSL certificates have become essential in securing access to our data centers and infrastructure. As VMware vSphere has evolved, so too have the processes, applications, and management of SSL certificates. Gone are the days of simply relying on self-signed certificates and never having to validate, renew, or really manage those certificates. In this session, we will discuss and demonstrate the new VMware Certificate Authority to show how to deploy and manage SSL certificates in vSphere 6. We'll also discuss upgrade considerations and how to evaluate the different deployment models for the VMware Certificate Authority in order to decide which is best for the user's organization.

Transcript

1. VMware Certificate Management for Mere Mortals Adam Eckerle, VMware, Inc Ryan Johnson, VMware, Inc INF8631 #INF8631
2. • This presentation may contain product features that are currently under development. • This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. • Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Technical feasibility and market demand will affect final delivery. • Pricing and packaging for any new technologies or features discussed or presented have not been determined. Disclaimer CONFIDENTIAL 2
4. Certificate Lifecycle Management 4 VMware Certificate Authority VMCA VMware Endpoint Certificate Store VECS Located on: Embedded Deployment and Platform Services Controller Located on: Embedded Deployment and vCenter Server Node
5. VMware Certificate Authority (VMCA) 5 Root CA • During installation, VMCA automatically creates a root CA certificate. • This certificate is capable of issuing other certificates. • All solutions and endpoint certificates are created and trusted through to this certificate. Issuer CA • Can replace the default root CA certificate created during installation. • Requires a CSR issued from VMCA to be used by an enterprise or 3rd party CA to generate a new issuing certificate. • Requires replacement of all issued default certificates after implementation.
6. VMware Endpoint Certificate Store (VECS) § Repository for Certificates and Private Keys § Mandatory Component § Key Stores: – Machine SSL Certificates – Trusted Roots – Solution Users Certificates § Generally managed via Certificate Manager § vecs-cli available for more advanced operations or automation § Does Not Manage Single Sign-On Certificates 6
7. VECS VMCA VMware Endpoint Certificate Store (VECS) 7 Signed VMCA Certificate Machine SSL Certificate
8. VMware vSphere 6.0 Certificate Types § ESXi Certificates § Machine SSL Certificate § Solution User Certificates § Single Sign-On Certificates 8
9. ESXi Certificates 9 § Post-install, ESXi always has an auto-generated certificate § VMCA will provision a signed certificate when host is joined to vCenter (default mode) § Custom certificates can be use if desired (custom mode) § ESXi certificates are stored locally on each host in the /etc/vmware/ssl § VMCA issued certificates can be renewed via the vSphere Web Client or PowerCLI
10. ESXi Certificates 10 Example: function refreshcerts { process { $hostid = Get-VMHost $vmhost | Get-View $hostParam = New-Object VMware.Vim.ManagedObjectReference[] (1) $hostParam[0] = New-Object VMware.Vim.ManagedObjectReference $hostParam[0].value = $hostid.moref.value $hostParam[0].type = 'HostSystem' $_this = Get-View -Id 'CertificateManager-certificateManager' $_this.CertMgrRefreshCertificates_Task($hostParam) } }
11. Machine SSL Certificates § Creates a server-side SSL socket § Server verification and secure communication e.g. HTTPS or LDAPS § Each node has its own Machine SSL Certificate. i.e. Embedded Deployment; Management Node; or Platform Services Controller § All services use a Machine SSL Certificate for endpoint encryption. § All services communicate through the reverse proxy § Traffic does not go to the services themselves e.g. The vpxd service uses the MACHINE_SSL_CERT to expose its endpoint. 11
12. Solution User Certificates Located in VECS: § machine – Used by component manager, license server, and the logging service § vpxd – vCenter service daemon (vpxd) store on management nodes and embedded deployments. vpxd uses the solution user certificate to authenticate to vCenter Single Sign-On § vpxd-extensions – Includes the Auto Deploy service, inventory service, and other services that are not part of other solution users § vsphere-webclient – Includes the vSphere Web Client and some additional services such as the performance chart service 12
13. Solution User Certificates § Encapsulates one or more vCenter Server services § Certificate authenticated by vCenter Single Sign-On and issued a SAML token to authenticate to other solution user and services § Each solution user must be authenticated to vCenter Single Sign-On § Re-authentication occurs after a reboot and after a timeout § The timeout configurable in the vSphere Web Client and defaults to 2592000 seconds (30 days) 13 30 DAYS
14. Single Sign-On Certificates VMware Directory Service SSL Certificate – § With custom certificates you may need to replace this SSL certificate explicitly. VMware vCenter Single Sign-On Signing Certificate – § Security Token Service (STS) – an identity provider that issues, validates, and renews SAML tokens that are used for authentication throughout vSphere § By default, the STS signing certificate is generated by VMCA § Manually refresh STS certificate via vSphere Web Client when the certificate expires or changes 14
15. Single Sign-On Certificates § Not stored in VECS § Not managed with certificate management tools § Changes are not necessary, but in special situations, you can replace these certificates 15
16. Certificate Replacement Options for vCenter 16 VMCA Default ••VMCA provides the Root certificate ••All vSphere certificates chain to VMCA ••Regenerate certificates on demand easily VMCA Enterprise ••Replace VMCA CA cert with a subordinate CA certificate from the Enterprise PKI ••Upon removal of the old VMCA CA certificate, all old certificates will be regenerated Custom ••Disable VMCA as CA ••Provision your own custom certificates for each solution user and endpoint ••More complicated For highly security conscious customers only Hybrid ••Replacement of the Machine_SSL certs ••VMCA for Hosts and Solution Users ••Very popular with high security customers ••Recommended
17. VMware vSphere 6.0 Certificate Manager 17 Appliance Deployment /usr/lib/vmware-vmca/ bin/certificate-manager Windows Deployment <Drive>:\Program Files\VMware\ vCenter Server\vmcad\ certificate-manager
18. Common Certificate Manager Use Cases 18 VMCA as Root CA (Default or Option 4) VMCA as Enterprise CA Subordinate (Option 2) Custom CA (Option 1 & 5) Hybrid (Option 1 & 6)
19. VMCA as Root CA 19 VMware KB 2108294
20. VMCA as Enterprise CA Subordinate Private Key Algorithm: RSA with 2048 bits. Standard: X.509 v3 Format: PEM (PKCS8 and PKCS1) with a header of ---BEGIN CERTIFICATE--- Recommended Signature Algorithms: SHA256, SHA384, or SHA512 § Does NOT support wildcard cards or SubjectAltName § You CANNOT create subsidiary CAs of VMCA § No explicit limit to the length of the certificate chain § Synchronize time for all nodes in environment 20
21. But… 21
22. Hybrid Approach 22
23. Custom certificates for the Web Client VMCA for everything else (User Solutions, ESX hosts) Hybrid Approach Concepts 23 Operations Security
24. Hybrid Mode: 3rd Party Cert for Web Client access 24 What many security concerned companies are using for their vSphere environments 3rd Party Certificate Authority DC1.lab.local VCSA vCenter Server https://vcsa.lab.local https://esxi-a.lab.local SSL Certificate issued by DC1.lab.local Certificate Authority SSL Certificate issued by vcsa.lab.local VMCA
25. Implementing Hybrid Mode (High Level) 25 1. Use Option 1 to: • Replace Machine_SSL cert on all PSCs in SSO Domain • Replace Machine_SSL cert on all vCenter Servers in SSO domain 2. Use option 6 to: • Replace everything else on PSC & vCenter Server nodes 3. Use Web Client to replace certs on ESX hosts
26. ESXi Certificate Management Modes 26 VMware ESXi 6.0 VMCA Authority Mode Custom Mode Thumbprint Mode
27. 27 Default Value = vmca Possible Values = vmca | custom | thumbprint Search for certmgmt
28. VMCA Authority Mode § The default mode § Post-install ESXi always has an auto-generated certificate § ESXi certificates are stored locally on each host in the /etc/vmware/ssl § VMCA provisions the host a signed certificate when added to vCenter Server § Host certificates include the full chain to VMCA § ESXi certificates can be renewed via the vSphere Web Client or PowerCLI 28 vpxd.certmgmt.mode = vmca
29. Custom Mode § Replacement is the same as vSphere 5.5 – ESXi Shell – HTTPS GET/PUT § vifs will wrap these operations. § Custom / 3rd Party certificates – Must change vpxd.certmgmt.mode to custom or risk replacement by VMCA – Must update TRUSTED_ROOTS store in VECS on vCenter with the custom root certificates to ensure trust relationship – use the vecs-cli entry create command 29 vpxd.certmgmt.mode = custom
30. Thumbprint Mode § Legacy mode § Fallback option for vSphere 6.0 § May be used to retains vSphere 5.5 certificates during an upgrade § DO NOT use this mode unless encountering issues with vmca or custom mode § vCenter 6.0 and later services may not work correctly in thumbprint mode § Switching from thumbprint to vmca mode requires extensive planning 30 vpxd.certmgmt.mode = thumbprint
31. Walkthrough
32. Detailed Process – Certificate Manager 32
33. Detailed Process – Generate CSR 33
34. Detailed Process – Copy CSR Files 34
35. Detailed Process – Copy Certificate 35
36. Detailed Process – Submit CSR 36
37. Detailed Process – Download Certificate & Chain 37
38. Detailed Process – Export Root Certificate 38
39. Detailed Process – Upload Root & New Certificate 39
40. Detailed Process – Replace MACHINE_SSL Certificate 40
41. Detailed Process – Certificate Replacement Completed 41
42. Detailed Process – Validation 42
43. Detailed Process – ESXi Hosts 43
44. Detailed Process – Script for Entire Clusters 44
45. Upgrades and Operational Considerations VMware vSphere 6.0 Certificate Management
46. Deployment Considerations § VMCA as Enterprise CA Subordinate – Perform the signing certificate replacement on all Platform Services Controllers to ensure trusted certificates for all vCenter Server 6.0 installations • The ‘24 Hour Rule’ – Signing certificate must have a valid date of 24 hours prior before renewing host certificates or adding new hosts to vCenter Server – Plan for this aging period when configuring an existing environment – Replace certificates prior to putting a new environment into production – 6.0 U2 adds vpxd.certmgmt.certs.minutesBefore parameter to mitigate this design characteristic 46
47. Managing Certificates • Supports replacing certificates • No CRL enforcement against PKI for vCenter Server and ESXi hosts • If you suspect that one of your certificates has been compromised, revoke and replace all existing certificates, including the VMCA root certificate • If you do not remove revoked certificates, a man-in-the-middle attack might enable compromise through impersonation with the account's credentials. 47
48. Upgrades & Auto Deploy § Host Upgrades and VMCA Signed Certificates – Upgrade process replaces self-signed certificates with VMCA-signed certificates – vCenter then monitors certificates and displays details vSphere Web Client § Host Upgrades and Custom Certificates – Custom certificates are retained – even if expired or invalid – Change vxd.certmgmt.mode to custom to ensure certificates are not replaced accidentally § Update Manager – Not compatible with the Machine SSL certificate template in vSphere 6.0. § Use the vSphere 5.5 certificate template for Update Manager 6.0 48
49. VMware Validated Designs
50. CONFIDENTIAL 50
51. Resources • VMworld 2015 Presentation, scripts, videos – http://tenthrityam.org/INF4529 • Mike Foley’s Blog – http://www.vmware.com/go/hybridvmca • Hands On Labs – HOL-SDC-1610 • VMware vCenter Server Appliance 6.0 Poster – https://blogs.vmware.com/vsphere • Platform Services Controller 6.0 Topology Decision Tree – https://vmware.com/go/psctree • VMware Validated Designs – http://vmware.com/go/vvd • Twitter - @eck79, @tenthirtyam 51