プライベートクラウドでの効率的な証明書配布戦略 / Efficient Certificate Distribution Strategy in Private Cloud

LINEヤフーでは大規模なプライベートクラウドと社内認証局を運用し、Public CAと併せて多数のTLS証明書を管理しています。証明書の期限管理や更新は運用コストが発生するだけでなく、手動更新に伴う工数増大やヒューマンエラーのリスクが深刻な課題となっているため、より効率的な証明書管理エコシステムの構築が急務となっています。
本セッションでは、まずプライベートクラウド上における証明書管理の現状と既存のセキュリティ施策について共有し、次に社内エコシステムに最適化したcert-manager（k8s）を活用した証明書自動更新の事例をご紹介します。さらに、今後展開が予定されている自動証明書管理基盤のビジョンについても共有します。

Transcript

1. Efficient Certificate Distribution Strategy in Private Cloud Infrastructure Group Security Platform Division Trust Technology Team TSURUDO Ryosuke YAMAGUCHI Katsuya
2. About us Introduction
3. Security Engineer Joined the former Yahoo Japan in 2023. Develops, maintains, and operates internal certificate issuance systems and authentication/authorization systems that use certificates. I also write technical books in my private time and host a casual chat podcast. uncrop.jp Who am I ? Introduction TSURUDO Ryosuke
4. YAMAGUCHI Katsuya • Career • Joined Yahoo Japan Corporation as a new graduate in 2017 • Engaged in the development of security infrastructure and key management systems • Involved in the construction of certificate management infrastructure • Focused on improving security using certificate-related mechanisms as a team leader Who am I ? Introduction
5. YAMAGUCHI Katsuya • Career • Joined Yahoo Japan Corporation as a new graduate in 2017 • Engaged in the development of security infrastructure and key management systems • Involved in the construction of certificate management infrastructure • Focused on improving security using certificate-related mechanisms as a team leader Who am I ? Introduction
6. The Number of Certificates in Our Private Cloud • A significant number of server certificates are in use within our infrastructure. • We strive to secure all internal communications with TLS. Private CA Certs Public CA Certs 1550 > 830 >
7. How should an ideal private cloud appear regarding certificate management?
8. A Secure-by-Default World
9. Strategy Certificate Ownership Access Control Multi-CA Support Certificate Issuance Policy Automatic Update Automatic Placement
10. Certificate Issuance Policy
11. Enforce a baseline of security and quality for every certificate issued.
12. Policy: Cryptographic Requirements e.g., RSA 4096-bit, ECDSA P- 256 Key Algorithm & Size e.g., SHA-256 or stronger. Signature Algorithm e.g., Forbid SHA-1 for signatures, or any key types with known weaknesses. Prohibited Standards
13. Policy: Certificate Profile & Contents e.g., Limit wildcard depth (*.sub.example.com only), restrict the number of SANs. Domain & Scope Control e.g., Block issuance for domains like "server.local" or "db.internal-network". Information Leakage Prevention e.g., Require specific Organization/Country fields; limit Extended Key Usage to "Server Authentication". Content & Usage Standardization
14. Improving the Developer Experience • Eliminate Guesswork and Ambiguity • Remove the Fear of Making Mistakes • Improve Service Reliability Ensuring a consistent, high standard of quality for every certificate, automatically. Benefits for Developers
15. Enforce Certificate Issuance Policy in Our Private Cloud Certificate Management System Handling all issue requests • All issuance requests are centralized through this system. • Policies are embedded in the system and automatically enforced on every request. • Regular reviews ensure these policies stay aligned with current security standards. CA (Certificate Authority) Enforce Our Certificate Issuance Policies Issue request
16. Certificate Ownership
17. • Ensure the certificate's owner is always identifiable • Who? • Which organization? • Which team? • Which project? • Always assign certificate management to a single group What is Certificate Ownership?
18. • Ambiguous Responsibility for Critical Tasks • Who handles certificate expirations? • Who responds to key compromises or leaks? Why Certificate Ownership Matters
19. • Ambiguous Responsibility for Critical Tasks • Who handles certificate expirations? • Who responds to key compromises or leaks? • Difficulty in Tracking and Allocating Costs • Inability to accurately bill departments or projects for certificate issuance. Why Certificate Ownership Matters
20. • Ambiguous Responsibility for Critical Tasks • Who handles certificate expirations? • Who responds to key compromises or leaks? • Difficulty in Tracking and Allocating Costs • Inability to accurately bill departments or projects for certificate issuance. • Maintaining Ownership Through Organizational Changes • How to manage ownership when admins leave or teams are reorganized? Why Certificate Ownership Matters
21. • Ambiguous Responsibility for Critical Tasks • Who handles certificate expirations? • Who responds to key compromises or leaks? • Difficulty in Tracking and Allocating Costs • Inability to accurately bill departments or projects for certificate issuance. • Maintaining Ownership Through Organizational Changes • How to manage ownership when admins leave or teams are reorganized? • Complexity in Cross-Functional Projects • Difficulty assigning a single owner when multiple departments and teams are involved. Why Certificate Ownership Matters
22. Certificate Ownership in Our Private Cloud Project A Project B
23. Certificate Ownership in Our Private Cloud Code master Project A Project B Project A system code Project B system code Separate certificate management by project and delegate oversight to a unified code master.
24. Multi-CA Support
25. • Cost Efficiency • Initial setup costs offset by long-term savings. • Internal Trust Infrastructure • Ensures secure communication between internal applications and services. • Faster Issuance • Issues certificates in minutes, unlike lengthy processes with public CAs. • Protection of Internal Domains • Keeps internal domains confidential by avoiding Public CA issuance, which exposes details in Certificate Transparency logs. Benefits of Private CA
26. Multi-CA Support in Our Private Cloud Project A Project B Issue request Issue request Project A’s certs Project B’s certs Certificate Management System Handling all issue requests Public CA Private CA Public CA
27. Multi-CA Support in Our Private Cloud Project A Project B Issue request Issue request Project A’s certs Project B’s certs Certificate Management System Handling all issue requests Handling differences between CAs Public CA Private CA Public CA
28. Multi-CA Support in Our Private Cloud Multi-CA Support Project A Project B Project A’s certs Project B’s certs Certificate Management System Handling all issue requests Offloading the domain validation task from users Public CA Fetch certificates DNS Domain verification using DNS Setting the challenge token in a DNS TXT record
29. Access Control
30. Access Control for Certificate Management Project A Project A’s certs Project B’s certs Certificate Management System Project B Grant Grant Deny
31. • We use Athenz as our standard platform for authentication and authorization. • Our private cloud's access control is built entirely on Athenz. • This provides a unified system for project-based access control, integrating seamlessly with our workflow. Access Control for Certificate Management In Our Private Cloud Athenz logo by https://github.com/AthenZ/athenz Athenz: https://www.athenz.io ※Athenz is a recognized CNCF (Cloud Native Computing Foundation) Sandbox project.
32. Project A Project A’s certs Project B’s certs Certificate Management System Grant Deny Access Control for Certificate Management In Our Private Cloud
33. Strategy Certificate Ownership Access Control Multi-CA Support Certificate Issuance Policy Automatic Update Automatic Placement
34. Automatic placement
35. What to deploy? Automatic placement Leaf certificate 01 Certificate Set with certificate 02 Private Key As needed 03 Intermediate Certificate
36. What to deploy? Automatic placement Leaf certificate 01 Certificate Set with certificate 02 Private Key As needed 03 Intermediate Certificate Deploy everything needed to use a certificate.
37. Automatic placement Expected Outcomes
38. Expected Outcomes Automatic placement B Function A DB Service A Function User B DB
39. Expected Outcomes Automatic placement B Function Service A Function User
40. Expected Outcomes Automatic placement B Function A DB Service A Function User B DB
41. Expected Outcomes Automatic placement Certificate renewal procedure for Service A Hoge fuga -------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- Certificate renewal procedure for Service B Hoge fuga -------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- Certificate renewal procedure for DB Hoge fuga -------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- Certificate renewal procedure for LB Hoge fuga -------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- -------------------------------- --------------------------------
42. Manually deploying certificates to a wide variety of components induces errors. Expected Outcomes Reduce operational errors Automatic placement It is very tedious to understand the procedures for each target system and execute them every time a new certificate is issued or renewed. Reduce the operational burden B Function Service A Function User Certificate renewal procedure for Service A Hoge fuga ------------ ------- Certificate renewal procedure for Service B Hoge fuga ------------ -------- Certificate renewal procedure for DB Hoge fuga ------------ -------- Certificate renewal procedure for LB Hoge fuga ------------ --------
43. Manually deploying certificates to a wide variety of components induces errors. Expected Outcomes Reduce operational errors Automatic placement It is very tedious to understand the procedures for each target system and execute them every time a new certificate is issued or renewed. Reduce the operational burden B Function Service A Function User Certificate renewal procedure for Service A Hoge fuga ------------ ------- Certificate renewal procedure for Service B Hoge fuga ------------ -------- Certificate renewal procedure for DB Hoge fuga ------------ -------- Certificate renewal procedure for LB Hoge fuga ------------ -------- Eliminate incidents associated with certificate renewal tasks
44. Automatic placement How should we do?
45. How should we do? Automatic placement REST API / ACME 01 Publishing an API Authentication and Authorization 02 Establishing Auth Centralized management 03 Building an ecosystem
46. • A tool to automate the management of certificates within a Kubernetes cluster. • It automatically obtains certificates from a certificate authority. • It monitors certificate expiration dates and automatically performs renewal processing before they expire. • You define the required certificates in a Kubernetes manifest file. Publishing an API Automatic renewal cert-manager cert-manager project logo (c) by Jetstack Ltd.
47. • A communication protocol for automating tasks such as certificate issuance, renewal, and revocation. • An ACME client installed on a web server communicates with the CA. • It automatically verifies domain ownership to complete certificate acquisition and renewal without manual intervention. Publishing an API Automatic renewal ACME (Automatic Certificate Management Environment)
48. How should we do? Automatic placement REST API / ACME 01 Publishing an API Authentication and Authorization 02 Establishing Auth Centralized management 03 Building an ecosystem
49. Establishing Auth Automatic placement Project A Certificate 01 Project B . . . Certificate 02 Certificate 03 Certificate 04
50. How should we do? Automatic placement REST API / ACME 01 Publishing an API Authentication and Authorization 02 Establishing Auth Centralized management 03 Building an ecosystem
51. Centralized management Automatic placement Centralized management system LB DB VM . . . CA 02 CA 01 CA 03
52. What to deploy? Automatic placement Leaf certificate 01 Certificate Set with certificate 02 Private Key As needed 03 Intermediate Certificate
53. Strategy Certificate Ownership Access Control Multi-CA Support Certificate Issuance Policy Automatic Update Automatic Placement
54. Automatic update
55. What to update? Automatic update Before it expires 01 Certificate Do not reuse 02 Private Key
56. What to update? Automatic update Before it expires 01 Certificate Do not reuse 02 Private Key Make the certificate renewal process seamless
57. Automatic update Expected Outcomes
58. Expected Outcomes Automatic update ~ 2025/06/30 ~ 2026/06/30 ~ 2025/07/15 ~ 2025/07/01 ~ 2026/05/15
59. Expected Outcomes Automatic update ~ 2025/06/30 ~ 2026/06/30 ~ 2025/07/15 ~ 2025/07/01 ~ 2026/05/15
60. Expected Outcomes Automatic update ~ 2026/06/30 ~ 2026/06/30 ~ 2026/07/15 ~ 2026/07/01 ~ 2026/05/15
61. Expected Outcomes Automatic update 398 days Current https://groups.google.com/a/groups.cabforum.org/g/servercert-wg/c/bvWh5RN6tYI
62. Expected Outcomes Automatic update 398 days 200 days 100 days 47 days Current 2026/03/15 2027/03/15 2029/03/15 https://groups.google.com/a/groups.cabforum.org/g/servercert-wg/c/bvWh5RN6tYI
63. Expected Outcomes Automatic update Automatic update Automatic placement &
64. Expected Outcomes Automatic update Automatic update Automatic placement & To an environment where developers don't have to be aware of certificates.
65. Expected Outcomes Automatic update
66. Expected Outcomes Automatic update
67. Expected Outcomes Automatic update
68. Expected Outcomes Automatic update Mitigating compromises and reducing the scope of impact when they occur
69. Automatic update How should we do?
70. Centralized management Automatic update Centralized management system CA 02 CA 01 CA 03
71. Centralized management Automatic update Centralized management system CA 02 CA 01 CA 03 Certificate A ~2025/06/30 Certificate B ~2025/07/01 Certificate C ~2025/07/15 Certificate D ~2026/05/15 Certificate E ~2026/06/30 . . .
72. Centralized management Automatic update Centralized management system CA 02 CA 01 CA 03 Certificate A ~2025/06/30 Certificate B ~2025/07/01 Certificate C ~2025/07/15 Certificate D ~2026/05/15 Certificate E ~2026/06/30 . . . Renew A/B Renew C
73. Centralized management Automatic update
74. What to update? Automatic update Before it expires 01 Certificate Do not reuse 02 Private Key
75. Strategy Certificate Ownership Access Control Multi-CA Support Certificate Issuance Policy Automatic Update Automatic Placement
76. Something like the announcements from the CA/Browser Forum Strategy based on the current situation
77. The need for proactive information gathering
78. Providing a seamlessly secure environment The need for proactive information gathering Security Engineer Web developer Application developer Back-end developer . . .
79. Providing a seamlessly secure environment The need for proactive information gathering Security Engineer Web developer Application developer Back-end developer . . . We continuously strive to make our private cloud more secure
80. Efficient Certificate Distribution Strategy in Private Cloud Infrastructure Group Security Platform Division Trust Technology Team TSURUDO Ryosuke YAMAGUCHI Katsuya