Auditing Your Automation's Access... Using More Automation

Originally presented at Datadog DASH 2022.

From CI/DI pipelines to container orchestrators to developer debugging tools, you spend years building the automation needed to scale your systems. As we grow our automation footprint over time, we grant more and more access to new systems. But how do you know if that automation is accessing the right systems at the right time? And more recently, we’ve encountered new incidents in which an unauthorized user can exploit automation, compromise credentials, and access systems with us completely unaware.

This talk will discuss ways to build observability into your automation in order to audit its access across different parts of the software development lifecycle. Whether clicking on various things in the console or uploading .tar.gz files, too much human access to the phases of the SDLC can be risky. We will look at automation within your CI/CD pipelines and services, and talk through use cases where the replacement of human access might benefit from automation.

We will also outline some techniques to maintain least-privilege access, mitigate blast radius of compromised credentials, and better observe interactions between services so you quickly identify unauthorized access.

You will learn about managing, aggregating, and configuring audit logging and metrics using:
- Service mesh for authorization
- Secrets management for credentials
- Secure access management for manual access to systems

By the end of this talk, you will be able to identify ways to better secure your automation and observe its access across a system.

Transcript

1. …with More Automation DASH | October 2022 Auditing Access in Automation
2. Automation Cycle Recognize and remediate. Implement automation. Identify functional issue.
3. We deploy and operate quickly, securely, and reliably…
4. until we find some anomalous activity.
5. ⚠ What? ⚠ Who? ⚠ Where?
6. Observability of Automation datadoghq.com/case-studies/hashicorp/
7. Developer Advocate HashiCorp 
 she/her 
 
 @joatmon08 | joatmon08.github.io Rosemary Wang
8. Automation Cycle (with Security) Recognize and remediate. Fix automation. Identify security issue.
9. How do you speed up the automation cycle with security?
10. Time to Resolution Reduce time for these stages. Fix automation. Identify security issue. Recognize and remediate.
11. What should you observe in automation for security?
12. Automation requires access.
13. Automation need lots of access. Example - Deployment Pipelines •Check out code •Build infrastructure •Deploy application •Test application •Check code quality Need access to… •Version control •Infrastructure provider •Platform/release repository •Data/other services •Quality assurance tool
14. Why audit access? Observe service interactions Maintain least privilege Identify and mitigate blast radius Standardize event information
15. Application Deployment Local Testing Integration Testing Production
16. Local Testing Auditing Engineering Access
17. Engineers need access to build. Engineers… •Test •Debug •Deploy Needs access to… •Production Applications •Databases •Managed Services •Infrastructure APIs •Platform APIs
18. Track human interactions Audit logs from cloud providers Audit logs from managed services Audit logs from secure access management tools
19. Database Automation. Audit local connection to database.
21. Is this fine? Identify unauthorized access to target.
23. Integration Testing Auditing Service Access
24. Services need access to other services. Services… •Test •Debug •Deploy Need access to… •Production Applications •Databases •Managed Services •Infrastructure APIs •Platform APIs
25. Track service interactions Access logs Traces Authentication requests Network flow logs
26. Service Automation. Tracing requests between services using APM.
28. Is this fine? Anomalous access between services.
30. Secrets provide access.
31. Database Access. Tracking authentication requests.
33. Production Auditing Even More Automation
34. Production access involves… Local Temporary operational user access Integration Dynamic service access
35. Track production interactions Audit logs from secrets manager Audit logs from providers Audit logs from secure access management tools
36. Event-Driven Access Control WEBHOOK DATADOG Declare incident. status: active BOUNDARY Authorizes engineers to open sessions. Create temporary role with access to production.
39. Event-Driven Access Control WEBHOOK DATADOG Resolve incident. status: resolved BOUNDARY Revokes access and terminates sessions. Delete temporary role with access to production.
40. Secrets provide access.
41. Audit usage of secrets.
42. Infrastructure Automation. Audit static secret use for automation.
44. Service Automation. Issuing certificates.
47. Is this fine? Identify root access to a static secret.
49. Automation requires access.
50. Other Use Cases Infrastructure as Code Controllers GitOps Managed Services
51. Why audit access? Observe service interactions Maintain least privilege Identify and mitigate blast radius Standardize event information
52. Datadog Cloud SIEM Define Custom Rules datadoghq.com/blog/hashicorp-vault-security-datadog
53. Time to Resolution Reduce time for these stages. Fix automation. Identify security issue. Recognize and remediate.
54. Learn more Datadog & HashiCorp datadoghq.com/blog/tag/hashicorp/ Learn More Example at 
 joatmon08/hashicorp-stack-demoapp
55. Rosemary Wang @joatmon08 joatmon08.github.io Thank you!