Centralized Logging and Security for Containers on AWS with FireLens 1 | © 2019 Palo Alto Networks. All Rights Reserved. Vinay Venkataraghavan Principal Cloud Architect Palo Alto Networks Vipin Mohan Global Segment Lead - Containers & Serverless Amazon Web Services

Our thanks to... • Akshay Ram • Wesley Pettit • Uttara Sridhar • Carmen Puccio • Curtis Rissi • Michael Hausenblas 2 | © 2019 Palo Alto Networks. All Rights Reserved. • Rohit Gupta • Manu Parbhakar • Steven Cacciaroni • Gerry Fierling • Stephanie Broyles • Jeanette Christensen

Logs can be interesting! 3 | © 2019 Palo Alto Networks. All Rights Reserved.

To begin… let’s first categorize the logs Application Logs Typically this is where developers are concerned because these logs give insight into application performance. E.g.: ○ Example Application – /var/log/myapp.log ○ Web Logs - journalctl -u nginx.service --since today ○ Transaction Logs Infrastructure Management Logs Typically this is where operations teams are concerned because these logs give insight into the stability of the platform as well as the availability of the services. E.g.: ○ Syslog and other OS Logs ○ Audit Logs ○ Performance Metrics 4 | © 2019 Palo Alto Networks. All Rights Reserved.

Managed container services How containerized apps are deployed, scheduled, scaled, and policies are enforced Where the containers run Amazon Elastic Container Service ECS Amazon Elastic Kubernetes Service EKS Amazon EC2 AWS Fargate Container image repository Amazon Elastic Container Registry ECR Orchestration Compute Engine Image registry 5 | © 2019 Palo Alto Networks. All Rights Reserved.

Logging challenges with containers • No permanent storage • If your container dies, your logs will most likely be gone too • No fixed location from a network address standpoint • You should not care about the IP addresses of your containers • No fixed location from a placement perspective • Even though you can target nodes with things like label selectors, it’s all up to the scheduler 6 | © 2019 Palo Alto Networks. All Rights Reserved. Transfer your logs off the host system and label them appropriately

What is Log Routing? app container runtime EC2 instance routing Dashboards Cloudwatch insights, Grafana etc. Alerts Cloudwatch events, Kinesis Firehose, etc. Long term storage S3, Elasticsearch etc. Log Sources Routing Component Log Sinks 7 | © 2019 Palo Alto Networks. All Rights Reserved.

Log Routing - Pain points • Data collection is hard • Not all logs are of equal importance. E.g. • Some require real-time analytics • Some need to be stored long-term for compliance or to be analyzed if needed • Need for a fast and optimized solution that can deal with • Different sources of information • Different data formats • Multiple destinations 8 | © 2019 Palo Alto Networks. All Rights Reserved.

photograph here • Open source data collector • Unified logging layer - one-stop component that can aggregate data from multiple sources • Unifies differently formatted data into JSON objects and routes to different output destinations • Written in Ruby • Developed a rich ecosystem consisting of 700+ plugins that extend its functionality Fluentd

Fluent Bit • Sister project to Fluentd, a popular CNCF Log collection and routing tool • Can collect data from any input source, unify and deliver it to multiple destinations • Open Source • Fully written in C • Fast & Performant 10 | © 2019 Palo Alto Networks. All Rights Reserved.

Fluent Bit Performance Log Lines Per second Data Out Fluent Bit CPU (vCPU/CPU Thread) Fluent Bit Memory 100 25 KB/s 0.30% 27 MB 1,000 250 KB/s 3% 44 MB 10,000 2.5 MB/s 19% 65 MB 11 | © 2019 Palo Alto Networks. All Rights Reserved.

awsfirelens Docker log driver awsfirelens image pull image pull image pull Amazon ECR Amazon ECS/EC2 Amazon EKS/EC2 Amazon ECS/AWS Fargate User Amazon Kinesis Data Firehose Amazon S3 Amazon Athena log shipping log shipping log shipping Daemonset Enter AWS FireLens! 12 | © 2019 Palo Alto Networks. All Rights Reserved.

place diagram here Container Security Posture Deploy Phase RUNTIME SECURITY Build Phase twistcli Containers Hosts Images Twistlock / Prisma Compute Platform Compliance Vulnerabilities Runtime

Microservice in practice? 14 | © 2019 Palo Alto Networks. All Rights Reserved. ECS/EKS Twistlock AWS Firelens ● Application and infrastructure orchestration ● Network policy for L3/L4 ● Shift left: deploy phase security ● Vulnerability, Compliance and Runtime security ● Integrates with logging frameworks ● Learn from application traffic ● Unified and consistent logging ● Role based access to logs ● Enables devsecops There is a solution! This is a s/Mess/Mesh

How it works. …. 15 | © 2019 Palo Alto Networks. All Rights Reserved. twistcli Console ECS/EKS Cluster Nodes Nodes Nodes Defender Defender Defender AWS Firelens Amazon Kinesis Data Firehose Amazon S3 Amazon Athena Deployment phase Runtime phase DevOps SecOps Apps Apps Apps App logs Defender logs Logs

ECS Task Configurations for Firelens (1) 16 | © 2019 Palo Alto Networks. All Rights Reserved.

ECS Task Configuration for Firelens (2) 17 | © 2019 Palo Alto Networks. All Rights Reserved.

Firelens Logs in CloudWatch

cont..

Benefits Recap • Done right provides comprehensive visibility • Both stakeholders: devops and secops • Policy enforcement by InfoSec / SecOps ❏ Define vulnerability and compliance policies for images ❏ Same policies for scanned images are applied at runtime ❏ Ensure all images are scanned in the registry with twistcli ❏ Ensure logging is enabled for: • Apps (devops team) • Twistlock (secops team) • Role based access control • Define IAM roles for: • DevOps team access to application logs • SecOps team access to security logs and events Logging is a critical piece! With clearly defined policies and templates logging provides the framework for a successful devsecops process.

Thank You! 21 | © 2019 Palo Alto Networks. All Rights Reserved. Vinay Venkataraghavan vvenkatara@paloaltonetworks.com Vipin Mohan vipmohan@amazon.com

Benefits recap ( to modify….. this is just the info) 22 | © 2019 Palo Alto Networks. All Rights Reserved. • Firstly, done right provides a comprehensive security posture for containerized apps. • Enables policy enforcement by InfoSec/SecOps • Define vulnerability and compliance policies for images • Same policies for scanned images are applied at runtime • Ensure all images are scanned in the registry with twistcli • Ensure logging is enabled for: • Apps (devops team) • Twistlock (secops team) • Define IAM roles for: • DevOps team access to application logs • SecOps team access to security logs and events • Complete visibility for both application teams and security teams • Use kinesis analytics to raise high fidelity alerts to effect remediation workflows. • Iterate security policies to match application requirements. Logging is a critical piece! With clearly defined policies and templates logging provides the framework for a successful devsecops process.