Centralize Logging Multi AWS Accounts

Transcript

1. Centralize Logging Multi-AWS Account
2. @zamirajaupaj About me Solution Architect AWS User Group Organizer AI & Serverless Lover AWS Customer since 2015
3. Why logs Cross Account Solutions Data Processing (Stream) Demo Agenda
4. Do I Have a logs?
5. Let’s log everything
6. Data overload
7. Problem with Logs
8. How we can mange all Accounts?
9. Cross-Account Solution
10. Stream Log Cross AWS Accounts CENTRAL LOGGING ACCOUNT Customer A Customer B Customer Z
11. Network logs Flow Logs Infrastrutture logs CloudTrail Amazon S3 ELB/ALB Lambda Kinesis Host based logs Application Logs Windows Events Logs Database Logs Security Logs Third party logs …. What kind of logs you are collecting?
12. Collect Logs in Source Account Source Amazon EC2 Amazon RDS Amazon ElastiCache Amazon DocumentDB AWS Lambda Amazon API Gateway AWS IoT Amazon CloudWatch SERVICE IAM ROLE CROSS-ROLE IAM CLOUDWATCH SUBSRIPTION IAM Role
13. Stream Log Cross-Accounts Account A Account B Account Z CloudWatch Subscription Data Stream Firehose Transformation Storage in s3 Central Account ES
14. Real-time analytics in Central log Account Data Stream Kinesis Data Analytics: Time window aggregation Kinesis Data Firehose: Error stream S3: Error records Record from Source Account Lambda: Alert function DynamoDB SNS: Notifications Log Destination
15. Fully Serverless Architetture No server or container management Flexible scaling $ No idle capacity High availability
16. Central Logging Service
17. IDEs Languages AWS Cloud9 AWS Toolkit for PyCharm AWS Toolkit for IntelliJ AWS Toolkit for VS Code AWS Nested apps Websocket support for API Gateway ALB support for Lambda Programming Model AWS Lambda layers Workflows Step Functions API Connectors Amazon Managed Streaming for Kafka Build increasingly powerful applications,faster
18. BUSINESS LOGIC LIB A LIB B BUSINESS LOGIC LIB A LIB B BUSINESS LOGIC LIB A LIB B BUSINESS LOGIC LIB A LIB B Before BUSINESS LOGIC BUSINESS LOGIC BUSINESS LOGIC BUSINESS LOGIC LIB A LIB B After Programming Model Lambda Layers Extend the Lambda execution environment with any binaries, dependencies, or runtimes
19. Streaming with Amazon Kinesis Collect, process, and analyze video and data streams in realtime Kinesis Data Firehose SQL Kinesis Data Analytics Kinesis Data Streams Kinesis Video Streams
20. Streaming dataingestion Amazon S3: Buffered files Kinesis Agent Record producers Amazon Redshift: Table loads Amazon Elasticsearch Service: Domain loads Amazon S3: Source record backup Transformed records Put Records Kinesis Firehose: Delivery stream AWS Lambda: Transformations & enrichment Amazon DynamoDB: Lookup tables Raw Lookup Transformed
21. Kinesis Bestpractices Tune Firehose buffer size and buffer interval • Larger objects = fewer Lambda invocations & Amazon S3 PUTs Enable compression to reduce storage costs Enable Parquet format transformation (columnar) Enable Encryption with KMS Enable Source Record Backup for transformations • Recover from transformation errors
22. CloudWatch VPC log Infrastrutture log Application log Alarms CloudWatch Metrics Event based Rules
23. CloudTrail Benefits • Reduce the risck for a long tampering • Combination with Amazon s3 Enable CloudTrail Cross-Regions Enable log file Validation Encrypted with KMS Integration with CloudWatch
24. Elasticsearch ElasticSearch Log Analytics Infrastrutture Monitoring Full Text Search
25. Benefits of Amazon ElastiSearch Service Highly Scalable Secure Easy to use Highly Available Support open Source API Integrated with other AWS Services Use Case • Troubleshooting • Root cause analysis • Application performance management • Security intelligence • Applicating trucking • Business analytics
26. Data Processing in Real-Time
27. Processing real-time streaming data Data Source Ingest Analyse Query Customer Data Source Ingest Analyse Query Customer Data Source Ingest Analyse Query Customer
28. Processing real-time streaming data Query Customer Analyse Data Source Ingest Data Source Ingest Data Source Ingest Query Customer Query Customer
29. Real-time analytics Data Stream Kinesis Data Analytics: Time window aggregation Kinesis Data Firehose: Error stream S3: Error records Record producers Lambda: Alert function DynamoDB SNS: Notifications
30. <DEMO TIME>
31. Thank you! Zamira Jaupaj Solution Architect AWS @zamirajaupaj