Centralize Logging Multi AWS Accounts

Slide 1

Slide 1 text

Centralize Logging Multi-AWS Account

Slide 2

Slide 2 text

@zamirajaupaj About me Solution Architect AWS User Group Organizer AI & Serverless Lover AWS Customer since 2015

Slide 3

Slide 3 text

Why logs Cross Account Solutions Data Processing (Stream) Demo Agenda

Slide 4

Slide 4 text

Do I Have a logs?

Slide 5

Slide 5 text

Let’s log everything

Slide 6

Slide 6 text

Data overload

Slide 7

Slide 7 text

Problem with Logs

Slide 8

Slide 8 text

How we can mange all Accounts?

Slide 9

Slide 9 text

Cross-Account Solution

Slide 10

Slide 10 text

Stream Log Cross AWS Accounts CENTRAL LOGGING ACCOUNT Customer A Customer B Customer Z

Slide 11

Slide 11 text

Network logs Flow Logs Infrastrutture logs CloudTrail Amazon S3 ELB/ALB Lambda Kinesis Host based logs Application Logs Windows Events Logs Database Logs Security Logs Third party logs …. What kind of logs you are collecting?

Slide 12

Slide 12 text

Collect Logs in Source Account Source Amazon EC2 Amazon RDS Amazon ElastiCache Amazon DocumentDB AWS Lambda Amazon API Gateway AWS IoT Amazon CloudWatch SERVICE IAM ROLE CROSS-ROLE IAM CLOUDWATCH SUBSRIPTION IAM Role

Slide 13

Slide 13 text

Stream Log Cross-Accounts Account A Account B Account Z CloudWatch Subscription Data Stream Firehose Transformation Storage in s3 Central Account ES

Slide 14

Slide 14 text

Real-time analytics in Central log Account Data Stream Kinesis Data Analytics: Time window aggregation Kinesis Data Firehose: Error stream S3: Error records Record from Source Account Lambda: Alert function DynamoDB SNS: Notifications Log Destination

Slide 15

Slide 15 text

Fully Serverless Architetture No server or container management Flexible scaling $ No idle capacity High availability

Slide 16

Slide 16 text

Central Logging Service

Slide 17

Slide 17 text

IDEs Languages AWS Cloud9 AWS Toolkit for PyCharm AWS Toolkit for IntelliJ AWS Toolkit for VS Code AWS Nested apps Websocket support for API Gateway ALB support for Lambda Programming Model AWS Lambda layers Workflows Step Functions API Connectors Amazon Managed Streaming for Kafka Build increasingly powerful applications,faster

Slide 18

Slide 18 text

BUSINESS LOGIC LIB A LIB B BUSINESS LOGIC LIB A LIB B BUSINESS LOGIC LIB A LIB B BUSINESS LOGIC LIB A LIB B Before BUSINESS LOGIC BUSINESS LOGIC BUSINESS LOGIC BUSINESS LOGIC LIB A LIB B After Programming Model Lambda Layers Extend the Lambda execution environment with any binaries, dependencies, or runtimes

Slide 19

Slide 19 text

Streaming with Amazon Kinesis Collect, process, and analyze video and data streams in realtime Kinesis Data Firehose SQL Kinesis Data Analytics Kinesis Data Streams Kinesis Video Streams

Slide 20

Slide 20 text

Streaming dataingestion Amazon S3: Buffered files Kinesis Agent Record producers Amazon Redshift: Table loads Amazon Elasticsearch Service: Domain loads Amazon S3: Source record backup Transformed records Put Records Kinesis Firehose: Delivery stream AWS Lambda: Transformations & enrichment Amazon DynamoDB: Lookup tables Raw Lookup Transformed

Slide 21

Slide 21 text

Kinesis Bestpractices Tune Firehose buffer size and buffer interval • Larger objects = fewer Lambda invocations & Amazon S3 PUTs Enable compression to reduce storage costs Enable Parquet format transformation (columnar) Enable Encryption with KMS Enable Source Record Backup for transformations • Recover from transformation errors

Slide 22

Slide 22 text

CloudWatch VPC log Infrastrutture log Application log Alarms CloudWatch Metrics Event based Rules

Slide 23

Slide 23 text

CloudTrail Benefits • Reduce the risck for a long tampering • Combination with Amazon s3 Enable CloudTrail Cross-Regions Enable log file Validation Encrypted with KMS Integration with CloudWatch

Slide 24

Slide 24 text

Elasticsearch ElasticSearch Log Analytics Infrastrutture Monitoring Full Text Search

Slide 25

Slide 25 text

Benefits of Amazon ElastiSearch Service Highly Scalable Secure Easy to use Highly Available Support open Source API Integrated with other AWS Services Use Case • Troubleshooting • Root cause analysis • Application performance management • Security intelligence • Applicating trucking • Business analytics

Slide 26

Slide 26 text

Data Processing in Real-Time

Slide 27

Slide 27 text

Processing real-time streaming data Data Source Ingest Analyse Query Customer Data Source Ingest Analyse Query Customer Data Source Ingest Analyse Query Customer

Slide 28

Slide 28 text

Processing real-time streaming data Query Customer Analyse Data Source Ingest Data Source Ingest Data Source Ingest Query Customer Query Customer

Slide 29

Slide 29 text

Real-time analytics Data Stream Kinesis Data Analytics: Time window aggregation Kinesis Data Firehose: Error stream S3: Error records Record producers Lambda: Alert function DynamoDB SNS: Notifications