Centralized Logging Solution using FireLens and Fluent Bit for ECS David Chou @ Golang Taipei 

@ Umbo Computer Vision @ Golang Taipei david74.chou @ facebook david74.chou @ medium david7482 @ github Golang Taipei Telegram Golang Taipei Facebook 

Why do we need centralized logging? 

Why do we need centralized logging? Usability Availability Analyzability 

Centralized logging architecture AWS Container Logging Deep Dive 

No content

Implemented in Ruby/C @ 2011 Ecosystem of Plugins (900+) Extend with Ruby Memory Usage (40MB) Higher CPU Usage Better for Aggregation Support Forward Protocol Implemented in C @ 2015 Plugins are Included (56) Extend with C/Go/Lua Reduced Memory Usage (500KB) Lower CPU Usage Better as a Sidecar Support Forward Protocol KubeCon 2019: Fluent Bit Extending Your Logging Pipeline with Go 

AWS Blog: Centralized Container Logging with Fluent Bit 

Output 1 Output 2 Output N Input Parser Filter Buffer Routing 

Output 1 Output 2 Output N Input Parser Filter Information gathering, e.g. forward, tail, syslog, memory Buffer Routing 

Output 1 Output 2 Output N Input Parser Filter Unstructured logs => Structured binary format, e.g. json, regx (nginx, syslog, etc) Buffer Routing 

Output 1 Output 2 Output N Input Parser Filter Input data manipulation, e.g. grep, record_modifier Buffer Routing 

Output 1 Output 2 Output N Input Parser Filter Message buffering & retry Buffer Routing 

Output 1 Output 2 Output N Input Parser Filter Tag routing Buffer Routing 

Output 1 Output 2 Output N Input Parser Filter Send data to different destinations, e.g. forward, es, Go plugins Buffer Routing 

How to write a Fluent Bit Go plugin? 

Fluent Bit Go Plugin fluent-bit plugin.so dynamic loading 

//export FLBPluginRegister func FLBPluginRegister(def unsafe.Pointer) int { // Gets called only once when the .so is loaded. return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") } //export FLBPluginInit func FLBPluginInit(plugin unsafe.Pointer) int { // Gets called once for each instance you have configured. param := output.FLBPluginConfigKey(plugin, "param") return output.FLB_OK } //export FLBPluginFlushCtx func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c // Gets called once for each message to be written to an instance. } //export FLBPluginExit func FLBPluginExit() int { // Gets called on teardown. return output.FLB_OK } func main() { } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 //export FLBPluginRegister func FLBPluginRegister(def unsafe.Pointer) int { // Gets called only once when the .so is loaded. return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") } 1 2 3 4 5 6 //export FLBPluginInit 7 func FLBPluginInit(plugin unsafe.Pointer) int { 8 // Gets called once for each instance you have configured. 9 param := output.FLBPluginConfigKey(plugin, "param") 10 return output.FLB_OK 11 } 12 13 //export FLBPluginFlushCtx 14 func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c 15 // Gets called once for each message to be written to an instance. 16 } 17 18 //export FLBPluginExit 19 func FLBPluginExit() int { 20 // Gets called on teardown. 21 return output.FLB_OK 22 } 23 24 func main() { 25 } 26 //export FLBPluginInit func FLBPluginInit(plugin unsafe.Pointer) int { // Gets called once for each instance you have configured. param := output.FLBPluginConfigKey(plugin, "param") return output.FLB_OK } //export FLBPluginRegister 1 func FLBPluginRegister(def unsafe.Pointer) int { 2 // Gets called only once when the .so is loaded. 3 return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") 4 } 5 6 7 8 9 10 11 12 13 //export FLBPluginFlushCtx 14 func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c 15 // Gets called once for each message to be written to an instance. 16 } 17 18 //export FLBPluginExit 19 func FLBPluginExit() int { 20 // Gets called on teardown. 21 return output.FLB_OK 22 } 23 24 func main() { 25 } 26 //export FLBPluginFlushCtx func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c // Gets called once for each message to be written to an instance. } //export FLBPluginRegister 1 func FLBPluginRegister(def unsafe.Pointer) int { 2 // Gets called only once when the .so is loaded. 3 return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") 4 } 5 6 //export FLBPluginInit 7 func FLBPluginInit(plugin unsafe.Pointer) int { 8 // Gets called once for each instance you have configured. 9 param := output.FLBPluginConfigKey(plugin, "param") 10 return output.FLB_OK 11 } 12 13 14 15 16 17 18 //export FLBPluginExit 19 func FLBPluginExit() int { 20 // Gets called on teardown. 21 return output.FLB_OK 22 } 23 24 func main() { 25 } 26 //export FLBPluginExit func FLBPluginExit() int { // Gets called on teardown. return output.FLB_OK } //export FLBPluginRegister 1 func FLBPluginRegister(def unsafe.Pointer) int { 2 // Gets called only once when the .so is loaded. 3 return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") 4 } 5 6 //export FLBPluginInit 7 func FLBPluginInit(plugin unsafe.Pointer) int { 8 // Gets called once for each instance you have configured. 9 param := output.FLBPluginConfigKey(plugin, "param") 10 return output.FLB_OK 11 } 12 13 //export FLBPluginFlushCtx 14 func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c 15 // Gets called once for each message to be written to an instance. 16 } 17 18 19 20 21 22 23 24 func main() { 25 } 26 func main() { } //export FLBPluginRegister 1 func FLBPluginRegister(def unsafe.Pointer) int { 2 // Gets called only once when the .so is loaded. 3 return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") 4 } 5 6 //export FLBPluginInit 7 func FLBPluginInit(plugin unsafe.Pointer) int { 8 // Gets called once for each instance you have configured. 9 param := output.FLBPluginConfigKey(plugin, "param") 10 return output.FLB_OK 11 } 12 13 //export FLBPluginFlushCtx 14 func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c 15 // Gets called once for each message to be written to an instance. 16 } 17 18 //export FLBPluginExit 19 func FLBPluginExit() int { 20 // Gets called on teardown. 21 return output.FLB_OK 22 } 23 24 25 26 

# Build fluent bit go plugin $ go build -o plugin.so -buildmode c-shared plugin.go # Run Fluent Bit with the new plugin $ fluent-bit -e plugin.so -i cpu -o gstdout 

# Build fluent bit go plugin $ go build -o plugin.so -buildmode c-shared plugin.go # Run Fluent Bit with the new plugin $ fluent-bit -e plugin.so -i cpu -o gstdout Go build modes default archive exe pie shared plugin c-archive c-shared 

AWS FireLens 

AWS FireLens Support a wide-array of AWS Services as log destinations Require no additional configuration beyond ECS Task Definition Use Open Source for extensibility: Fluent Bit, Fluentd Facilitate partner integration AWS Blog: Under the hood: FireLens for Amazon ECS Tasks 

AWS for Fluent Bit A Fluent Bit image with AWS Go plugins on Docker Hub Regional public images hosted on ECR amazon/aws-for-fluent-bit 

AWS for Fluent Bit - Log destination Kinesis Data Firehose S3 Amazon ElasticSearch Service Kinesis Data Streams CloudWatch Logs Kafka Self-hosted ElasticSearch ( ) DataDog Forward to a Fluentd aggregator #698 

AWS for Fluent Bit - Log destination $ cat entrypoint.sh exec /fluent-bit/bin/fluent-bit \ -e /fluent-bit/firehose.so \ -e /fluent-bit/cloudwatch.so \ -e /fluent-bit/kinesis.so \ -c /fluent-bit/etc/fluent-bit.conf github: aws-for-fluent-bit 

Use AWS FireLens in ECS aws-samples/amazon-ecs-firelens-examples 

{ "essential": true, "image": "amazon/aws-for-fluent-bit:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit", "options": { "enable-ecs-log-metadata": "true", "config-file-type": "file", "config-file-value": "/json.conf" } } }, { "essential": true, "image": "", "name": "app", "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "cloudwatch", "region": "us-west-2", "log_group_name": "firelens-fluent-bit", "log_stream_prefix": "from-fluent-bit", "include-pattern": "[Ee]rror" } } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 "image": "amazon/aws-for-fluent-bit:latest", "image": "", { 1 "essential": true, 2 3 "name": "log_router", 4 "firelensConfiguration": { 5 "type": "fluentbit", 6 "options": { 7 "enable-ecs-log-metadata": "true", 8 "config-file-type": "file", 9 "config-file-value": "/json.conf" 10 } 11 } 12 }, 13 { 14 "essential": true, 15 16 "name": "app", 17 "logConfiguration": { 18 "logDriver": "awsfirelens", 19 "options": { 20 "Name": "cloudwatch", 21 "region": "us-west-2", 22 "log_group_name": "firelens-fluent-bit", 23 "log_stream_prefix": "from-fluent-bit", 24 "include-pattern": "[Ee]rror" 25 } 26 } 27 } 28 "essential": true, "firelensConfiguration": { "type": "fluentbit", "options": { "enable-ecs-log-metadata": "true", "config-file-type": "file", "config-file-value": "/json.conf" } } { 1 2 "image": "amazon/aws-for-fluent-bit:latest", 3 "name": "log_router", 4 5 6 7 8 9 10 11 12 }, 13 { 14 "essential": true, 15 "image": "", 16 "name": "app", 17 "logConfiguration": { 18 "logDriver": "awsfirelens", 19 "options": { 20 "Name": "cloudwatch", 21 "region": "us-west-2", 22 "log_group_name": "firelens-fluent-bit", 23 "log_stream_prefix": "from-fluent-bit", 24 "include-pattern": "[Ee]rror" 25 } 26 } 27 } 28 "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "cloudwatch", "region": "us-west-2", "log_group_name": "firelens-fluent-bit", "log_stream_prefix": "from-fluent-bit", "include-pattern": "[Ee]rror" } { 1 "essential": true, 2 "image": "amazon/aws-for-fluent-bit:latest", 3 "name": "log_router", 4 "firelensConfiguration": { 5 "type": "fluentbit", 6 "options": { 7 "enable-ecs-log-metadata": "true", 8 "config-file-type": "file", 9 "config-file-value": "/json.conf" 10 } 11 } 12 }, 13 { 14 "essential": true, 15 "image": "", 16 "name": "app", 17 18 19 20 21 22 23 24 25 26 } 27 } 28 [INPUT] Name forward unix_path /var/run/fluent.sock [INPUT] Name forward Listen 0.0.0.0 Port 24224 [FILTER] Name grep Match app-firelens* Regex log [Ee]rror [FILTER] Name record_modifier Match * Record ecs_cluster firelens-example Record ecs_task_arn arn:aws:ecs:us-west-2:0123456 Record ecs_task_definition firelens-example-sessi @INCLUDE /json.conf [OUTPUT] Name cloudwatch Match app-firelens* region us-west-2 log_group_name firelens-fluent-bit log_stream_prefix from-fluent-bit 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 [INPUT] Name forward unix_path /var/run/fluent.sock [INPUT] Name forward Listen 0.0.0.0 Port 24224 1 2 3 4 5 6 7 8 9 [FILTER] 10 Name grep 11 Match app-firelens* 12 Regex log [Ee]rror 13 14 [FILTER] 15 Name record_modifier 16 Match * 17 Record ecs_cluster firelens-example 18 Record ecs_task_arn arn:aws:ecs:us-west-2:0123456 19 Record ecs_task_definition firelens-example-sessi 20 21 @INCLUDE /json.conf 22 23 [OUTPUT] 24 Name cloudwatch 25 Match app-firelens* 26 region us-west-2 27 log_group_name firelens-fluent-bit 28 log_stream_prefix from-fluent-bit 29 [FILTER] Name record_modifier Match * Record ecs_cluster firelens-example Record ecs_task_arn arn:aws:ecs:us-west-2:0123456 Record ecs_task_definition firelens-example-sessi @INCLUDE /json.conf [INPUT] 1 Name forward 2 unix_path /var/run/fluent.sock 3 4 [INPUT] 5 Name forward 6 Listen 0.0.0.0 7 Port 24224 8 9 [FILTER] 10 Name grep 11 Match app-firelens* 12 Regex log [Ee]rror 13 14 15 16 17 18 19 20 21 22 23 [OUTPUT] 24 Name cloudwatch 25 Match app-firelens* 26 region us-west-2 27 log_group_name firelens-fluent-bit 28 log_stream_prefix from-fluent-bit 29 [FILTER] Name grep Match app-firelens* Regex log [Ee]rror [OUTPUT] Name cloudwatch Match app-firelens* region us-west-2 log_group_name firelens-fluent-bit log_stream_prefix from-fluent-bit [INPUT] 1 Name forward 2 unix_path /var/run/fluent.sock 3 4 [INPUT] 5 Name forward 6 Listen 0.0.0.0 7 Port 24224 8 9 10 11 12 13 14 [FILTER] 15 Name record_modifier 16 Match * 17 Record ecs_cluster firelens-example 18 Record ecs_task_arn arn:aws:ecs:us-west-2:0123456 19 Record ecs_task_definition firelens-example-sessi 20 21 @INCLUDE /json.conf 22 23 24 25 26 27 28 29 ECS task defition fluent-bit.conf 

FireLens internal AWS Blog: Under the hood: FireLens for Amazon ECS Tasks 

FireLens internal ECS generates the Fluent Bit config file in Firelens container /fluent-bit/etc/fluent-bit.conf Fluent TCP interface FLUENT_HOST (127.0.0.1), FLUENT_PORT (24224) Use SDK to send logs into Fluent Bit instead of stdout 

FireLens internal FireLens tag -firelens- app-firelens-dcef9dee-d960-4af8-a206-46c31a7f1e67 Currently, cannot customized this tag Workaround Fluent Bit stream processor CREATE STREAM myTag WITH (tag='myTag') AS SELECT * from TAG:'app-firelens-*' 

FireLens log loss test AWS Blog: Under the hood: FireLens for Amazon ECS Tasks 

FireLens resource usage AWS Blog: Under the hood: FireLens for Amazon ECS Tasks 

Future improvement 

Future improvement Enhance the reliability of FireLens on Fargate ( ) Logs could be lost if Fluent Bit container goes down unexpectedly Firelens container needs to be essential Fargate containers are ephemeral #700 

Better Support for Multiple Go Plugins $ ls -lh bin/fluent-bit *.so root root 18M Dec 12 23:14 bin/fluent-bit root root 24M Jan 08 20:50 cloudwatch.so root root 26M Dec 12 23:04 firehose.so root root 26M Dec 12 23:03 kinesis.so Future improvement 

Better Support for Multiple Go Plugins Future improvement 

Better Support for Multiple Go Plugins Future improvement 

Any Question? Golang Taipei Telegram Golang Taipei Facebook 問卷往這邊！！