Centralized Logging Solution using FireLens and Fluent Bit for ECS

Transcript

1. Centralized Logging Solution using FireLens and Fluent Bit for ECS David Chou @ Golang Taipei
2. @ Umbo Computer Vision @ Golang Taipei david74.chou @ facebook david74.chou @ medium david7482 @ github Golang Taipei Telegram Golang Taipei Facebook
3. Why do we need centralized logging?
4. Why do we need centralized logging? Usability Availability Analyzability
5. Centralized logging architecture AWS Container Logging Deep Dive
7. Implemented in Ruby/C @ 2011 Ecosystem of Plugins (900+) Extend with Ruby Memory Usage (40MB) Higher CPU Usage Better for Aggregation Support Forward Protocol Implemented in C @ 2015 Plugins are Included (56) Extend with C/Go/Lua Reduced Memory Usage (500KB) Lower CPU Usage Better as a Sidecar Support Forward Protocol KubeCon 2019: Fluent Bit Extending Your Logging Pipeline with Go
8. AWS Blog: Centralized Container Logging with Fluent Bit
9. Output 1 Output 2 Output N Input Parser Filter Buffer Routing
10. Output 1 Output 2 Output N Input Parser Filter Information gathering, e.g. forward, tail, syslog, memory Buffer Routing
11. Output 1 Output 2 Output N Input Parser Filter Unstructured logs => Structured binary format, e.g. json, regx (nginx, syslog, etc) Buffer Routing
12. Output 1 Output 2 Output N Input Parser Filter Input data manipulation, e.g. grep, record_modifier Buffer Routing
13. Output 1 Output 2 Output N Input Parser Filter Message buffering & retry Buffer Routing
14. Output 1 Output 2 Output N Input Parser Filter Tag routing Buffer Routing
15. Output 1 Output 2 Output N Input Parser Filter Send data to different destinations, e.g. forward, es, Go plugins Buffer Routing
16. How to write a Fluent Bit Go plugin?
17. Fluent Bit Go Plugin fluent-bit plugin.so dynamic loading
18. //export FLBPluginRegister func FLBPluginRegister(def unsafe.Pointer) int { // Gets called only once when the .so is loaded. return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") } //export FLBPluginInit func FLBPluginInit(plugin unsafe.Pointer) int { // Gets called once for each instance you have configured. param := output.FLBPluginConfigKey(plugin, "param") return output.FLB_OK } //export FLBPluginFlushCtx func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c // Gets called once for each message to be written to an instance. } //export FLBPluginExit func FLBPluginExit() int { // Gets called on teardown. return output.FLB_OK } func main() { } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 //export FLBPluginRegister func FLBPluginRegister(def unsafe.Pointer) int { // Gets called only once when the .so is loaded. return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") } 1 2 3 4 5 6 //export FLBPluginInit 7 func FLBPluginInit(plugin unsafe.Pointer) int { 8 // Gets called once for each instance you have configured. 9 param := output.FLBPluginConfigKey(plugin, "param") 10 return output.FLB_OK 11 } 12 13 //export FLBPluginFlushCtx 14 func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c 15 // Gets called once for each message to be written to an instance. 16 } 17 18 //export FLBPluginExit 19 func FLBPluginExit() int { 20 // Gets called on teardown. 21 return output.FLB_OK 22 } 23 24 func main() { 25 } 26 //export FLBPluginInit func FLBPluginInit(plugin unsafe.Pointer) int { // Gets called once for each instance you have configured. param := output.FLBPluginConfigKey(plugin, "param") return output.FLB_OK } //export FLBPluginRegister 1 func FLBPluginRegister(def unsafe.Pointer) int { 2 // Gets called only once when the .so is loaded. 3 return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") 4 } 5 6 7 8 9 10 11 12 13 //export FLBPluginFlushCtx 14 func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c 15 // Gets called once for each message to be written to an instance. 16 } 17 18 //export FLBPluginExit 19 func FLBPluginExit() int { 20 // Gets called on teardown. 21 return output.FLB_OK 22 } 23 24 func main() { 25 } 26 //export FLBPluginFlushCtx func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c // Gets called once for each message to be written to an instance. } //export FLBPluginRegister 1 func FLBPluginRegister(def unsafe.Pointer) int { 2 // Gets called only once when the .so is loaded. 3 return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") 4 } 5 6 //export FLBPluginInit 7 func FLBPluginInit(plugin unsafe.Pointer) int { 8 // Gets called once for each instance you have configured. 9 param := output.FLBPluginConfigKey(plugin, "param") 10 return output.FLB_OK 11 } 12 13 14 15 16 17 18 //export FLBPluginExit 19 func FLBPluginExit() int { 20 // Gets called on teardown. 21 return output.FLB_OK 22 } 23 24 func main() { 25 } 26 //export FLBPluginExit func FLBPluginExit() int { // Gets called on teardown. return output.FLB_OK } //export FLBPluginRegister 1 func FLBPluginRegister(def unsafe.Pointer) int { 2 // Gets called only once when the .so is loaded. 3 return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") 4 } 5 6 //export FLBPluginInit 7 func FLBPluginInit(plugin unsafe.Pointer) int { 8 // Gets called once for each instance you have configured. 9 param := output.FLBPluginConfigKey(plugin, "param") 10 return output.FLB_OK 11 } 12 13 //export FLBPluginFlushCtx 14 func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c 15 // Gets called once for each message to be written to an instance. 16 } 17 18 19 20 21 22 23 24 func main() { 25 } 26 func main() { } //export FLBPluginRegister 1 func FLBPluginRegister(def unsafe.Pointer) int { 2 // Gets called only once when the .so is loaded. 3 return output.FLBPluginRegister(def, "gstdout", "Stdout GO!") 4 } 5 6 //export FLBPluginInit 7 func FLBPluginInit(plugin unsafe.Pointer) int { 8 // Gets called once for each instance you have configured. 9 param := output.FLBPluginConfigKey(plugin, "param") 10 return output.FLB_OK 11 } 12 13 //export FLBPluginFlushCtx 14 func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.c 15 // Gets called once for each message to be written to an instance. 16 } 17 18 //export FLBPluginExit 19 func FLBPluginExit() int { 20 // Gets called on teardown. 21 return output.FLB_OK 22 } 23 24 25 26
19. # Build fluent bit go plugin $ go build -o plugin.so -buildmode c-shared plugin.go # Run Fluent Bit with the new plugin $ fluent-bit -e plugin.so -i cpu -o gstdout
20. # Build fluent bit go plugin $ go build -o plugin.so -buildmode c-shared plugin.go # Run Fluent Bit with the new plugin $ fluent-bit -e plugin.so -i cpu -o gstdout Go build modes default archive exe pie shared plugin c-archive c-shared
21. AWS FireLens
22. AWS FireLens Support a wide-array of AWS Services as log destinations Require no additional configuration beyond ECS Task Definition Use Open Source for extensibility: Fluent Bit, Fluentd Facilitate partner integration AWS Blog: Under the hood: FireLens for Amazon ECS Tasks
23. AWS for Fluent Bit A Fluent Bit image with AWS Go plugins on Docker Hub Regional public images hosted on ECR amazon/aws-for-fluent-bit
24. AWS for Fluent Bit - Log destination Kinesis Data Firehose S3 Amazon ElasticSearch Service Kinesis Data Streams CloudWatch Logs Kafka Self-hosted ElasticSearch ( ) DataDog Forward to a Fluentd aggregator #698
25. AWS for Fluent Bit - Log destination $ cat entrypoint.sh exec /fluent-bit/bin/fluent-bit \ -e /fluent-bit/firehose.so \ -e /fluent-bit/cloudwatch.so \ -e /fluent-bit/kinesis.so \ -c /fluent-bit/etc/fluent-bit.conf github: aws-for-fluent-bit
26. Use AWS FireLens in ECS aws-samples/amazon-ecs-firelens-examples
27. { "essential": true, "image": "amazon/aws-for-fluent-bit:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit", "options": { "enable-ecs-log-metadata": "true", "config-file-type": "file", "config-file-value": "/json.conf" } } }, { "essential": true, "image": "<your-app-image>", "name": "app", "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "cloudwatch", "region": "us-west-2", "log_group_name": "firelens-fluent-bit", "log_stream_prefix": "from-fluent-bit", "include-pattern": "[Ee]rror" } } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 "image": "amazon/aws-for-fluent-bit:latest", "image": "<your-app-image>", { 1 "essential": true, 2 3 "name": "log_router", 4 "firelensConfiguration": { 5 "type": "fluentbit", 6 "options": { 7 "enable-ecs-log-metadata": "true", 8 "config-file-type": "file", 9 "config-file-value": "/json.conf" 10 } 11 } 12 }, 13 { 14 "essential": true, 15 16 "name": "app", 17 "logConfiguration": { 18 "logDriver": "awsfirelens", 19 "options": { 20 "Name": "cloudwatch", 21 "region": "us-west-2", 22 "log_group_name": "firelens-fluent-bit", 23 "log_stream_prefix": "from-fluent-bit", 24 "include-pattern": "[Ee]rror" 25 } 26 } 27 } 28 "essential": true, "firelensConfiguration": { "type": "fluentbit", "options": { "enable-ecs-log-metadata": "true", "config-file-type": "file", "config-file-value": "/json.conf" } } { 1 2 "image": "amazon/aws-for-fluent-bit:latest", 3 "name": "log_router", 4 5 6 7 8 9 10 11 12 }, 13 { 14 "essential": true, 15 "image": "<your-app-image>", 16 "name": "app", 17 "logConfiguration": { 18 "logDriver": "awsfirelens", 19 "options": { 20 "Name": "cloudwatch", 21 "region": "us-west-2", 22 "log_group_name": "firelens-fluent-bit", 23 "log_stream_prefix": "from-fluent-bit", 24 "include-pattern": "[Ee]rror" 25 } 26 } 27 } 28 "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "cloudwatch", "region": "us-west-2", "log_group_name": "firelens-fluent-bit", "log_stream_prefix": "from-fluent-bit", "include-pattern": "[Ee]rror" } { 1 "essential": true, 2 "image": "amazon/aws-for-fluent-bit:latest", 3 "name": "log_router", 4 "firelensConfiguration": { 5 "type": "fluentbit", 6 "options": { 7 "enable-ecs-log-metadata": "true", 8 "config-file-type": "file", 9 "config-file-value": "/json.conf" 10 } 11 } 12 }, 13 { 14 "essential": true, 15 "image": "<your-app-image>", 16 "name": "app", 17 18 19 20 21 22 23 24 25 26 } 27 } 28 [INPUT] Name forward unix_path /var/run/fluent.sock [INPUT] Name forward Listen 0.0.0.0 Port 24224 [FILTER] Name grep Match app-firelens* Regex log [Ee]rror [FILTER] Name record_modifier Match * Record ecs_cluster firelens-example Record ecs_task_arn arn:aws:ecs:us-west-2:0123456 Record ecs_task_definition firelens-example-sessi @INCLUDE /json.conf [OUTPUT] Name cloudwatch Match app-firelens* region us-west-2 log_group_name firelens-fluent-bit log_stream_prefix from-fluent-bit 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 [INPUT] Name forward unix_path /var/run/fluent.sock [INPUT] Name forward Listen 0.0.0.0 Port 24224 1 2 3 4 5 6 7 8 9 [FILTER] 10 Name grep 11 Match app-firelens* 12 Regex log [Ee]rror 13 14 [FILTER] 15 Name record_modifier 16 Match * 17 Record ecs_cluster firelens-example 18 Record ecs_task_arn arn:aws:ecs:us-west-2:0123456 19 Record ecs_task_definition firelens-example-sessi 20 21 @INCLUDE /json.conf 22 23 [OUTPUT] 24 Name cloudwatch 25 Match app-firelens* 26 region us-west-2 27 log_group_name firelens-fluent-bit 28 log_stream_prefix from-fluent-bit 29 [FILTER] Name record_modifier Match * Record ecs_cluster firelens-example Record ecs_task_arn arn:aws:ecs:us-west-2:0123456 Record ecs_task_definition firelens-example-sessi @INCLUDE /json.conf [INPUT] 1 Name forward 2 unix_path /var/run/fluent.sock 3 4 [INPUT] 5 Name forward 6 Listen 0.0.0.0 7 Port 24224 8 9 [FILTER] 10 Name grep 11 Match app-firelens* 12 Regex log [Ee]rror 13 14 15 16 17 18 19 20 21 22 23 [OUTPUT] 24 Name cloudwatch 25 Match app-firelens* 26 region us-west-2 27 log_group_name firelens-fluent-bit 28 log_stream_prefix from-fluent-bit 29 [FILTER] Name grep Match app-firelens* Regex log [Ee]rror [OUTPUT] Name cloudwatch Match app-firelens* region us-west-2 log_group_name firelens-fluent-bit log_stream_prefix from-fluent-bit [INPUT] 1 Name forward 2 unix_path /var/run/fluent.sock 3 4 [INPUT] 5 Name forward 6 Listen 0.0.0.0 7 Port 24224 8 9 10 11 12 13 14 [FILTER] 15 Name record_modifier 16 Match * 17 Record ecs_cluster firelens-example 18 Record ecs_task_arn arn:aws:ecs:us-west-2:0123456 19 Record ecs_task_definition firelens-example-sessi 20 21 @INCLUDE /json.conf 22 23 24 25 26 27 28 29 ECS task defition fluent-bit.conf
28. FireLens internal AWS Blog: Under the hood: FireLens for Amazon ECS Tasks
29. FireLens internal ECS generates the Fluent Bit config file in Firelens container /fluent-bit/etc/fluent-bit.conf Fluent TCP interface FLUENT_HOST (127.0.0.1), FLUENT_PORT (24224) Use SDK to send logs into Fluent Bit instead of stdout
30. FireLens internal FireLens tag <container name>-firelens-<task ID> app-firelens-dcef9dee-d960-4af8-a206-46c31a7f1e67 Currently, cannot customized this tag Workaround Fluent Bit stream processor CREATE STREAM myTag WITH (tag='myTag') AS SELECT * from TAG:'app-firelens-*'
31. FireLens log loss test AWS Blog: Under the hood: FireLens for Amazon ECS Tasks
32. FireLens resource usage AWS Blog: Under the hood: FireLens for Amazon ECS Tasks
33. Future improvement
34. Future improvement Enhance the reliability of FireLens on Fargate ( ) Logs could be lost if Fluent Bit container goes down unexpectedly Firelens container needs to be essential Fargate containers are ephemeral #700
35. Better Support for Multiple Go Plugins $ ls -lh bin/fluent-bit *.so root root 18M Dec 12 23:14 bin/fluent-bit root root 24M Jan 08 20:50 cloudwatch.so root root 26M Dec 12 23:04 firehose.so root root 26M Dec 12 23:03 kinesis.so Future improvement
36. Better Support for Multiple Go Plugins Future improvement
37. Better Support for Multiple Go Plugins Future improvement
38. Any Question? Golang Taipei Telegram Golang Taipei Facebook 問卷往這邊！！