Share
Embed
Reliable Log Aggregation System in Multi-Tenant Kubernetes cluster
Hiroki Sakamoto
LINE / Verda Reliability Engineering Team / Site Reliability Engineer
https://linedevday.linecorp.com/2021/ja/sessions/40
https://linedevday.linecorp.com/2021/en/sessions/40
https://linedevday.linecorp.com/2021/ko/sessions/40
LINE DEVDAY 2021
November 10, 2021
Transcript
99 slides
-
Speaker: Hiroki Sakamoto - Role: Site Reliability Engineer at Verda
- Mission: Improving System Reliability - Interest: Kubernetes, Distributed System -
What is Verda? is based on OpenStack. since 2016~ FaaS
PaaS IaaS NAT LB Bare metal -
SRE Teams for Verda Platform wide SRE Provide Verda-Internal platform
to improve Verda services reliability Infra Management Manage our physical infrastructure resources to host Verda services -
SRE Teams for Verda Platform wide SRE Provide Verda-Internal platform
to improve Verda services reliability Here! -
Presentation Targets Targets - The people who address the
issues similar to ours - The people who are thinking about Multi-Tenant architecture - The people who can make decision about architecture Theme - Config Management in Multi-Tenant Kuberenetes - Operation for Fluentd in Multi-Tenant Kubernetes -
Benefits - Get a useful idea to address the
config management in Multi- Tenant Kubernetes - Get knowledge about log management and config management in Multi-Tenant Kuberetes before your release -
Provide Multi-Tenant Kubernetes for Verda Purpose - Aggregate infra resources
- Standardize operations - Provide internal platform tools to reduce operation costs /PWB /FVUSPO ,FZTUPOF .POJUPSJOH -PBECBMBODFS /"5 %BTICPBSE %FTJHOBUF $JOEFS -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch - Fluentd and Logrotated are in a Pod as sidecars - Emptydir is used to share log files among the containers -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch Generate logs! -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch Tail logs -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch Send logs -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch Rotate logs If needed -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch Pain Points - Too many sidecars in all of the Pods - All developers must maintain Fluentd regardless of their knowledge - Lack of monitoring, taking care of performance, reliability and durability -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch Hard to schedule pods efficiently due to too many containers!! -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch Hard to schedule pods efficiently due to too many containers!! Quality depends on each teams -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch Hard to schedule pods efficiently due to too many containers!! Quality depends on each teams Need to send “Audit logs” but don’t have enough monitoring -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch 5XFNPKJzCZ$PQZSJHIU5XJUUFS *ODBOEPUIFSDPOUSJCVUPSTJTMJDFOTFEVOEFS$$#: -
Original log aggregation mechanism /PWB"1* emptydir -PHSPUBUFE /FVUSPO"1* emptydir -PHSPUBUFE
Elasticsearch We need to re-think! -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Managed scope -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Forwarders - Collect logs and send them to aggregators - Deployed as Daemonset It means that a node has only one Fluentd container -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Aggregators - Receive logs from forwarders - Process and filter logs - Send logs to datastore like ES - Deployed as StatefulSet - With PersistentVolume -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Why split? - Not need much resource for Daemonset - Improve Scalability - Reduce changing scope when deploying -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Pods output logs to stdout -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
"HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout Forwarder Docker log driver copies from stdout to other log file logfile logfile -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
"HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout Forwarder Tail from the logs logfile logfile -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
"HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout Forwarder Aggregate and process logs logfile logfile -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
"HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout Forwarder Send logs logfile logfile -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Fluentd is shared resource -
For durability /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout "HHSFHBUPS/PEFT
Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout Forwarder - Buffer logs in each host directory - Flush buffers at shutdown - Save the position it’s already read into files - Require ack response from aggregator logfile logfile -
For durability /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile
"HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder - Buffer logs in each PersistentVolume - Flush buffers at shutdown - Distribute Pods across nodes -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Pros - Developers don’t need to maintain Fluentd - Fluentd can buffer logs while the destinations are down - Easy to scale aggregators - Monitored by SRE Team so developers don’t need to do that - Ensured durability, reliability and performance by SRE Team -
Managed Fluentd Cluster /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile /PEF# ,FZTUPOF /"5 -PBECBMBODFS stdout stdout stdout logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Forwarder Aggregator Aggregator Aggregator So... How do developers apply their own logging config? 5XFNPKJzCZ$PQZSJHIU5XJUUFS *ODBOEPUIFSDPOUSJCVUPSTJTMJDFOTFEVOEFS$$#: -
Shared Fluentd Issues /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder -
Shared Fluentd Issues /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder A team would apply... -
Shared Fluentd Issues /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder The other team would apply... -
Shared Fluentd Issues /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Conflict! Conflict! Conflict! -
Shared Fluentd Issues /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Some team would apply broken config... ! -
Shared Fluentd Issues /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Process down!! -
Shared Fluentd Issues /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout
logfile "HHSFHBUPS/PEFT Elasticsearch Forwarder Aggregator Aggregator Aggregator /PEF# /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile Forwarder Process down!! What happen?! -
Shared Fluentd Issues - Changes to config would be conflicted
among some teams - Invalid config causes process down - Developers need to take care of Fluentd when applying config -
Requirements - All configs should be validated before applying
- All configs shouldn’t affect other teams’ configs - All configs shouldn’t cause process down - All applying config shouldn’t make developers operate manually -
LoggingPipeline Forwarder Config Fluentd Config Operator Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE -
Fluentd Config Operator LoggingPipeline Forwarder Config Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE - Automatically validate config written in CRD “LoggingPipeline” - Automatically compile the Fluentd config to CM if the config is valid - Automatically notify Fluentd to reload new config - Automatically block config if the config is invalid -
LoggingPipeline Forwarder Config Fluentd Config Operator Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE Apply to specify managed Fluentd -
LoggingPipeline Forwarder Config Fluentd Config Operator Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE Apply logging config -
LoggingPipeline Forwarder Config Fluentd Config Operator Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE Start to reoncile -
LoggingPipeline Forwarder Config Fluentd Config Operator Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE Compile config to Configmaps for validation for forwarder for aggregator -
LoggingPipeline Forwarder Config Fluentd Config Operator Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE for forwarder for aggregator Start validation -
LoggingPipeline Forwarder Config Fluentd Config Operator Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE Compile & update configs If success -
LoggingPipeline Forwarder Config Fluentd Config Operator Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE Notify Fluentd specified in FluentdNode CRD if config get updated -
Fluentd Config Operator LoggingPipeline Forwarder Config Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE All developers need to do is specify log source and destination in CRD. 5XFNPKJzCZ$PQZSJHIU5XJUUFS *ODBOEPUIFSDPOUSJCVUPSTJTMJDFOTFEVOEFS$$#: -
Fluentd Config Operator LoggingPipeline Forwarder Config Forwarder Aggregator Fluentd
Config Operator Aggregator Config FluentdNode Developers SRE Dive into this more detail -
Compile LoggingPipeline - Compile separately for forwarders and aggregators
- Automatically complicate important parameters - Automatically wrap config with label to isolate it not to affect others - Automatically change directory to buffer logs to ensure durability -
Config Validation - Run static validation - Run pod
with dry-run command for forwarders - Run pod with actual-run to ensure connectivity to destinations for aggregators -
Load Test /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile
/PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile -
Load Test /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile
/PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile "HHSFHBUPS/PEFT Forwarder Aggregator Aggregator Aggregator Forwarder Dark launch in prod -
Load Test /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile
/PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile "HHSFHBUPS/PEFT Forwarder Aggregator Aggregator Aggregator Forwarder About 250GB/day -
Load Test Results… The CPU usage of some aggregators
got very higher periodically regardless of plenty of aggregator instances - Event thread in Fluentd was hanging up - Connections between the aggregator and forwarders were not too much - It means that aggregation, processing and writing buffer are heavy - But I/O was not hanging up -
Load Test Results… The CPU usage of some aggregators
got very higher periodically regardless of plenty of aggregator instances - Event thread in Fluentd was hanging up - Connections between the aggregator and forwarders were not too much - It means that aggregation, processing and writing buffer are heavy - But I/O was not hanging up Log chunk size may be too much Let’s make chunk size be lower! 5XFNPKJzCZ$PQZSJHIU5XJUUFS *ODBOEPUIFSDPOUSJCVUPSTJTMJDFOTFEVOEFS$$#: -
Load Test Results… The CPU usage of some aggregators
got very higher periodically regardless of plenty of aggregator instances - Event thread in Fluentd was hanging up - Connections between the aggregator and forwarders were not too much - It means that aggregation, processing and writing buffer are heavy - But I/O was not hanging up Resolved 5XFNPKJzCZ$PQZSJHIU5XJUUFS *ODBOEPUIFSDPOUSJCVUPSTJTMJDFOTFEVOEFS$$#: -
Monitoring AlertManager Cluster Remote write Monitor VM Alert Fire alerts
Query periodically /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile "HHSFHBUPS/PEFT Forwarder Aggregator Aggregator Aggregator Forwarder -
Monitoring AlertManager Cluster Remote write Monitor VM Alert Fire alerts
Query periodically /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile "HHSFHBUPS/PEFT Forwarder Aggregator Aggregator Aggregator Forwarder - Prometheus in the cluster scrape metrics from Fluentd containers - It is monitored by a Prometheus which is out of the cluster - The Prometheus writes metrics to VictoriaMetrics TSDB - VM Alert query pre-defined rules for VictoriaMetrics periodically - VM Alert fire alerts to AlertManager if match with the rules - AlertManager sends notifications to the destinations like Slack, PagerDuty -
Monitoring - CPU, Memory Usage - Process down, Pod down,
Pod restart count - Whether there is no logs which is sent to the destination - log inflow speed < log processing speed - Disk usage for buffering and buffered bytes - Number of errors and slow flush about Fluentd - Number of errors about Fluentd Config Operator -
Project Results Provided Managed Fluentd Cluster and Fluentd Config
Operator - Got developers off maintaining Fluentd - All developers need to do is to manage their own logging config itself - Reduced about 172 containers in a cluster - Improve reliability, durability and performance about logging - Found undetected error about logging by monitoring -
However… Some issues occurs after release - Docker JSON
Log Driver splits the logs more than 16k so broken json log come to our Fluentd… - We need mechanism to notify developers parsing error -
Introduce Kafka and… Elasticsearch /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout
stdout logfile /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile "HHSFHBUPS/PEFT Forwarder Aggregator Aggregator Aggregator Forwarder -
Introduce Kafka and… Elasticsearch /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout
stdout logfile /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile "HHSFHBUPS/PEFT Forwarder Aggregator Aggregator Aggregator Forwarder - Remove direct dependency between forwarder and aggregator to improve scalability - Enable developers to send logs from out of the cluster -
Introduce Kafka and… Elasticsearch /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout
stdout logfile /PEF" /PWB /FVUSPO ,FZTUPOF stdout stdout stdout logfile "HHSFHBUPS/PEFT Forwarder Aggregator Aggregator Aggregator Forwarder Standardize logging across all Verda services!