Cox Communications Inc. February 28th, 2018 smalenfant@apache.org Traffic Control and Elasticsearch @Cox Steve Malenfant, Engineer

Cox Communications A Broadband communication and entertainment company

3 Enabling new services

4 This is a sample image All of Olympic streaming coverage on all devices

• Team • 6 Engineers. Half split between Dev and Ops. • CDN Deployment • Hundreds of physical servers in 13 datacenters • Hundreds of containers (LXC and Docker) • Apache Traffic Server and Traffic Control • Elasticsearch for access logs CDN Team and Deployment overview

6 • Set of components that can be used to build, monitor, configure, and provision a large scale content delivery network (CDN) http://trafficcontrol.apache.org/ Apache Traffic Control DNS/HTTP client steering to closest/best available cache Implements CDN health protocol and present states to Traffic Router Acquire CDN wide statistics and store information in InfluxDB An API driven configuration management and configuration file generation system A client facing UI used to manage and operate a CDN

No content

Traffic Portal 8 Management interface

• Supported by the enterprise • Minimal amount of access logs (<100GB) • Functional • Search and visualize logs • Provides reports and dashboards • Triggers alarms We have a logging platform - 2015 9

• Explosion in CDN usage • Electronic Program Guides • Images and poster arts • Increasing IP Video Delivery • Experiencing slowdown in reports • Retention time reduction • Filtering events, losing visibility • Getting too expensive… We have a logging problem - 2016 10

Elastic proof of concept – early 2016 Production Lab Edge/Mid Traffic Server Logs Filebeat - File Input - Filters - Add Tags - Beat Output Logstash Indexers N+1 - Beat Input - Filtering (KV) - Redis TCP Localhost Output Logstash Indexers N+1 - Redis Input - Elasticsearch Output Elasticsearch (Data 3..N) Elastic Search - HTTP API - Clustered - Replicated Redis (2) Redis - TCP Input/Output - Buffers events - Processed data

Our current logging pipeline Production Edge/Mid Traffic Server Logs Filebeat - File Input - Filters - Add Tags - Kafka Output Logtash Indexers N+1 - Kafka Input - KV Filter - Elasticsearch Output Elasticsearch (Data 3..N) Elastic Search - HTTP API - Clustered - Replicated Kafka (3) - Multiple Topics - Retention Policies - Replicas Traffic Router Access Logs Filebeat - File Input - Add Tags - Kafka Output Kafka Stream Aggregator N+1 - Downsample data - Elasticsearch Output

• Ansible playbooks with Docker • Elasticsearch • Logstash • Curator Elasticsearch Deployment 13 - name: Create elasticsearch container docker_container: name: "{{ inventory_hostname }}" image: "docker.elastic.co/elasticsearch/elasticsearch:6.1.2" env: xpack.monitoring.enabled: "{{ xpack_monitor_enabled | lower}}” bootstrap.memory_lock: “true” restart_policy: unless-stopped path.data: "{{ item['es_disks'] | join(',') }}" cpu_set: "{{ item.cpu_set | default([]) }}" … ports: - 9200:9200 - 9300:9300

14 Early on… Then… Now… • 5 Physical Servers (192GB) • Single node • 6 x 1.8TB 10k SAS (RAID0) • 10 Physical Servers (192GB) • 3 nodes per servers • 3 x 2TB SSDs • 10 Physical Servers (192GB) • 1 node (96GB tmpfs) - HOT • 1 node (SSDs) - WARM Elastic Nodes deployment

• Using Hot/Warm architecture • tmpfs (RAM) • NVMe (10us writes/500K IOPS) • Hourly indices • Curator to move indexes to SSD Nodes Increasing indexing performance

• This was across 10 Physical hosts - Older E5-26xx. • Limited by Logstash (6 instances) Applying to production

• Lower Logstash indexers CPU usage • Better filtering capabilities? • Enable other consumers • Increase Filebeat CPU usage at Edge 1519769996.088 chi=174.79.69.70 phn=edge01.rd.at.cox.net shn=example.com url=http://cdn.example.ott.cox.net/test cqhm=GET cqhv=HTTP/1.1 pssc=200 ttms=0 b=52505 sssc=200 sscl=52505 cfsc=FIN pfsc=FIN crc=TCP_MISS phr=DIRECT uas="NING/1.0" range="-" JSON instead of RAW KV Logs Improving pipeline efficiency

Dashboard - Examples

Dashboards – Examples 19

Thank You!

Translog – async vs ? 21 Streaming performances