Elastic at Paylocity: Analyzing 800GB a day

Elastic{ON} Tour Chicago - September 13, 2017

Learn how Paylocity uses Elastic for analysis.

Justin Purdy | Systems Administrator | Paylocity

Transcript

1. 1 Justin Purdy Systems Administrator, Paylocity Elastic at Paylocity: Analyzing 800GB a day
2. 2 2 Way back in 2013…
3. 3 Starting Architecture Beats Log Files Windows Events Firewall Load Balancers Nodes (2) Logstash Elasticsearch Kibana X-pack Authentication X-pack AD Instances (2) Master/Data Nodes (4) Version 5.0 across the board 2TB Pure storage per elasticsearch node The worst grok filter ever
4. 4 For real, it was so bad… grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" \"(?<x_forwarded_for>%{IP:xff_clientip}(?:, .*)?)\" %{INT:TimeTakenMS}" ] add_tag => "iislogtype_A“ tag_on_failure => "nottypeA“ } if "nottypeA" in [tags] { grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" - %{INT:TimeTakenMS}" ] add_tag => "iislogtype_A2“ tag_on_failure => "nottypeA2“ } mutate { remove_tag => [ "nottypeA" ] } } if "nottypeA2" in [tags] { grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" %{INT:TimeTakenMS} \"(?<x_forwarded_for>%{IP:xff_clientip}(?:, .*)?)\"" ] add_tag => "iislogtype_B" tag_on_failure => "nottypeB“ } mutate { remove_tag => [ "nottypeA2" ] } } if "nottypeB" in [tags] { grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" %{INT:TimeTakenMS} -" ] add_tag => "iislogtype_B2“ } mutate { remove_tag => [ "nottypeB" ] } }
5. 5 For real, it was so bad… grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" \"(?<x_forwarded_for>%{IP:xff_clientip}(?:, .*)?)\" %{INT:TimeTakenMS}" ] }
6. 6 For real, it was so bad… grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc- status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" \"(?<x_forwarded_for>%{IP:xff_clientip}(?:, .*)?)\" %{INT:TimeTakenMS}" ] add_tag => "iislogtype_A“ tag_on_failure => "nottypeA“ } if "nottypeA" in [tags] { grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" - %{INT:TimeTakenMS}" ] add_tag => "iislogtype_B“ tag_on_failure => "nottypeB“ } mutate { remove_tag => [ "nottypeA" ] }
7. 7 For real, it was so bad… grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" \"(?<x_forwarded_for>%{IP:xff_clientip}(?:, .*)?)\" %{INT:TimeTakenMS}" ] add_tag => "iislogtype_A“ tag_on_failure => "nottypeA“ } if "nottypeA" in [tags] { grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" - %{INT:TimeTakenMS}" ] add_tag => "iislogtype_A2“ tag_on_failure => "nottypeA2“ } mutate { remove_tag => [ "nottypeA" ] } } if "nottypeA2" in [tags] { grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" %{INT:TimeTakenMS} \"(?<x_forwarded_for>%{IP:xff_clientip}(?:, .*)?)\"" ] add_tag => "iislogtype_B" tag_on_failure => "nottypeB“ } mutate { remove_tag => [ "nottypeA2" ] } } if "nottypeB" in [tags] { grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" %{INT:TimeTakenMS} -" ] add_tag => "iislogtype_B2“ } mutate { remove_tag => [ "nottypeB" ] } }
8. 8 NOTE: USE THIS LAYOUT FOR PLACING ONE IMAGE THAT SUPPORTS YOUR STORY
9. 9 It was this simple… (?:\"%{DATA}\"|-)
10. 10 Not as pretty, but just as elegant grok { match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH} %{IP} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-bytes} %{WORD} %{NOTSPACE} %{IPORHOST} %{NOTSPACE} %{NOTSPACE} %{QS} (?:\"%{DATA}\"|-) %{NOTSPACE} %{NUMBER} %{NUMBER:cs-bytes} (?:\"%{DATA}\"|-) \"%{DATA}\" %{INT:time_taken|-|\"(?<x_forwarded_for>%{HOSTNAME:xff_clientip}(?:, .*)?|-)(:%{INT}|)\"} %{INT:time_taken|-|\"(?<x_forwarded_for>%{HOSTNAME:xff_clientip}(?:, .*)?|-)(:%{INT}|)\"}$" ] add_tag => “grokked“ }
11. 11 For Fools rush in where Angels fear to tread. - Alexander Pope
12. 12 “Upgraded” Architecture Diagram Beats Log Files Metrics Windows Events Firewall Load Balancers Elasticsearch X-pack Master/Data Nodes (8) Kibana X-pack Instances (2) Nodes (2) Logstash Authentication AD MOAR NODES Bumped each ES node up to 4TB, then 6TB Balanced index sizes
13. 13 Actually Upgraded Architecture Diagram Elasticsearch Kibana X-pack Authentication Notification X-pack AD Instances (2) Master Nodes (3) Coordinating Nodes (3) Data Nodes (10) Beats Log Files Metrics Windows Events Firewall Load Balancers Nodes (2) Logstash Heartbeats X-pack Cloud Monitoring Chad came to visit Dedicated roles Still more nodes
14. 14 Current Architecture Diagram Elasticsearch Kibana X-pack Authentication Notification X-pack AD Instances (2) Master Nodes (3) Coordinating Nodes (3) Data Nodes (10) Beats Log Files Metrics Windows Events Firewall Load Balancers Nodes (2) Logstash Heartbeats X-pack Cloud Monitoring Nodes (3) Production Non-production Master Nodes (3) Data Nodes (8) Production Non-production Instances (1) Production Non-production A place to test TLS all the things Cross-cluster search
15. 15 HOSTS Big Numbers DOCUMENTS PER DAY DOCUMENTS INDEXED totaling ~50TB 1400 500M 30B
16. 16 Basic Reindex
17. 17 Reindexing logs, with script
18. 18 Reindexing logs, with script + slicing
19. 19 Slice testing
20. 20 More tuning
21. 21 Zoom!
22. 22 Future Architecture Kafka Messaging Queue Kibana X-pack Instances (3) Firewall Load Balancers Beats Log Files Metrics Windows Events Heartbeats Nodes (3) Logstash X-pack Elasticsearch X-pack Master Nodes (3) Coordinating Nodes (3) Data Nodes (12) Authentication Notification AD Cloud Monitoring Database Redis, SQL Kafka replays More storage, of course
23. 23 23 From this…
24. 24 24 …to this.
25. 25 I enjoy the added flexibility the Elastic Stack has provided our team's efforts. It's allowed us to stretch our reach into new data investigation territory and snap back on adversaries in a shorter period of time. - Abe Miller, InfoSec
26. 26 It's like Elastic is a modern engine, where before we were driving like the Flintstones. - Alex Hooker, InfoSec
27. 27 "The Elastic Stack has armed our teams with the powerful tools and insights to combat previously undetected enemies hidden within our codebase. In short: we can slay dragons now. - Joe Stetzer, Dev Lead
28. 28 Elastic has become the de facto standard for application troubleshooting among our dev teams. - Justin Ivins, Director of Internal Tech Operations
29. 29 Hands up and gimme all yer storage! - Brian Boos, Principal Engineer
30. 30 Justin Purdy Systems Administrator, Paylocity Elastic at Paylocity: Analyzing 800GB a day