LogStash: Yes, Logging Can Be Awesome

Logging. Everyone does it. Many don't know why they do it. It is often considered a boring chore. A chore that is done by habit rather than for a purpose. But it doesn't have to be! Learn how to build a powerful, scalable open source logging environment with LogStash.

Transcript

1. James Turnbull @kartar Yes, Logging Can Be Awesome
2. who operations chap Puppet chap erstwhile Ruby chap funny accent (photo by Jennie Rainsford)
4. other matters author hack-n-slash developer pontification http://www.jamesturnbull.net https://github.com/jamtur01 http://www.kartar.net
5. books Pro Puppet Pro Linux System Administration Pro Nagios 2.0 Hardening Linux
6. the logstash book
7. So who are you folks?
8. so what's a log (photo by Rick Payette)
9. timestamp + data = log M a y 7 1 6 : 0 7 : 1 0 p e l i n s y s t e m d [ 1 ] : S t a r t i n g C o m m a n d S c h e d u l e r . . . M a y 7 1 6 : 0 7 : 1 0 < t i m e s t a m p p e l i n s y s t e m d [ 1 ] : S t a r t i n g C o m m a n d S c h e d u l e r . . . < d a t a
10. lifecycle of a log
11. actual lifecycle of a log
12. actual actual lifecycle of a log
13. so why isn't logging awesome?
14. I'll tell you a story
15. 1 2 3 . 1 5 1 . 1 4 8 . 1 8 2 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 4 8 : 2 5 - 0 4 0 0 ] " G E T / 2 0 1 0 / 0 8 / r a g - o f - t h e - w e e k - b u s t e d / t r a c k b a c k H T T P / 1 . 1 " 3 0 2 5 " h t t p : / / w w w . s t u m p d i n p d x . c o m / " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; S o s o s p i d e r / 2 . 0 ; + h t t p : / / h e l p . s o s o . c o m / w e b s p i d e r . h t m ) " 1 2 3 . 1 5 1 . 1 4 8 . 1 8 2 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 4 8 : 2 5 - 0 4 0 0 ] " G E T / 2 0 1 0 / 0 8 / r a g - o f - t h e - w e e k - b u s t e d / H T T P / 1 . 1 " 2 0 0 1 1 6 7 8 " h t t p : / / w w w . s t u m p d i n p d x . c o m / " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; S o s o s p i d e r / 2 . 0 ; + h t t p : / / h e l p . s o s o . c o m / w e b s p i d e r . h t m ) " 9 6 . 1 2 6 . 1 2 7 . 1 0 8 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 4 8 : 3 5 - 0 4 0 0 ] " P O S T / w p - c r o n . p h p ? d o i n g _ w p _ c r o n = 1 3 6 8 3 1 9 7 1 5 . 1 5 6 3 2 5 1 0 1 8 5 2 4 1 6 9 9 2 1 8 7 5 H T T P / 1 . 0 " 2 0 0 0 " - " " W o r d P r e s s / 3 . 5 . 1 ; h t t p : / / w w w . s t u m p d i n p d x . c o m " 1 2 3 . 1 5 1 . 1 4 8 . 1 8 2 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 4 8 : 3 5 - 0 4 0 0 ] " G E T / 2 0 1 0 / 0 8 / r a g - o f - t h e - w e e k - b u s t e d / f e e d H T T P / 1 . 1 " 3 0 1 5 " h t t p : / / w w w . s t u m p d i n p d x . c o m / " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; S o s o s p i d e r / 2 . 0 ; + h t t p : / / h e l p . s o s o . c o m / w e b s p i d e r . h t m ) " 1 2 3 . 1 5 1 . 1 4 8 . 1 8 2 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 4 8 : 3 5 - 0 4 0 0 ] " G E T / 2 0 1 0 / 0 8 / r a g - o f - t h e - w e e k - b u s t e d / f e e d / H T T P / 1 . 1 " 2 0 0 2 5 5 9 " h t t p : / / w w w . s t u m p d i n p d x . c o m / " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; S o s o s p i d e r / 2 . 0 ; + h t t p : / / h e l p . s o s o . c o m / w e b s p i d e r . h t m ) " 1 0 7 . 2 0 . 2 0 2 . 4 6 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 5 2 : 3 4 - 0 4 0 0 ] " G E T / f e e d / H T T P / 1 . 1 " 2 0 0 1 3 5 9 6 9 " - " " M o z i l l a / 5 . 0 ( M a c i n t o s h ; U ; I n t e l M a c O S X 1 0 _ 6 _ 6 ; e n - U S ) A p p l e W e b K i t / 5 3 4 . 1 6 ( K H T M L , l i k e G e c k o ) C h r o m e / 1 0 . 0 . 6 4 8 . 2 0 4 S a f a r i / 5 3 4 . 1 6 " 1 0 7 . 2 0 . 2 0 2 . 4 6 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 5 2 : 3 4 - 0 4 0 0 ] " G E T / f e e d / H T T P / 1 . 1 " 2 0 0 1 3 5 9 6 9 " - " " M o z i l l a / 5 . 0 ( M a c i n t o s h ; U ; I n t e l M a c O S X 1 0 _ 6 _ 6 ; e n - U S ) A p p l e W e b K i t / 5 3 4 . 1 6 ( K H T M L , l i k e G e c k o ) C h r o m e / 1 0 . 0 . 6 4 8 . 2 0 4 S a f a r i / 5 3 4 . 1 6 " 9 6 . 1 2 6 . 1 2 7 . 1 0 8 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 5 4 : 0 2 - 0 4 0 0 ] " P O S T / w p - c r o n . p h p ? d o i n g _ w p _ c r o n = 1 3 6 8 3 2 0 0 4 2 . 6 0 6 5 4 9 9 7 8 2 5 6 2 2 5 5 8 5 9 3 7 5 H T T P / 1 . 0 " 2 0 0 0 " - " " W o r d P r e s s / 3 . 5 . 1 ; h t t p : / / w w w . s t u m p d i n p d x . c o m " 9 2 . 6 4 . 2 5 4 . 2 2 5 - - [ 1 1 / M a y / 2 0 1 3 : 2 0 : 5 4 : 0 3 - 0 4 0 0 ] " P O S T / w p - l o g i n . p h p H T T P / 1 . 0 " 2 0 0 4 4 5 2 " - " " M o z i l l a / 3 . 0 ( c o m p a t i b l e ; I n d y L i b r a r y ) " 2 0 9 . 8 5 . 2 3 8 . 2 3 3 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 0 7 : 0 1 - 0 4 0 0 ] " G E T / f e e d / H T T P / 1 . 1 " 2 0 0 4 6 0 9 9 " - " " F e e d f e t c h e r - G o o g l e ; ( + h t t p : / / w w w . g o o g l e . c o m / f e e d f e t c h e r . h t m l ; 4 8 s u b s c r i b e r s ; f e e d - i d = 5 3 1 2 9 6 8 8 3 2 0 4 3 9 7 1 3 4 4 ) " 1 2 1 . 2 1 9 . 5 7 . 1 9 5 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 0 8 : 2 1 - 0 4 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 6 1 4 2 " - " " R e e d e r / 1 0 2 0 . 0 9 . 0 0 C F N e t w o r k / 5 9 6 . 3 . 3 D a r w i n / 1 2 . 3 . 0 ( x 8 6 _ 6 4 ) ( M a c B o o k P r o 8 % 2 C 2 ) " 1 2 1 . 2 1 9 . 5 7 . 1 9 5 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 0 8 : 2 1 - 0 4 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 6 1 4 2 " - " " R e e d e r / 1 0 2 0 . 0 9 . 0 0 C F N e t w o r k / 5 9 6 . 3 . 3 D a r w i n / 1 2 . 3 . 0 ( x 8 6 _ 6 4 ) ( M a c B o o k P r o 8 % 2 C 2 ) " 9 6 . 1 2 6 . 1 2 7 . 1 0 8 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 0 : 5 1 - 0 4 0 0 ] " P O S T / w p - c r o n . p h p ? d o i n g _ w p _ c r o n = 1 3 6 8 3 2 1 0 5 1 . 2 9 8 0 6 4 9 4 7 1 2 8 2 9 5 8 9 8 4 3 7 5 H T T P / 1 . 0 " 2 0 0 0 " - " " W o r d P r e s s / 3 . 5 . 1 ; h t t p : / / w w w . s t u m p d i n p d x . c o m " 9 4 . 1 2 5 . 1 8 0 . 9 0 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 0 : 5 1 - 0 4 0 0 ] " P O S T / w p - l o g i n . p h p H T T P / 1 . 0 " 2 0 0 4 4 5 2 " - " " M o z i l l a / 3 . 0 ( c o m p a t i b l e ; I n d y L i b r a r y ) " 2 1 7 . 3 4 . 1 8 1 . 7 6 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 0 : 5 1 - 0 4 0 0 ] " P O S T / w p - l o g i n . p h p H T T P / 1 . 0 " 2 0 0 4 4 5 2 " - " " M o z i l l a / 3 . 0 ( c o m p a t i b l e ; I n d y L i b r a r y ) " 9 6 . 1 2 6 . 1 2 7 . 1 0 8 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 2 : 0 9 - 0 4 0 0 ] " P O S T / w p - c r o n . p h p ? d o i n g _ w p _ c r o n = 1 3 6 8 3 2 1 1 2 9 . 5 5 0 1 3 6 0 8 9 3 2 4 9 5 1 1 7 1 8 7 5 0 H T T P / 1 . 0 " 2 0 0 0 " - " " W o r d P r e s s / 3 . 5 . 1 ; h t t p : / / w w w . s t u m p d i n p d x . c o m " 1 9 0 . 1 9 9 . 6 0 . 1 5 0 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 2 : 0 9 - 0 4 0 0 ] " P O S T / w p - l o g i n . p h p H T T P / 1 . 0 " 2 0 0 4 4 6 3 " h t t p : / / w w w . s t u m p d i n p d x . c o m / w p - l o g i n . p h p " " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 6 . 0 ; W i n d o w s N T 5 . 1 ; S V 1 ) " 1 8 4 . 1 5 4 . 1 0 0 . 2 0 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 2 : 5 6 - 0 4 0 0 ] " G E T / 2 0 1 2 / 1 2 / 5 0 - t h i n g s - i - w i l l - m i s s - a b o u t - p o r t l a n d / c o m m e n t - p a g e - 1 / H T T P / 1 . 0 " 2 0 0 1 2 6 9 9 " h t t p : / / w w w . s t u m p d i n p d x . c o m / " " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 8 . 0 ; W i n d o w s N T 6 . 0 ; T r i d e n t / 4 . 0 ; M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 6 . 0 ; W i n d o w s N T 5 . 1 ; S V 1 ) ; . N E T C L R 3 . 5 . 3 0 7 2 9 ) " 9 6 . 1 2 6 . 1 2 7 . 1 0 8 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 3 : 2 9 - 0 4 0 0 ] " P O S T / w p - c r o n . p h p ? d o i n g _ w p _ c r o n = 1 3 6 8 3 2 1 2 0 9 . 4 3 7 7 1 4 0 9 9 8 8 4 0 3 3 2 0 3 1 2 5 0 H T T P / 1 . 0 " 2 0 0 0 " - " " W o r d P r e s s / 3 . 5 . 1 ; h t t p : / / w w w . s t u m p d i n p d x . c o m " 2 1 7 . 9 1 . 3 7 . 3 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 3 : 2 9 - 0 4 0 0 ] " P O S T / w p - l o g i n . p h p H T T P / 1 . 0 " 2 0 0 4 4 5 2 " - " " M o z i l l a / 3 . 0 ( c o m p a t i b l e ; I n d y L i b r a r y ) " 8 0 . 9 3 . 2 1 3 . 2 4 9 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 5 : 3 2 - 0 4 0 0 ] " G E T / 2 0 1 0 / 0 5 / f o o d - c a r t s - o f - m e l b o u r n e - a l l - f o u r - o f - t h e m / H T T P / 1 . 1 " 2 0 0 1 6 5 6 9 " h t t p : / / w w w . s t u m p d i n p d x . c o m / " " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 6 . 0 ; W i n d o w s N T 5 . 1 ; S V 1 ; F u n W e b P r o d u c t s ; . N E T C L R 1 . 1 . 4 3 2 2 ; P e o p l e P a l 6 . 2 ) " 8 0 . 9 3 . 2 1 3 . 2 4 9 - - [ 1 1 / M a y / 2 0 1 3 : 2 1 : 1 5 : 3 3 - 0 4 0 0 ] " G E T / 2 0 1 2 / 1 2 / 5 0 - t h i n g s - i - w i l l - m i s s - a b o u t - p o r t l a n d / c o m m e n t - p a g e - 1 / H T T P / 1 . 1 " 2 0 0 1 2 7 2 0 " h t t p : / / w w w . s t u m p d i n p d x . c o m / " " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 6 . 0 ; W i n d o w s N T 5 . 1 ; S V 1 ; F u n W e b P r o d u c t s ; . N E T C L R 1 . 1 . 4 3 2 2 ; P e o p l e P a l 6 . 2 ) "
16. [ 1 1 - M a y - 2 0 1 3 1 4 : 1 0 : 0 4 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 1 1 : 3 2 U T C ] P H P F a t a l e r r o r : C a l l t o a m e m b e r f u n c t i o n s e t t i n g ( ) o n a n o n - o b j e c t i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / f e e d w o r d p r e s s . p h p o n l i n e 6 0 6 [ 1 1 - M a y - 2 0 1 3 1 5 : 2 1 : 5 8 U T C ] P H P F a t a l e r r o r : C a l l t o a m e m b e r f u n c t i o n s e t t i n g ( ) o n a n o n - o b j e c t i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / f e e d w o r d p r e s s . p h p o n l i n e 6 0 6 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 5 : 5 0 : 0 3 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 7 : 1 0 : 0 7 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1 [ 1 1 - M a y - 2 0 1 3 1 7 : 1 0 : 0 7 U T C ] P H P W a r n i n g : I n v a l i d a r g u m e n t s u p p l i e d f o r f o r e a c h ( ) i n / v a r / w w w / h t m l / p l a n e t d e v o p s / w p - c o n t e n t / p l u g i n s / f e e d w o r d p r e s s / m a g p i e f r o m s i m p l e p i e . c l a s s . p h p o n l i n e 5 3 1
17. J u n 4 , 2 0 1 1 1 0 : 0 1 : 0 6 A M o r g . a p a c h e . c o y o t e . h t t p 1 1 . H t t p 1 1 P r o t o c o l i n i t I N F O : I n i t i a l i z i n g C o y o t e H T T P / 1 . 1 o n h t t p - 8 0 8 0 J u n 4 , 2 0 1 1 1 0 : 2 4 : 4 8 A M o r g . a p a c h e . c a t a l i n a . l o a d e r . W e b a p p C l a s s L o a d e r c l e a r T h r e a d L o c a l M a p S E V E R E : T h e w e b a p p l i c a t i o n [ ] c r e a t e d a T h r e a d L o c a l w i t h k e y o f t y p e [ n u l l ] ( v a l u e [ c l o j u r e . l a n g . V a r $ 1 @ 5 6 4 c a 9 3 0 ] ) a n d a v a l u e o f t y p e [ c l o j u r e . l a n g . V a r . F r a m e ] ( v a l u e [ c l o j u r e . l a n g . V a r $ F r a m e @ 4 2 f 7 b a 9 3 ] ) b u t f a i l e d t o r e m o v e i t w h e n t h e w e b a p p l i c a t i o n w a s s t o p p e d . T h i s i s v e r y l i k e l y t o c r e a t e a m e m o r y l e a k . J u n 4 , 2 0 1 1 1 0 : 2 4 : 4 8 A M o r g . a p a c h e . c a t a l i n a . l o a d e r . W e b a p p C l a s s L o a d e r c l e a r T h r e a d L o c a l M a p S E V E R E : T h e w e b a p p l i c a t i o n [ ] c r e a t e d a T h r e a d L o c a l w i t h k e y o f t y p e [ j a v a . l a n g . T h r e a d L o c a l ] ( v a l u e [ j a v a . l a n g . T h r e a d L o c a l @ 1 5 f a 2 b 3 e ] ) a n d a v a l u e o f t y p e [ c l o j u r e . l a n g . L o c k i n g T r a n s a c t i o n ] ( v a l u e [ c l o j u r e . l a n g . L o c k i n g T r a n s a c t i o n @ 5 b 2 c f e b 7 ] ) b u t f a i l e d t o r e m o v e i t w h e n t h e w e b a p p l i c a t i o n w a s s t o p p e d . T h i s i s v e r y l i k e l y t o c r e a t e a m e m o r y l e a k . J u n 4 , 2 0 1 1 1 0 : 2 4 : 5 0 A M o r g . a p a c h e . c a t a l i n a . c o r e . S t a n d a r d C o n t e x t r e s o u r c e s S t a r t S E V E R E : E r r o r s t a r t i n g s t a t i c R e s o u r c e s j a v a . l a n g . I l l e g a l A r g u m e n t E x c e p t i o n : D o c u m e n t b a s e / v a r / l i b / t o m c a t 6 / w e b a p p s / R O O T d o e s n o t e x i s t o r i s n o t a r e a d a b l e d i r e c t o r y a t o r g . a p a c h e . n a m i n g . r e s o u r c e s . F i l e D i r C o n t e x t . s e t D o c B a s e ( F i l e D i r C o n t e x t . j a v a : 1 4 2 ) a t o r g . a p a c h e . c a t a l i n a . c o r e . S t a n d a r d C o n t e x t . r e s o u r c e s S t a r t ( S t a n d a r d C o n t e x t . j a v a : 4 2 4 9 ) a t o r g . a p a c h e . c a t a l i n a . c o r e . S t a n d a r d C o n t e x t . s t a r t ( S t a n d a r d C o n t e x t . j a v a : 4 4 1 8 ) a t o r g . a p a c h e . c a t a l i n a . s t a r t u p . H o s t C o n f i g . c h e c k R e s o u r c e s ( H o s t C o n f i g . j a v a : 1 2 4 4 ) a t o r g . a p a c h e . c a t a l i n a . s t a r t u p . H o s t C o n f i g . c h e c k ( H o s t C o n f i g . j a v a : 1 3 4 2 ) a t o r g . a p a c h e . c a t a l i n a . s t a r t u p . H o s t C o n f i g . l i f e c y c l e E v e n t ( H o s t C o n f i g . j a v a : 3 0 3 ) a t o r g . a p a c h e . c a t a l i n a . u t i l . L i f e c y c l e S u p p o r t . f i r e L i f e c y c l e E v e n t ( L i f e c y c l e S u p p o r t . j a v a : 1 1 9 ) a t o r g . a p a c h e . c a t a l i n a . c o r e . C o n t a i n e r B a s e . b a c k g r o u n d P r o c e s s ( C o n t a i n e r B a s e . j a v a : 1 3 3 7 ) a t o r g . a p a c h e . c a t a l i n a . c o r e . C o n t a i n e r B a s e $ C o n t a i n e r B a c k g r o u n d P r o c e s s o r . p r o c e s s C h i l d r e n ( C o n t a i n e r B a s e . j a v a : 1 6 0 1 ) a t o r g . a p a c h e . c a t a l i n a . c o r e . C o n t a i n e r B a s e $ C o n t a i n e r B a c k g r o u n d P r o c e s s o r . p r o c e s s C h i l d r e n ( C o n t a i n e r B a s e . j a v a : 1 6 1 0 ) a t o r g . a p a c h e . c a t a l i n a . c o r e . C o n t a i n e r B a s e $ C o n t a i n e r B a c k g r o u n d P r o c e s s o r . r u n ( C o n t a i n e r B a s e . j a v a : 1 5 9 0 ) a t j a v a . l a n g . T h r e a d . r u n ( T h r e a d . j a v a : 6 6 2 ) J u n 4 , 2 0 1 1 1 0 : 2 4 : 5 0 A M o r g . a p a c h e . c a t a l i n a . c o r e . S t a n d a r d C o n t e x t s t a r t S E V E R E : E r r o r i n r e s o u r c e S t a r t ( ) J u n 4 , 2 0 1 1 1 0 : 2 4 : 5 0 A M o r g . a p a c h e . c a t a l i n a . c o r e . S t a n d a r d C o n t e x t s t a r t S E V E R E : E r r o r g e t C o n f i g u r e d
18. all of these logs tell us (useful) stories
19. pretty confusing stories though eh?
20. so what's wrong? so many sodding formats don't even get me started on timestamps no context really unhelpful error messages doesn't scale
21. enter logstash, parsing heavily
22. what? collects, transmits, interprets, stores free and open source primarily written by Jordan Sissel maxim: if a new user has a bad time, it's a bug in logstash awesome!
23. logstash architecture
24. how does it work? 2 0 2 . 4 6 . 5 2 . 2 0 - - [ 2 1 / J a n / 2 0 1 3 : 1 4 : 5 9 : 3 9 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 1 " - " " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 7 . 0 ; W i n d o w s N T 6 . 0 ) " 1 1 9 . 6 3 . 1 9 3 . 1 9 6 - - [ 2 1 / J a n / 2 0 1 3 : 1 5 : 0 0 : 2 7 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 1 " - " " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 7 . 0 ; W i n d o w s N T 6 . 0 ) " 2 0 8 . 1 1 5 . 1 1 3 . 8 8 - - [ 2 1 / J a n / 2 0 1 3 : 1 5 : 0 4 : 3 0 - 0 8 0 0 ] " G E T / r o b o t s . t x t H T T P / 1 . 1 " 4 0 4 2 9 7 " - " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; E z o o m s / 1 . 0 ; e z o o m s . b o t @ g m a i l . c o m ) " 1 8 8 . 1 3 8 . 8 8 . 1 7 1 - - [ 2 1 / J a n / 2 0 1 3 : 1 5 : 0 9 : 4 6 - 0 8 0 0 ] " G E T / w 0 0 t w 0 0 t . a t . I S C . S A N S . D F i n d : ) H T T P / 1 . 1 " 4 0 0 3 1 5 " - " " - " 2 2 0 . 1 8 1 . 1 0 8 . 8 1 - - [ 2 1 / J a n / 2 0 1 3 : 1 5 : 2 1 : 3 4 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 5 " - " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; B a i d u s p i d e r / 2 . 0 ; + h t t p : / / w w w . b a i d u . c o m / s e a r c h / s p i d e r . h t m l ) " 1 2 3 . 1 2 5 . 7 1 . 3 1 - - [ 2 1 / J a n / 2 0 1 3 : 1 5 : 2 1 : 5 8 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 5 " - " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; B a i d u s p i d e r / 2 . 0 ; + h t t p : / / w w w . b a i d u . c o m / s e a r c h / s p i d e r . h t m l ) " 1 2 3 . 1 5 1 . 1 4 8 . 1 6 2 - - [ 2 1 / J a n / 2 0 1 3 : 1 5 : 3 7 : 1 1 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 1 " - " " S o s o s p i d e r + ( + h t t p : / / h e l p . s o s o . c o m / w e b s p i d e r . h t m ) " 1 1 9 . 6 3 . 1 9 6 . 2 8 - - [ 2 1 / J a n / 2 0 1 3 : 1 5 : 4 1 : 2 8 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 0 " - " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; B a i d u s p i d e r / 2 . 0 ; + h t t p : / / w w w . b a i d u . c o m / s e a r c h / s p i d e r . h t m l ) " 2 0 9 . 8 5 . 2 3 8 . 1 7 4 - - [ 2 1 / J a n / 2 0 1 3 : 1 5 : 4 5 : 2 0 - 0 8 0 0 ] " G E T / ? t y p e = a t o m 1 0 H T T P / 1 . 1 " 2 0 0 9 3 0 " - " " F e e d f e t c h e r - G o o g l e ; ( + h t t p : / / w w w . g o o g l e . c o m / f e e d f e t c h e r . h t m l ; 2 s u b s c r i b e r s ; f e e d - i d = 1 6 1 5 7 8 5 6 2 5 7 6 0 1 6 2 9 8 2 2 ) " 1 8 8 . 1 3 8 . 8 8 . 1 7 1 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 1 7 : 0 6 - 0 8 0 0 ] " G E T / w 0 0 t w 0 0 t . a t . I S C . S A N S . D F i n d : ) H T T P / 1 . 1 " 4 0 0 3 1 5 " - " " - " 1 2 3 . 1 2 5 . 7 1 . 3 5 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 1 9 : 2 2 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 2 7 " - " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; B a i d u s p i d e r / 2 . 0 ; + h t t p : / / w w w . b a i d u . c o m / s e a r c h / s p i d e r . h t m l ) " 2 2 0 . 1 8 1 . 1 0 8 . 7 8 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 1 9 : 2 9 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 2 7 " - " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; B a i d u s p i d e r / 2 . 0 ; + h t t p : / / w w w . b a i d u . c o m / s e a r c h / s p i d e r . h t m l ) " 1 8 0 . 7 6 . 5 . 5 5 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 2 0 : 1 4 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 0 " - " " M o z i l l a / 5 . 0 ( c o m p a t i b l e ; B a i d u s p i d e r / 2 . 0 ; + h t t p : / / w w w . b a i d u . c o m / s e a r c h / s p i d e r . h t m l ) "
25. simple is as simple does i n p u t { f i l e { t y p e = > " w e b " p a t h = > " / v a r / l o g / h t t p d / a c c e s s . l o g " } } f i l t e r { g r o k { t y p e = > " w e b " p a t t e r n = > " % { C O M B I N E D A P A C H E L O G } " } d a t e { t y p e = > " w e b " t i m e s t a m p = > " d d / M M M / y y y y : H H : m m : s s Z " } } o u t p u t { e l a s t i c s e a r c h { } }
26. the input i n p u t { f i l e { t y p e = > " w e b " p a t h = > " / v a r / l o g / h t t p d / a c c e s s . l o g " } }
27. turns 2 0 2 . 4 6 . 6 3 . 1 9 2 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 4 1 : 3 8 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 5 " - " " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 7 . 0 ; W i n d o w s N T 6 . 0 ) "
28. into { " @ s o u r c e " = > " f i l e : / / p e l i n . e x a m p l e . c o m / v a r / h t t p d / a c c e s s . l o g " , " @ t a g s " = > [ ] , " @ f i e l d s " = > { } , " @ t i m e s t a m p " = > " 2 0 1 3 - 0 1 - 2 1 T 1 6 : 4 1 : 3 8 . 0 3 0 Z " , " @ s o u r c e _ h o s t " = > " p e l i n . e x a m p l e . c o m " , " @ s o u r c e _ p a t h " = > " / v a r / l o g / h t t p d / a c c e s s . l o g " , " @ m e s s a g e " = > " 2 0 2 . 4 6 . 6 3 . 1 9 2 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 4 1 : 3 8 - 0 8 0 0 ] G E T / H T T P / 1 . 1 2 0 0 9 3 5 - M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 7 . 0 ; W i n d o w s N T 6 . 0 ) " , " @ t y p e " = > " w e b " }
29. still looks like a mess eh? but it's now a structured mess!
30. structured data for the win!
31. the filters g r o k { t y p e = > " w e b " p a t t e r n = > " % { C O M B I N E D A P A C H E L O G } " }
32. use the power of regex
33. to add context
34. instead of ... evil ... like: ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : ( ? : ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | " ( ? : [ ^ \ " \ r \ \ ] | \ \ . | ( ? : ( ? : \ r \ n ) ? [ \ t ] ) ) * " ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ( ? : \ . ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | " ( ? : [ ^ \ " \ r \ \ ] | \ \ . | ( ? : ( ? : \ r \ n ) ? [ \ t ] ) ) * " ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ) * @ ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | \ [ ( [ ^ \ [ \ ] \ r \ \ ] | \ \ . ) * \ ] ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ( ? : \ . ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | \ [ ( [ ^ \ [ \ ] \ r \ \ ] | \ \ . ) * \ ] ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ) * | ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | " ( ? : [ ^ \ " \ r \ \ ] | \ \ . | ( ? : ( ? : \ r \ n ) ? [ \ t ] ) ) * " ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) * \ < ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : @ ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | \ [ ( [ ^ \ [ \ ] \ r \ \ ] | \ \ . ) * \ ] ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ( ? : \ . ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | \ [ ( [ ^ \ [ \ ] \ r \ \ ] | \ \ . ) * \ ] ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ) * ( ? : , @ ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | \ [ ( [ ^ \ [ \ ] \ r \ \ ] | \ \ . ) * \ ] ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ( ? : \ . ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | \ [ ( [ ^ \ [ \ ] \ r \ \ ] | \ \ . ) * \ ] ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ) * ) * : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ? ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | " ( ? : [ ^ \ " \ r \ \ ] | \ \ . | ( ? : ( ? : \ r \ n ) ? [ \ t ] ) ) * " ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ( ? : \ . ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | " ( ? : [ ^ \ " \ r \ \ ] | \ \ . | ( ? : ( ? : \ r \ n ) ? [ \ t ] ) ) * " ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ) * @ ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | \ [ ( [ ^ \ [ \ ] \ r \ \ ] | \ \ . ) * \ ] ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ( ? : \ . ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | \ [ ( [ ^ \ [ \ ] \ r \ \ ] | \ \ . ) * \ ] ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ) * \ > ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) | ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | " ( ? : [ ^ \ " \ r \ \ ] | \ \ . | ( ? : ( ? : \ r \ n ) ? [ \ t ] ) ) * " ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) * : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : ( ? : ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | " ( ? : [ ^ \ " \ r \ \ ] | \ \ . | ( ? : ( ? : \ r \ n ) ? [ \ t ] ) ) * " ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ( ? : \ . ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ( ? : [ ^ ( ) < > @ , ; : \ \ " . \ [ \ ] \ 0 0 0 - \ 0 3 1 ] + ( ? : ( ? : ( ? : \ r \ n ) ? [ \ t ] ) + | \ Z | ( ? = [ \ [ " ( ) < > @ , ; : \ \ " . \ [ \ ] ] ) ) | " ( ? : [ ^ \ " \ r \ \ ] | \ \ . | ( ? : ( ? : \ r \ n ) ? [ \ t ] ) ) * " ( ? : ( ? : \ r \ n ) ? [ \ t ] ) * ) ) * @ ( ? : ( ? : \ r \ n ) ? [ \ t ]
35. %{SYNTAX:SEMANTIC} L o g : M a y 1 2 0 3 : 3 6 : 3 1 p e l i n d h c l i e n t [ 2 3 3 5 ] : D H C P A C K f r o m 9 7 . 1 0 7 . 1 4 3 . 3 8 ( x i d = 0 x 6 f 6 2 5 7 2 d ) G r o k : % { S Y S L O G T I M E S T A M P : t i m e s t a m p } % { H O S T N A M E : h o s t } % { S Y S L O G P R O G : p r o g r a m } : % { D A T A : m e s s a g e } S Y S L O G T I M E S T A M P : % { M O N T H } + % { M O N T H D A Y } % { T I M E } H O S T N A M E : \ b ( ? : [ 0 - 9 A - Z a - z ] [ 0 - 9 A - Z a - z - ] { 0 , 6 2 } ) ( ? : \ . ( ? : [ 0 - 9 A - Z a - z ] [ 0 - 9 A - Z a - z - ] { 0 , 6 2 } ) ) * ( \ . ? | \ b ) S Y S L O G P R O G % { P R O G : p r o g r a m } ( ? : \ [ % { P O S I N T : p i d } \ ] ) ?
36. remember this? { " @ s o u r c e " = > " f i l e : / / p e l i n . e x a m p l e . c o m / v a r / h t t p d / a c c e s s . l o g " , " @ t a g s " = > [ ] , " @ f i e l d s " = > { } , " @ t i m e s t a m p " = > " 2 0 1 3 - 0 1 - 2 1 T 1 6 : 4 1 : 3 8 . 0 3 0 Z " , " @ s o u r c e _ h o s t " = > " p e l i n . e x a m p l e . c o m " , " @ s o u r c e _ p a t h " = > " / v a r / l o g / h t t p d / a c c e s s . l o g " , " @ m e s s a g e " = > " 2 0 2 . 4 6 . 6 3 . 1 9 2 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 4 1 : 3 8 - 0 8 0 0 ] G E T / H T T P / 1 . 1 2 0 0 9 3 5 - M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 7 . 0 ; W i n d o w s N T 6 . 0 ) " , " @ t y p e " = > " w e b " }
37. with grok it becomes { " @ s o u r c e " = > " f i l e : / / p e l i n . e x a m p l e . c o m / v a r / h t t p d / a c c e s s . l o g " , " @ t a g s " = > [ ] , " @ f i e l d s " = > { " c l i e n t i p " : [ " 2 0 2 . 4 6 . 6 3 . 1 9 2 " ] , " i d e n t " : [ " - " ] , " a u t h " : [ " - " ] , " t i m e s t a m p " : [ " 2 1 / J a n / 2 0 1 3 : 1 6 : 4 1 : 3 8 - 0 8 0 0 " ] , " v e r b " : [ " G E T " ] , " r e q u e s t " : [ " / " ] , " h t t p v e r s i o n " : [ " 1 . 1 " ] , " r e s p o n s e " : [ " 2 0 0 " ] , " b y t e s " : [ " 9 3 5 " ] , " r e f e r r e r " : [ " \ " - \ " " ] , " a g e n t " : [ " \ " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 7 . 0 ; W i n d o w s N T 6 . 0 ) \ " " ] } , " @ t i m e s t a m p " = > " 2 0 1 3 - 0 1 - 2 1 T 1 6 : 4 1 : 3 8 . 0 3 0 Z " , " @ s o u r c e _ h o s t " = > " p e l i n . e x a m p l e . c o m " , " @ s o u r c e _ p a t h " = > " / v a r / l o g / h t t p d / a c c e s s . l o g " , " @ m e s s a g e " = > " 2 0 2 . 4 6 . 6 3 . 1 9 2 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 4 1 : 3 8 - 0 8 0 0 ] G E T / H T T P / 1 . 1 2 0 0 9 3 5 - M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 7 . 0 ; W i n d o w s N T 6 . 0 ) " , " @ t y p e " = > " w e b " }
38. grok makes better over 100 patterns numbers, strings, hosts, network addresses, urls, etc chain patterns together easy to extend, easy to test
39. you can test your patterns http://grokdebug.herokuapp.com/
40. or you can even write tests for your patterns you write tests right?
41. did I mention time? d a t e { t y p e = > " w e b " t i m e s t a m p = > " d d / M M M / y y y y : H H : m m : s s Z " } }
42. problem? so many fucking time formats seriously. stop adding time formats.
43. solution. standardize with the time filter.
44. filters rock 30+ filters munge, mangle, mutate lookup, research, aggregate
45. filters turn abstract information like 2 0 2 . 4 6 . 6 3 . 1 9 2 - - [ 2 1 / J a n / 2 0 1 3 : 1 6 : 4 1 : 3 8 - 0 8 0 0 ] " G E T / H T T P / 1 . 1 " 2 0 0 9 3 5 " - " " M o z i l l a / 4 . 0 ( c o m p a t i b l e ; M S I E 7 . 0 ; W i n d o w s N T 6 . 0 ) "
46. into
47. the truth will set you free ... or at least wake you up.
48. outputs o u t p u t { e l a s t i c s e a r c h { } }
49. outputs 50+ outputs search, store, transit email, irc, alert graph, aggregate, execute
50. all of the pretty things
51. all of the pretty things
52. scales like a mofo
53. all of the logstashes logstash-users@googlegroups.com #logstash on freenode irc logstash.net logstash.jira.com
54. Questions?
55. references Doctor Who © BBC He-Man © Mattel