Laisse pas trainer ton log !

Given at #forumphp Paris 2014.
Slides are in english, talk was given in french.

Joind.in : https://joind.in/talk/view/11942
Vidéo (in french): https://www.youtube.com/watch?v=1r1SOeaDqH4&list=PL9zDdgiGjkIeeVlrsz9A8o3HtZhvERHT-&index=7

Transcript

1. LAISSE PAS TRAINER TON LOG @odolbeau 1
2. WHO AM I? Olivier Dolbeau @odolbeau Work at BlaBlaCar 2
4. Logfile 4
5. –Wikipedia “In computing, a logfile (or simply log) is a file that records either the events which happen while an operating system or other software runs, […].” 5
6. –Wikipedia “In computing, a logfile (or simply log) is a file that records either the events which happen while an operating system or other software runs, […].” 6
7. Which logs are we
 talking about? 7
8. access logs 8
9. syslog syslog 9
10. application logs 10
11. Access 11
12. SSH 12
13. Analyze 13
14. tail grep cat 14
16. My roommate uses this to colorise his access logs… 15
17. This is specific to its access logs My roommate uses this to colorise his access logs… 15
23. Inputs Filters Outputs 41 inputs • syslog • udp • varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … 21
24. Inputs Filters Outputs 41 inputs • syslog • udp • varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … And there are also some codecs 21
25. Kibana 22
27. ELK 24
29. syslog syslog 26
31. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat 28
32. input { udp { port => 514 type => syslog } } Logstash - Input 29
33. filter { if [type] == "syslog" { grok { match => [ "message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: % {GREEDYDATA:syslog_message}" ] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] add_tag => [ "rsyslog" ] } } } Logstash - Filter 30
34. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200 index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 31
35. application logs 32
37. • StreamHandler • ErrorLogHandler • SwiftMailerHandler • SyslogUdpHandler • FirePHPHandler • FingersCrossedHandler • NullHandler • … More than 36 handlers! It’s just some outputs! 34
38. http://symfony.com/doc/current/cookbook/logging/channels_handlers.html 35
42. WHAT CAN I DO WITH THAT 39
43. I want to have all my logs displayed on the console. 40
45. handlers 41
46. handlers channels 41
47. I want to add web context informations to my logs. 42
49. I want to see all my logs in a PRETTY interface! 44
51. input { gelf { port => 12201 type => gelf } } Logstash - Input 46
52. Logstash - Filter 47
53. Logstash - Filter 47 This space has intentionally been left blank.
54. Logstash - Filter 47 This space has intentionally been left blank. We don’t need any filter
55. Logstash - Filter 47 This space has intentionally been left blank. We don’t need any filter Because logstash works well!
56. Logstash - Filter 47 This space has intentionally been left blank. We don’t need any filter Because logstash works well! With Heka you need to write a lot of Lua
57. This troll was dedicated to @lyrixx 48
58. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200 index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 49
59. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200 index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 49 It’s a duplicate slide!
60. syslog 50
63. Channels! \o/ 53
65. DEMO 55
66. “C’est pas beautiful ça ?” 56
67. @odolbeau https://joind.in/11942 57 On recrute !