Pro Yearly is on sale from $80 to $50! »

Laisse pas trainer ton log !

Laisse pas trainer ton log !

Given at #forumphp Paris 2014.
Slides are in english, talk was given in french.

Joind.in : https://joind.in/talk/view/11942
Vidéo (in french): https://www.youtube.com/watch?v=1r1SOeaDqH4&list=PL9zDdgiGjkIeeVlrsz9A8o3HtZhvERHT-&index=7

418997665c4a3368515ecf9c3d746b95?s=128

Olivier Dolbeau

October 23, 2014
Tweet

Transcript

  1. LAISSE PAS TRAINER TON LOG @odolbeau 1

  2. WHO AM I? Olivier Dolbeau @odolbeau Work at BlaBlaCar 2

  3. 3

  4. Logfile 4

  5. –Wikipedia “In computing, a logfile (or simply log) is a

    file that records either the events which happen while an operating system or other software runs, […].” 5
  6. –Wikipedia “In computing, a logfile (or simply log) is a

    file that records either the events which happen while an operating system or other software runs, […].” 6
  7. Which logs are we
 talking about? 7

  8. access logs 8

  9. syslog syslog 9

  10. application logs 10

  11. Access 11

  12. SSH 12

  13. Analyze 13

  14. tail grep cat 14

  15. 15

  16. My roommate uses this to colorise his access logs… 15

  17. This is specific to its access logs My roommate uses

    this to colorise his access logs… 15
  18. 16

  19. 17

  20. 18

  21. 19

  22. 20

  23. Inputs Filters Outputs 41 inputs • syslog • udp •

    varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … 21
  24. Inputs Filters Outputs 41 inputs • syslog • udp •

    varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … And there are also some codecs 21
  25. Kibana 22

  26. 23

  27. ELK 24

  28. 25

  29. syslog syslog 26

  30. 27

  31. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat 28

  32. input { udp { port => 514 type => syslog

    } } Logstash - Input 29
  33. filter { if [type] == "syslog" { grok { match

    => [ "message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: % {GREEDYDATA:syslog_message}" ] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] add_tag => [ "rsyslog" ] } } } Logstash - Filter 30
  34. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200

    index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 31
  35. application logs 32

  36. 33

  37. • StreamHandler • ErrorLogHandler • SwiftMailerHandler • SyslogUdpHandler • FirePHPHandler

    • FingersCrossedHandler • NullHandler • … More than 36 handlers! It’s just some outputs! 34
  38. http://symfony.com/doc/current/cookbook/logging/channels_handlers.html 35

  39. 36

  40. 37

  41. 38

  42. WHAT CAN I DO WITH THAT 39

  43. I want to have all my logs displayed on the

    console. 40
  44. 41

  45. handlers 41

  46. handlers channels 41

  47. I want to add web context informations to my logs.

    42
  48. 43

  49. I want to see all my logs in a PRETTY

    interface! 44
  50. 45

  51. input { gelf { port => 12201 type => gelf

    } } Logstash - Input 46
  52. Logstash - Filter 47

  53. Logstash - Filter 47 This space has intentionally been left

    blank.
  54. Logstash - Filter 47 This space has intentionally been left

    blank. We don’t need any filter
  55. Logstash - Filter 47 This space has intentionally been left

    blank. We don’t need any filter Because logstash works well!
  56. Logstash - Filter 47 This space has intentionally been left

    blank. We don’t need any filter Because logstash works well! With Heka you need to write a lot of Lua
  57. This troll was dedicated to @lyrixx 48

  58. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200

    index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 49
  59. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200

    index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 49 It’s a duplicate slide!
  60. syslog 50

  61. 51

  62. 52

  63. Channels! \o/ 53

  64. 54

  65. DEMO 55

  66. “C’est pas beautiful ça ?” 56

  67. @odolbeau https://joind.in/11942 57 On recrute !