La Suite ELK au service des développeurs et de la production

Comment la Suite ELK facilite la centralisation et la corrélation d'indicateurs hétérogènes, pour une meilleure prise de décision.

Transcript

1. Suite ELK
   au service des développeurs et de la production
   

   View Slide

2. observer, mesurer, améliorer
   Icons by Alexander Skowalsky, Shawn Schmidt and Andreas Bjurenborg from the Noun Project
   

   View Slide

3. logiciels libres
   au service de l’entreprise
   

   View Slide

4. metrics
   

   View Slide

5. metrics
   métier
   

   View Slide

6. metrics
   • utilisateurs connectés
   • nombre de commandes
   • panier moyen
   • état du stock
   métier
   

   View Slide

7. metrics
   • utilisateurs connectés
   • nombre de commandes
   • panier moyen
   • état du stock
   métier système
   

   View Slide

8. metrics
   • utilisateurs connectés
   • nombre de commandes
   • panier moyen
   • état du stock
   • temps de chargement
   • taux d’échec
   • réponse de la BDD
   • disponibilité des services
   métier système
   

   View Slide

9. sources hétérogènes
   

   View Slide

10. sources hétérogènes
    

    View Slide

11. sources hétérogènes
    

    View Slide

12. sources hétérogènes
    

    View Slide

13. sources hétérogènes
    

    View Slide

14. sources hétérogènes
    

    View Slide

15. étude de cas
    

    View Slide

16. xeonys-fr-01
    HAProxy
    Apache/FPM A1
    Apache/FPM A2
    BDD-A Queues
    Varnish
    Apache/FPM B1
    Apache/FPM B2
    BDD-B Queues
    Apache/FPM B3
    xeonys-uk
    HAProxy
    Varnish
    Apache/FPM 1
    Apache/FPM 2
    BDD Queues
    xeonys-de
    HAProxy
    Varnish
    Apache/FPM 1
    Apache/FPM 2
    BDD Queues
    Apache/FPM 3
    

    View Slide

17. Icon by Chameleon Design from the Noun Project
    

    View Slide

18. Icon by Chameleon Design from the Noun Project
    

    View Slide

19. Icon by Chameleon Design from the Noun Project
    indicateurs imprécis
    

    View Slide

20. Icon by Chameleon Design from the Noun Project
    indicateurs imprécis
    

    View Slide

21. Icon by Chameleon Design from the Noun Project
    indicateurs imprécis logs éparpillés
    

    View Slide

22. Icons by Chameleon Design, il Capitano, Shmidt Sergey and Kirby Wu from the Noun Project
    

    View Slide

23. Icons by Chameleon Design, il Capitano, Shmidt Sergey and Kirby Wu from the Noun Project
    

    View Slide

24. Icons by Chameleon Design, il Capitano, Shmidt Sergey and Kirby Wu from the Noun Project
    

    View Slide

25. Icons by Chameleon Design, il Capitano, Shmidt Sergey and Kirby Wu from the Noun Project
    

    View Slide

26. ELK
    Icons by Chameleon Design, il Capitano, Shmidt Sergey and Kirby Wu from the Noun Project
    

    View Slide

27. ELK
    

    View Slide

28. ELK
    Elasticsearch
    

    View Slide

29. ELK
    Elasticsearch • Logstash
    

    View Slide

30. ELK
    Elasticsearch • Logstash • Kibana
    

    View Slide

31. Logstash
    

    View Slide

32. Logstash
    Extract Transform Load
    

    View Slide

33. Logstash
    Extract Transform Load
    inputs:
    beats:
    port: 5044
    gelf:
    port: 12201
    use_udp: true
    

    View Slide

34. Logstash
    Extract Transform Load
    inputs:
    beats:
    port: 5044
    gelf:
    port: 12201
    use_udp: true
    filters:
    grok:
    pattern: httpd
    date:
    match: dd-MMM-YYYY HH:mm:ss
    mutate:
    convert: [line: integer]
    geoip:
    src: remote_ip
    target: geo
    

    View Slide

35. Logstash
    Extract Transform Load
    inputs:
    beats:
    port: 5044
    gelf:
    port: 12201
    use_udp: true
    filters:
    grok:
    pattern: httpd
    date:
    match: dd-MMM-YYYY HH:mm:ss
    mutate:
    convert: [line: integer]
    geoip:
    src: remote_ip
    target: geo
    outputs:
    elasticsearch:
    host: 127.0.0.1:9200
    index: logs-%{type}-%{+YYYY.MM.dd}
    stdout:
    codec: rubydebug
    

    View Slide

36. Elasticsearch
    la base de donnée au cœur de tout
    

    View Slide

37. Elasticsearch
    

    View Slide

38. Elasticsearch
    Performance
    

    View Slide

39. Elasticsearch
    Performance
    Interface REST/HTTP
    

    View Slide

40. Elasticsearch
    Performance
    Interface REST/HTTP
    Syntaxe JSON
    

    View Slide

41. Elasticsearch
    Performance
    Interface REST/HTTP
    Syntaxe JSON
    Cluster natif
    

    View Slide

42. Elasticsearch
    Performance
    Interface REST/HTTP
    Syntaxe JSON
    Cluster natif
    Simple et souple
    

    View Slide

43. Elasticsearch
    indexation de documents
    Icons by Chameleon Design and Shmidt Sergey from the Noun Project
    

    View Slide

44. Elasticsearch
    indexation de documents
    Icons by Chameleon Design and Shmidt Sergey from the Noun Project
    

    View Slide

45. Elasticsearch
    indexation de documents
    Icons by Chameleon Design and Shmidt Sergey from the Noun Project
    

    View Slide

46. index
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

47. logs-http
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

48. logs-http-{YMD}
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

49. logs-http-{YMD}
    2016.10.10
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

50. logs-http-{YMD}
    2016.10.11
    2016.10.10
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

51. logs-http-{YMD}
    2016.10.12
    2016.10.11
    2016.10.10
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

52. logs-http-{YMD}
    2016.10.13
    2016.10.12
    2016.10.11
    2016.10.10
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

53. logs-http-{YMD}
    2016.10.14
    2016.10.13
    2016.10.12
    2016.10.11
    2016.10.10
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

54. logs-http-{YMD}
    2016.10.14
    2016.10.13
    2016.10.12
    2016.10.11
    2016.10.10
    2015.08.02
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

55. logs-http-{YMD}
    2016.10.14
    2016.10.13
    2016.10.12
    2016.10.11
    2016.10.10
    2015.08.02
    ❌
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

56. logs-http-{YMD}
    2016.10.14
    2016.10.13
    2016.10.12
    2016.10.11
    2016.10.10
    2015.08.02
    recherche ciblée : j-2
    ❌
    {
    Icon by Shmidt Sergey from the Noun Project
    

    View Slide

57. administration facile
    

    View Slide

58. administration facile
    paquets à installer
    

    View Slide

59. administration facile
    paquets à installer
    très peu de config de départ
    

    View Slide

60. administration facile
    paquets à installer
    très peu de config de départ
    cluster trivial
    

    View Slide

61. administration facile
    paquets à installer
    très peu de config de départ
    cluster trivial
    sauvegardes par snapshot
    

    View Slide

62. documents
    

    View Slide

63. $ curl -XGET 'http://localhost:9200/_search' -d '{
    "query": {
    "bool" : {
    "must" : {
    "query_string" : {
    "query" : "some query string here"
    }
    },
    "filter" : {
    "term" : { "user" : "kimchy" }
    }
    }
    }
    }
    '
    documents
    

    View Slide

64. $ curl -XGET 'http://localhost:9200/_search' -d '{
    "query": {
    "bool" : {
    "must" : {
    "query_string" : {
    "query" : "some query string here"
    }
    },
    "filter" : {
    "term" : { "user" : "kimchy" }
    }
    }
    }
    }
    '
    documents {
    "_shards":{
    "total" : 5,
    "successful" : 5,
    "failed" : 0
    },
    "hits":{
    "total" : 1,
    "hits" : [
    {
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "1",
    "_source" : {
    "user" : "kimchy",
    "postDate" : "2009-11-15T14:12:12",
    "message" : "hello Elasticsearch"
    }
    }
    ]
    }
    }
    

    View Slide

65. agrégations
    

    View Slide

66. agrégations
    GET /cars/transactions/_search
    {
    "size" : 0,
    "aggs": {
    "colors": {
    "terms": {
    "field": "color"
    },
    "aggs": {
    "avg_price": {
    "avg": {
    "field": "price"
    }
    }
    }
    }
    }
    }
    

    View Slide

67. agrégations
    GET /cars/transactions/_search
    {
    "size" : 0,
    "aggs": {
    "colors": {
    "terms": {
    "field": "color"
    },
    "aggs": {
    "avg_price": {
    "avg": {
    "field": "price"
    }
    }
    }
    }
    }
    }
    {
    ...
    "aggregations": {
    "colors": {
    "buckets": [
    {
    "key": "red",
    "doc_count": 4,
    "avg_price": {
    "value": 32500
    }
    },
    {
    "key": "blue",
    "doc_count": 2,
    "avg_price": {
    "value": 20000
    }
    },
    {
    "key": "green",
    "doc_count": 2,
    "avg_price": {
    "value": 21000
    }
    }
    ]
    }
    }
    ...
    }
    

    View Slide

68. recherche à facettes
    

    View Slide

69. recherche à facettes
    

    View Slide

70. recherche à facettes
    

    View Slide

71. recherche à facettes
    

    View Slide

72. recherche à facettes
    

    View Slide

73. Kibana
    

    View Slide

74. Kibana
    Interface graphique web
    

    View Slide

75. Kibana
    Interface graphique web
    Basé sur les agrégations
    

    View Slide

76. Kibana
    Interface graphique web
    Basé sur les agrégations
    Visualisations
    

    View Slide

77. Kibana
    Interface graphique web
    Basé sur les agrégations
    Visualisations
    Fouille de données
    

    View Slide

78. Kibana
    

    View Slide

79. logs HTTP
    

    View Slide

80. logs HTTP
    

    View Slide

81. éléments de corrélation
    

    View Slide

82. éléments de corrélation
    HTTP
    

    View Slide

83. éléments de corrélation
    PHP
    HTTP
    

    View Slide

84. éléments de corrélation
    PHP
    HTTP Proxy
    

    View Slide

85. éléments de corrélation
    Queues
    PHP
    HTTP Proxy
    

    View Slide

86. éléments de corrélation
    Queues
    PHP
    HTTP Proxy
    date
    request
    method
    referrer
    user-agent
    host
    stack
    

    View Slide

87. éléments de corrélation
    Queues
    PHP
    HTTP Proxy
    date
    request
    method
    referrer
    user-agent
    host
    stack
    date
    file
    line
    class
    function
    message
    severity
    app_name
    app_env
    host
    stack
    

    View Slide

88. éléments de corrélation
    Queues
    PHP
    HTTP Proxy
    date
    request
    method
    referrer
    user-agent
    host
    stack
    date
    file
    line
    class
    function
    message
    severity
    app_name
    app_env
    host
    stack
    date
    request
    method
    frontend
    backend
    host
    stack
    

    View Slide

89. éléments de corrélation
    Queues
    PHP
    HTTP Proxy
    date
    request
    method
    referrer
    user-agent
    host
    stack
    date
    file
    line
    class
    function
    message
    severity
    app_name
    app_env
    host
    stack
    date
    request
    method
    frontend
    backend
    host
    stack
    date
    message
    severity
    app_name
    app_env
    host
    stack
    

    View Slide

90. index dédiés
    • logs-http-access
    • logs-http-error
    • logs-gelf
    • logs-redis
    • logs-pgsql
    • logs-fpm
    

    View Slide

91. collecte
    Icons by Chameleon Design from the Noun Project
    

    View Slide

92. collecte
    Icons by Chameleon Design from the Noun Project
    

    View Slide

93. collecte
    + lib/gelf.php
    Icons by Chameleon Design from the Noun Project
    

    View Slide

94. collecte
    inputs:
    beats:
    port: 5044
    gelf:
    port: 12201
    use_udp: true
    Logstash
    + lib/gelf.php
    Icons by Chameleon Design from the Noun Project
    

    View Slide

95. collecte
    inputs:
    beats:
    port: 5044
    gelf:
    port: 12201
    use_udp: true
    Logstash
    + lib/gelf.php
    Icons by Chameleon Design from the Noun Project
    

    View Slide

96. collecte
    inputs:
    beats:
    port: 5044
    gelf:
    port: 12201
    use_udp: true
    Logstash
    + lib/gelf.php
    prospectors:
    paths: /var/log/*.log
    input_type: log
    output:
    logstash:
    hosts: logstash:5044
    Filebeat
    Icons by Chameleon Design from the Noun Project
    

    View Slide

97. collecte
    inputs:
    beats:
    port: 5044
    gelf:
    port: 12201
    use_udp: true
    Logstash
    + lib/gelf.php
    prospectors:
    paths: /var/log/*.log
    input_type: log
    output:
    logstash:
    hosts: logstash:5044
    Filebeat
    Icons by Chameleon Design from the Noun Project
    

    View Slide

98. Beat
    

    View Slide

99. Beat
    framework Go
    

    View Slide

100. Beat
     framework Go
     Filebeat
     

     View Slide

101. Beat
     framework Go
     Filebeat
     Metricbeat
     

     View Slide

102. Beat
     framework Go
     Filebeat
     Metricbeat
     Packetbeat
     

     View Slide

103. et les données métier ?
     

     View Slide

104. 2 profils d’utilisation
     

     View Slide

105. 2 profils d’utilisation
     au moins
     

     View Slide

106. 2 profils d’utilisation
     au moins
     chercher des causes
     

     View Slide

107. 2 profils d’utilisation
     au moins
     chercher des causes
     valider des hypothèses
     

     View Slide

108. popularité croissante
     

     View Slide

109. popularité croissante
     Hébergement et Infogérance Open Source
     

     View Slide

110. adopter prudemment
     

     View Slide

111. adopter prudemment
     license
     

     View Slide

112. adopter prudemment
     license
     modèle économique
     

     View Slide

113. adopter prudemment
     license
     modèle économique
     gouvernance du projet
     

     View Slide

114. adopter prudemment
     license
     modèle économique
     gouvernance du projet
     place de la communauté
     

     View Slide

115. adopter prudemment
     license
     modèle économique
     gouvernance du projet
     place de la communauté
     ✔
     

     View Slide

116. open-source préservé
     

     View Slide

117. open-source préservé
     gros projets externes pérennisés
     

     View Slide

118. open-source préservé
     gros projets externes pérennisés
     fort engagement dans Lucene
     

     View Slide

119. modèle économique
     

     View Slide

120. modèle économique
     Libre, gratuit
     • Elasticsearch
     • Logstash
     • Kibana
     • Beat
     • Curator
     • …
     

     View Slide

121. modèle économique
     Libre, gratuit
     • Elasticsearch
     • Logstash
     • Kibana
     • Beat
     • Curator
     • …
     Services
     • formation
     • assistance
     • consulting
     • X-Pack
     

     View Slide

122. modèle économique
     Libre, gratuit
     • Elasticsearch
     • Logstash
     • Kibana
     • Beat
     • Curator
     • …
     Services
     • formation
     • assistance
     • consulting
     • X-Pack
     Cloud
     • As A Service
     • Hébergé
     

     View Slide

123. conclusions
     

     View Slide

124. Hébergement et Infogérance Open Source
     Jérémy Lecour
     

     View Slide

125. forge.evolix.org
     www.evolix.fr
     @evolix – @jlecour
     info@evolix.fr
     @
     wiki.evolix.org
     Git logo from git-scm.org
     Chain by Bohdan Burmich from the Noun Project
     Paw Print by Mattijs Dekkers from the Noun Project
     Twitter logo from twitter.com
     

     View Slide