$30 off During Our Annual Pro Sale. View Details »

ELK how we use it

August 02, 2016

ELK how we use it


August 02, 2016

More Decks by dknx01

Other Decks in Programming


  1. Logging with ELK How we use it a secured environment

    or mabye a strange environment
  2. Some basic infos • PHP developer • working at: Little

    Bird GmbH • Needed to handle our log (exception) much better
  3. What is ELK Elasticsearch • search engine (Lucene based) •

    indices data • provides REST-API • saves data as NoSQL (JSON)
  4. What is ELK Logstash • Log data processor / ETL

    • 165+ plugins for inputs (files, TCP/UDP-Ports, STDIN, etc.) • filter, normalise and transform data (Date-Format, GeoIP, RegExp, Anonymize, etc.) • send data to STDOUT, databases, files, chats, nagios etc.
  5. What is ELK Kibana • nodeJS based GUI for elasticsearch

    • data visualisation (charts) • data analyses
  6. Current situation and why to change • Several hosting partner

    with different systems • Writing separate log files for each customer (Symptoms and Apache) • Exception are catched and send via email to a shared mail account • Not every developer has access to mail account • Bugs are not always reported with stackstrace/infos • Hard to track if exception occurred on other system or many times
  7. Our environment - our problems • Many hosting providers •

    Different OS: ◦ Debian ◦ Ubuntu ◦ SuSe • Different access rights ◦ Full access (internal system) ◦ Only one port: 443 ◦ Non-HTTP(S)/SMTP access only on request • No central server • One separate log file per customer (environment)
  8. How we solved it • Central logging server on our

    in-house system • Tunneling the desired connection through port 443 • Application server only needs ◦ Stunnel - a tunneling software ◦ Java - we need it for the application, too ◦ Logstash as standalone downloaded application • Customer (environment) name is extracted from file name
  9. The infrastructure with logging server

  10. That’s how it looks like

  11. That’s how it looks like

  12. What to do next • Central log server fail over

    (more than one Redis, Elasticsearch server) • Automatical analysation of log and notify developers • Only process neccessary log messages
  13. End