Structured Logs

The idea behind structured logging is simple: instead of having applications write logs that need to be parsed via regular expressions into JSON objects that you index into Elasticsearch, you make the application write JSON objects directly.

Transcript

1. Structured Logs embrace the power of machine-readable logs HELIO MEDEIROS @helmedeiros
2. Logging: The process of recording events during a computer program execution “
5. web.app { logs
 stream past present
6. 22:55:40 INFO Initializing hub 22:55:40 INFO Starting web server 22:55:42 INFO Incoming connection from 127.0.0.1:37843 22:55:42 DEBUG Connection upgraded 22:55:42 DEBUG Receive message: “ping”
7. >>> logging
8. log = logging.getLogger() log.info("Starting web server”) log.debug(“Actual memory: %s”, $mem)
9. Structured logging: Logging events with specific fields (key= value) “
10. INFO[06-06|22:55:40] Incoming connection component= websocket.serveWs remote= 127.0.0.1:37843 event= connection.new INFO[06-06|22:55:40] Connection upgraded component= websocket.serveWs remote= 127.0.0.1:37843 event= connection.upgraded DEBUG[06-06|22:55:42] Receive message component= websocket.connection remote= 127.0.0.1:37843 event= message.new message= ping
11. >>> logging structured
12. >>> log = logging.getLogger() >>> log = log.bind( … user=“helio” … remote=“127.0.0.1:36509” … )
13. >>> log.info(“message.new”, message=“ping”) 22:55:42 DEBUG user=“helio” remote=“127.0.0.1:37843” event=“message.new” message=“ping”
14. “Unstructured” logs suffer from two big problems
15. 1 It is difficult to parse
16. 2 Individual messages lack context
17. { “timestamp”: “Jun 6 15:33:06 +0000”, “app”: “web”, “message”: “Receive Message”, “event”: “message.new”, “ip”: “127.0.0.1”, “Trace-Id”: “48e276bd-cb52-4e12- abaf-2f0fcf746786” }
18. correlation identifiers: Unique identifier that allow reference to particular transactions or event chains “
19. Structured Logs embrace the power of machine-readable logs HELIO MEDEIROS @helmedeiros