Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)

Markus H
August 03, 2019

Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)

My talk from PyCon Australia 2019

A write-up of my talk is available at https://markusholtermann.eu/2019/04/logging-rethought/

Markus H

August 03, 2019
Tweet

More Decks by Markus H

Other Decks in Technology

Transcript

  1. Logging Rethought 2
    The Actions of Frank Taylor Jr.
    markusholtermann.eu • gitlab.com/MarkusH • github.com/MarkusH • @m_holtermann

    View Slide

  2. Hi, I’m
    Markus Holtermann
    ●
    Engineer at
    ●
    Django Core Contributor

    View Slide

  3. @m_holtermann
    The Problem(s)
    We’re Facing

    View Slide

  4. @m_holtermann
    The Current State
    Of Logging

    View Slide

  5. import logging
    logger = logging.getLogger(__name__)
    logger.error(
    "Login failed because the "
    "connection to the authentication "
    "provider %s timed out.",
    provider_name
    )

    View Slide

  6. [2019-04-01T10:47:04.139+00:00] [ERROR]
    [srv01] Login failed because the connection to
    the authentication provider google timed out.

    View Slide

  7. @m_holtermann
    Our Logging Is Broken

    View Slide

  8. @m_holtermann
    What IP and host did the
    server try to connect to?

    View Slide

  9. @m_holtermann
    What was the timeout
    limit at the time?

    View Slide

  10. @m_holtermann
    How many other attempts
    were made to talk to that
    authentication provider?

    View Slide

  11. @m_holtermann
    Where other outgoing
    connections affected by
    timeout errors as well?

    View Slide

  12. @m_holtermann
    Were other servers affected
    by the same problem as well?

    View Slide

  13. @m_holtermann
    Adding Structure
    To Our Logs

    View Slide

  14. import structlog
    logger = structlog.get_logger("structlog")
    logger.error(
    "auth_provider_failed",
    provider_name=provider_name,
    provider_ip=provider_ip,
    timeout=timeout,
    )

    View Slide

  15. 2019-04-01 10:47:04.139 [error ]
    auth_provider_failed
    provider_ip='8.8.8.8'
    provider_name='google'
    server='srv01'
    timeout=5

    View Slide

  16. {
    "event": "auth_provider_failed",
    "timestamp": 1554115624.139,
    "level": "error",
    "provider_name": "google",
    "provider_ip": "8.8.8.8",
    "timeout": 5
    }

    View Slide

  17. WYSIWYU
    What You See Is What You Understand

    View Slide

  18. @m_holtermann

    View Slide

  19. @m_holtermann
    A Picture Says More Than
    A Thousand Words

    View Slide

  20. @m_holtermann

    View Slide

  21. @m_holtermann
    Tracing Events

    View Slide

  22. import uuid, structlog
    logger = structlog.get_logger("structlog")
    def structlog_middleware(get_response):
    def _inner(request):
    request.trace_id = (
    request.META.get("HTTP_X_TRACE_ID")
    or str(uuid.uuid4())
    )
    log = logger.new(trace_id=request.trace_id)
    return get_response(request)
    return _inner

    View Slide

  23. import structlog
    logger = structlog.get_logger("structlog")
    def structlog_user_middleware(get_response):
    def _inner(request):
    if request.user.is_authenticated:
    logger.bind(user_id=request.user.pk)
    return get_response(request)
    return _inner

    View Slide

  24. @m_holtermann
    Logging For
    European Users

    View Slide

  25. @m_holtermann
    Never log any secrets. EVER!

    View Slide

  26. @m_holtermann
    Ensure your log data store is
    secured and not world-
    readable! NO, REALLY!

    View Slide

  27. @m_holtermann
    Explicitly log whatever
    should end up in logs.

    View Slide

  28. # DO
    logger.error("some_event", foo="bar", lorem="ipsum")
    # DON'T DO
    logger.error("some_event", **some_object.__dict__)

    View Slide

  29. @m_holtermann
    Who the heck is
    Frank Taylor Jr.?

    View Slide

  30. @m_holtermann
    Frank William
    Abagnale Jr.!

    View Slide

  31. @m_holtermann
    Demo

    View Slide

  32. Demo Example
    gitlab.com/MarkusH/django-structlog

    View Slide

  33. @m_holtermann

    View Slide

  34. @m_holtermann

    View Slide

  35. @m_holtermann

    View Slide

  36. @m_holtermann

    View Slide

  37. @m_holtermann

    View Slide

  38. Example
    gitlab.com/MarkusH/django-structlog

    View Slide

  39. @m_holtermann
    Thank you!

    View Slide