Slide 1

Slide 1 text

Why Logging is Important ...and how Logality can help you with that By Thanos - @thanpolas Node.js

Slide 2

Slide 2 text

Hi, I am Thanos Nearly 50 published NPM packages. Over 100 OSS repos. Contributor in major and small OSS projects. Retired Software Engineer turned Gardener . Founder of and . skgtech.io devitconf.org twitter.com/thanpolas

Slide 3

Slide 3 text

Today's Menu 1. Why Logging Is Important 2. Logging Requirements 3. What and How to Log 4. Where and When to Log 5. How Logality Can Help 6. Other Logging Packages

Slide 4

Slide 4 text

Why Logging Is Important Having a log at the right place, with the right kind of information can save your 🥓🥓🥓.

Slide 5

Slide 5 text

Logging Helps You With... Why Logging is Important Debugging. Error Tracing. Performance Troubleshooting. Accounting. Audit Trail. Security.

Slide 6

Slide 6 text

Logging Requirements Your Environment Has multiple instances of your application. Has multiple workers. Probably has multiple stacks and runtimes. Uses multiple services and databases. Runs on multiple platforms.

Slide 7

Slide 7 text

Consuming Logs Logging Requirements Common Logging Aggregator. Queryable Aggregator --> JSON Logs. Common Logging Schema.

Slide 8

Slide 8 text

Consuming Logs Logging Requirements Common Logging Schema Allow me to repeat this...

Slide 9

Slide 9 text

What and How to Log Who am I? Where am I? What am I? Metadata Flags and Tagging A Log Message Should Contain

Slide 10

Slide 10 text

Who Am I? What and How to Log What is my runtime? (Node.js?) What hostname am I on? (abc.aws.com) What is my process id? (10420) What is my process name? (npm start)

Slide 11

Slide 11 text

Where and What Am I? What and How to Log What is my logging Level & Severity? (Info, Warn) What Module is the log from? (app/service/db.js) What function is this log from? (init()) What time is it? (2021-05-06T12:54:31.978Z)

Slide 12

Slide 12 text

Flags and Tagging What and How to Log In a JSON logger, you should be able to add certain [boolean] flags so you can later easily query and filter for them. Example Flags: security: true audit: true error: true

Slide 13

Slide 13 text

How To Log What and How to Log Pretty Print for local development. Muted for automated testing. JSON output for production. [2021-04-10T13:05:32.712Z] ▶ notice /app/services.boot.js - Booting Services... 1 {"level":"notice","severity":5,"dt":"2021-04- 10T13:04:03.215Z","message":"Booting Services...","context":{"runtime": {"application":"skgbot"},"source": {"file_name":"/app/services.boot.js"},"system": {"hostname":"192.168.1.74","pid":49862,"process_name":"/Users/thanpolas/.nvm/ver sions/node/v14.15.5/bin/node"}},"event":{}} 1 Pretty Print JSON

Slide 14

Slide 14 text

Where and When to Log

Slide 15

Slide 15 text

Debugging & Error Tracing Where and When to Log Application Boot (OS, User, NODE_ENV). Modules / Services Booting Up. Node.js Error Handlers (uncaughtException, uncaught promise rejection). Express & http Error Handlers. All your error catch statements, if they don't bubble up.

Slide 16

Slide 16 text

Accounting, Audit, Security Where and When to Log All mutating operations. Becomes an Audit Trail when saved on immutable store. Highly Secure Application require audit trails on querying as well. Use flags to tag log types (audit, security).

Slide 17

Slide 17 text

Accounting, Audit, Security Where and When to Log Beware of PII! (emails, IPs, names) tokenise data

Slide 18

Slide 18 text

How Logality Can Help thanpolas/logality

Slide 19

Slide 19 text

What is Logality? Logality is a versatile and extensible logger for Node.JS

Slide 20

Slide 20 text

Logality Features JSON and Pretty Print log messages. Extend or alter the logging schema to fit your needs. Customise built-in serialisers by overwriting them to create your own logging schema. Middleware support.

Slide 21

Slide 21 text

Logality Features Allows full manipulation of output. Asynchronous operation. Use in libraries and compose multiple Logality instances on the root project. Automatically detects the module filename and path and includes them in the log.

Slide 22

Slide 22 text

JSON Output Logality Features { "severity": 6, "level": "info", "dt": "2018-05-18T16:25:57.815Z", "message": "hello world", "event": {}, "context": { "runtime": { "application": "testLogality" }, "source": { "file_name": "/test/spec/surface.test.js" }, "system": { "hostname": "localhost", "pid": 36255, "process_name": "node ." } } }

Slide 23

Slide 23 text

JSON Output Logality Features { "severity": 6, "level": "info", "dt": "2018-05-18T16:25:57.815Z", "message": "hello world", "event": {}, "context": { "runtime": { "application": "testLogality" }, "source": { "file_name": "/test/spec/surface.test.js" }, "system": { "hostname": "localhost", "pid": 36255, "process_name": "node ." } } } Automatic Detection

Slide 24

Slide 24 text

Pretty Print Output Logality Features Logality Uses the Syslog Severity Levels (RFC 5424)

Slide 25

Slide 25 text

Logality Serialisers Logality Features Serialisers are triggered by defined keys in the context object. Each serialiser is configured to listen to a specific key. log.info('User Logged in', { user: udo, }); 1 2 3

Slide 26

Slide 26 text

Built-In Serialisers Logality Features User Serialiser log.info('User Logged in', { user: user }) Output "context": { "user": { "id": 10, "email": "[email protected]", } }

Slide 27

Slide 27 text

Built-In Serialisers Logality Features Output "event":{ "error":{ "name":"Error", "message":"Broke", "backtrace": "Stack Trace...", } } Error Serialiser log.error('User Logged in', { error: exception })

Slide 28

Slide 28 text

Built-In Serialisers Logality Features Output "event":{ "http_request": { "headers": {}, "host": "localhost", "method": "GET", "path": "/", "query_string": "", "scheme": "http" } } Express Request Serialiser function index (req, res) { log.info('Index request', { req }); }

Slide 29

Slide 29 text

Custom Serialisers Logality Features You can define your own serialisers const mySerialisers = { order: function (order) { return { path: 'context.order', value: { order_id: order.id, sku_id: order.sku, total_price: order.item_price * order.quantity, quantity: order.quantity, }, }; }, }; log.info('New order', {order: orderItem})

Slide 30

Slide 30 text

Overwrite Built-In Serialisers Logality Features You can overwrite built-in serialisers const mySerialisers = { user: function (user) { return { path: 'context.user', value: { id: user.id, email: user.email, first_name: user.first_name, last_name: user.last_name, }, }; }, }; log.info('User Logged In', {user: req.user})

Slide 31

Slide 31 text

Middleware Support Logality Features Middleware are invoked after serialisers. You may add multiple middleware. The "logContext" object is a JS Native Object, representing the entire log message. "logContext" is Mutable. logality.use((logContext) => { delete logContext.user; });

Slide 32

Slide 32 text

Async Middleware Support Logality Features You can configure Logality for asynchronous operation logality.use(async (logContext) => { await db.write(logContext); }); logality.use(async (logContext) => { await slack.send(slack.format(logContext)); }); Consequently, all logging commands need async invocation: await log.info('Something happened');

Slide 33

Slide 33 text

Output Manipulation Logality Features You can fully manipulate the master output: const Logality = require('logality'); const logality = Logality({ appName: 'service-something', prettyPrint: false, async: false, output: (logContext) => { const logMessage = JSON.stringify(logContext); process.stdout.write(logMessage); }, }); The "logContext" object is a JS Native Object, representing the entire log message. It is mutable.

Slide 34

Slide 34 text

Use in Libraries Logality Features Logality can safely be used in libraries! const thirdPartyLib = require('thirdPartyLibrary'); /** ... */ const myLogality = Logality(); myLogality.pipe(thirdPartyLib.logality); 1 2 3 4 5 6 7

Slide 35

Slide 35 text

Other Logging Libraries

Slide 36

Slide 36 text

Other Logging Libraries Winston 6.1m downloads per week https://github.com/winstonjs/winston Pino.js 1.5m downloads per week https://github.com/pinojs/pino Bunyan 1.4m downloads per week https://github.com/trentm/node-bunyan

Slide 37

Slide 37 text

Comparison Logality Winston Bunyan Pino JSON Output ✅ ✅ ✅ ✅ Pretty Print ✅ ✅ ❌ ✅ Custom Log Levels ❌ ✅ ❌ ✅ Serialisers ✅ ❌ ✅ ✅ Middleware ✅ ✅ ❌ ✅ Mutate JSON Schema ✅ ✅ ❌ ❌

Slide 38

Slide 38 text

Comparison Logality Winston Bunyan Pino Output Destination ✅ ✅ ✅ ✅ Mutate Output ✅ ✅ ❌ ❌ Async Operation ✅ ❌ ❌ ❌ Filename Detection ✅ ❌ ❌ ❌ Speed Optimised ❌ ❌ ❌ ✅ Used in Libraries ✅ ❌ ❌ ❌

Slide 39

Slide 39 text

Recap Logging is important for: Debugging and troubleshooting. Security and Audit. Performance monitoring. Common Schema and appropriate logs can make all the difference. You have many choices for a logger, choose what is appropriate for you.

Slide 40

Slide 40 text

Thank you Thanos Polychronakis @thanpolas https://speakerdeck.com/thanpolas

Slide 41

Slide 41 text

Thank you Thanos Polychronakis @thanpolas https://speakerdeck.com/thanpolas Questions?