LOGGING WITH Alex Stupka, Developer [email protected] Thursday, January 20, 2011 1.Introduction 1.Developer at Customink 2.If you have any questions, feel free to stop me and ask 

CUSTOMINK USAGE Products and Services, EComm, and Operations Technology are putting finishing touches on a Catalog service which uses Mongoid in place of an RDBMS & ActiveRecord. Operations Technology uses the Central Logger in production for all back end services and order processing. Mongoid is used on Operations Technology’s Supplier and Decorator Service for tracking supplier information/blanks inventory and printer/vendor data respectively. Thursday, January 20, 2011 Catalog manages products, styles clothing, and other items we sell. The primary functions of the supplier service are to: • store general supplier information • store blanks inventory data (e.g. availability and pricing) • support automated blanks ordering • support ordering logistics Decorator Service This service maintains all relevant data to describe decorator's (aka printer/vendor). • Vendor's details • Capacity • Supply inventory • Capabilities 

PROBLEM Complex Production Environments Orchestrated Services Need Hoptoad Complementary Auditing Thursday, January 20, 2011 1.Problem 1.Load balanced applications 2.Orchestrated services 

LOGGING OPTIONS Network File System Syslog Database Hoptoad Thursday, January 20, 2011 1. Solutions 1. Network File System logs 2. Syslog 1. File based 3. Database 1. Relational 4. Hoptoad 1. Exceptions 2. Throttles logging 

WHY MONGO? Introduces the Company to MongoDB Capped Collections are Great Easy to Add New Fields Easy to Install and Deploy Lightweight Development Thursday, January 20, 2011 1. Good way to introduce developers and the company to Mongo 1. not-mission critical. Logs are teed to file as backup 2. model is tinker-friendly 2. Capped collections 1. emulate rolling logs 2. Never get larger than their configured size 3. natural ordering 4. fast, can handle large amount of audit data 3. Easy to add new fields 4. Easy to install and deploy 

MONGO LOGGER Phil Burrows https://github.com/peburrows/mongo_db_logger Thursday, January 20, 2011 Original idea & code 

CENTRAL LOGGER Subclasses BufferedLogger It’s a Gem, Loaded as a Railtie Captures Useful Metadata Thursday, January 20, 2011 1. Subclasses the Buffered Logger in Rails in order to tee logs to a mongo database 2. Its a gem 1. Works with Rails 2.3.8 to 3.0 2. Rail-Tie (replaces the buffered logger during rails boot) 3. Works with Heroku by using separate config 4. database.yml 5. central_logger.yml 3. default captures metadata you’re probably interested in 1. fields (log message, request_time, params, application, controller, elapsed execution time) 

RECORD STRUCTURE Thursday, January 20, 2011 Go over individual records 

ACTUAL DATA Thursday, January 20, 2011 Go over individual elements 

INSTALLATION gem install central-logger Update ApplicationController Create Config File Estimate the Size of Your Capped Collection Thursday, January 20, 2011 1. gem install 2. Update ApplicationController 1. uses around_filter to audit all logging that occurs in an action 3. config file 1. database.yml - if using RDBMS 2. mongoid.yml - for ODM 3. central_logger.yml 4. Do some work to figure out how large your capped collection needs to be. 1. How many days of log data do you want? 2. count number of apps, current size of logs divided by days (assuming current load is normal) and multiply by number of machines for each app and the number of days you need 3. May want index for tail functionality 1. index uses gt to do this 2. link to tail article 

APPLICATION CONTROLLER Thursday, January 20, 2011 Module that goes around 

CONFIG FILE Thursday, January 20, 2011 Mongo key in your db yaml or central logger yml or mongoid yml 

AROUND FILTER Thursday, January 20, 2011 Benchmark call in around filter wraps call to the controller Exceptions are logged Insert always attempted 

IS IT WORKING? http://localhost:3000/log/index Thursday, January 20, 2011 1. rails console - look at type of Rails.logger 2. Go to an action you want to log, such as a listing page that extends ActionController 3. mongo console 1. use system_log (or whatever database you specified 2. show collections 3. db.development_log.count() 4. Falls back to Buffered Logger if logging isn’t working. 

EXTRAS Replica Set Support Mongo Auth Thursday, January 20, 2011 1. Replica set support - 1. in case your logs are extra important 2. Mongo Auth 1. Mongo recommends no auth, and its just basic 2. should use inside trusted network or over vpn (secure at other network layer) 

GOTCHAS Beware of Unserializable Types Exceptions will have Central Logger in the call stack Thursday, January 20, 2011 1. Beware of buried BSON unserializable data types in your log statements 1. Will add this feature 2. Exceptions will usually have central_logger in the stack trace because of the around filter 

CENTRAL LOG VIEWER Need Flattened Log Messages Need Log Messages Across Servers Grep Tail -f Functionality Thursday, January 20, 2011 1. Central Log Viewer 1. Want to see flattened log messages inside a document rooted at the action 2. Need to see log messages orchestrated across multiple services 3. Grep 4. Tail -f while making request 

ARCHITECTURE Trivial Rails Controller JQuery Handlebars CoffeeScript Thursday, January 20, 2011 rails controller uses eval (oh no)! no ODM or ORM installed, just jason returned to the client Handlebars - Templating DSL based on Mustache, but runs on client Coffeescript - concise language that eliminates complexity and ceremony of JS 

INSTALLATION Just a Rails App Modify Config File Add Indexes Thursday, January 20, 2011 1. Currently just a rails app piggybacking off the central_logger 2. Change the config file to point to your central logger 3. Add indexes 

HANDLEBARS Thursday, January 20, 2011 relative path specified in JSON document rooted at action compile step - creates an AST consisting of jquery functions that evaluate incoming JSON 

FIRE IT UP Thursday, January 20, 2011 1. Quick and dirty query generation 2. find logs for particular apps 3. Grep through messages 4. Tail 

http://web.progress.com/en/esb-architecture-lifecycle-definition.html http://boingboing.net/2007/04/interview-with-john-.html http://bornatthecrestoftheempire.blogspot.com/2007_08_05_archive.html http://www.ugo.com/movies/greatest-movie-ever-step-brothers?page=17 http://www.autofish.net/asjad/images/internet_videos/help_computer.jpg http://upload.wikimedia.org/wikipedia/commons/f/f3/Question_mark_alternate.png http://images.wikia.com/simpsons/images/7/7b/Homerclones.jpg http://technoccult.net/wp-content/uploads/2010/02/scanners-headexplode.jpg http://electrorash.com/wp-content/uploads/2010/06/46ff126b3093a90cc280c35e3f9b15f4.jpg Images Thursday, January 20, 2011 

QUESTIONS? Alex Stupka [email protected] https://github.com/astupka twitter: astupka https://github.com/customink/central_logger Thursday, January 20, 2011 

THANKS! Thursday, January 20, 2011