Metrics Based Decisions @ SoftShake 2014

Transcript

1. Metrics Based Decisions Johnny Graber
2. About Johnny Graber •Software Engineer @ Swiss Medical Association •Twitter: @j_graber •E-Mail: JG@JGraber.ch •Blog: ImproveAndRepeat.com
3. Agenda •The Big Unknown •Enlighten your Application •Experiment & Decide
4. The Big Unknown
5. The World of Big Data •Every action is traced •Every click is analysed •Products meet exactly our needs •Privacy is dead
6. Our Application – the Black Box!? •Users? •Needs? •Purpose?
7. Assumptions •Never in short supply •Hope for the best •No replacement for facts
8. Validated Learning 1. Specify a goal 2. Specify a metric 3. Act to achieve the goal 4. Analyse the metric 5. Improve and try again
9. Enlighten your Application
10. Log Files •Widely used •Flat •Usefulness often limited •Only implicit meaning
11. Structured Logging •Next generation of log files •Explicit meaning •Easy to understand
12. Serilog •Structured logging for .Net •Open Source (Apache License, Version 2.0) •Web: http://serilog.net/ •Easy to use & powerful
13. Basic Configuration Log.Logger = new LoggerConfiguration() .WriteTo.Console() .CreateLogger(); Log.Information("A simple message"); 2014-10-xx xx:xx:xx [Information] A simple message
14. Sinks for Serilog •Write log events to various storage formats •More than 20 out of the box supported •Can be extended
15. Write JSON Log.Logger = new LoggerConfiguration() .WriteTo.Sink( new FileSink("logs.txt", new JsonFormatter(), null)) .CreateLogger(); { "Timestamp":"2014-10-08T21:34:25.39+02:00", "Level":"Information", "MessageTemplate":"A simple message"}
16. Meaningful Log Messages •Creation is simple •But we have to care! •Otherwise: flat log files
17. Bad Log Messages  Don’t use String.Concat() and positional placeholders: Log.Information("{0} ordered {1}", customer, order);
18. Better Log Messages  Use named parameters: Log.Information("{customer} ordered {order}", customer, order);
19. Whole Objects  Dump whole objects with the @ modifier: Log.Information("{@customer} ordered {@order}", customer, order);
20. Meaningful Log Message  Use what makes sense: Log.Information("{customer} ordered {orderId}", customer, order.Id);
21. Seq •Search server for structured log events •Written in .Net •Free for a single user, plans start at 75$/5 users •Web: GetSeq.net
22. Search for Events using •Full-text search •Directly paste error messages •Partial messages using Contains() •Properties of the event
23. The Dashboard •Overview on your application •Counters to measure important events •Charts to display them •Queries are fully customisable
24. Demo: Using Seq and Serilog How simple is it to write structured log messages? Demo
25. Elasticsearch •End-to-end search and analytics platform •Open Source (Apache License, Version 2.0) •For massive amounts of structured & non-structured data •REST API
26. Elasticsearch & Serilog // Create URI to the Elasticsearch server var elasticServer = new Uri("http://localhost:9200"); // Create Logger Log.Logger = new LoggerConfiguration() .WriteTo.ElasticSearch(server: elasticServer) .CreateLogger();
27. Kibana •Powerful front-end for Elasticsearch •Great to visualise log messages •Fully customisable
28. Kibana Dashboard
29. Visualise Matching Events
30. Experiment & Decide
31. Challenges •Correlation or causation? •Observer effect •Privacy vs transparency
32. Respecting the privacy of our users •Only collect necessary data •Obfuscate ID’s •Omit specific details •Work with an approximation
33. Let’s Experiment •Given: A web application in ASP.Net •Goal: Our customers only use a secure operating system •Metric: #users on Windows XP
34. Before we Act •Know the current situation •Identify missing information •Create a baseline
35. Existing Data •Internal • IIS log files • Google Analytics • CRM •External • StatsCounter.com
36. Collecting Additional Data •Collect the operating system at log in •We use • Serilog to write log messages • Seq as a sink • System.Web.Mobile.MobileCapabilities for OS detection
37. Demo: Collecting Data How can Serilog help us to collect data? Demo
38. Options to Act •Article in the newsletter •Warning message on the login screen •Direct mailing to the users
39. Measure & Decide •Improve what works •Stop doing what doesn’t work •Decide if continuing is worth the effort •Repeat with the next assumption
40. Conclusion •Measuring is routine work •Serilog can collect powerful data •The hard part are the experiments
41. Resources •Examples: http://ImproveAndRepeat.com •Serilog: http://serilog.net/ •Seq: http://getseq.net/ •Elasticsearch: http://www.elasticsearch.org/
42. Questions? JG@JGraber.ch
43. Thank You!
44. Demo: Using Seq and Serilog How simple is it to write structured log messages?
45. Run SeqConfiguration Log.Information("A simple info message"); Log.Debug("Debug is only for internal use"); Log.Error("Now we have an error!"); try { var result = a / b; } catch (Exception ex) { Log.Fatal(ex, “Division {dividend}/{divisor} failed", a, b); }
46. Stored Events in Seq
47. Stored Events in Seq Back
48. Demo: Collecting Data How can Serilog help us to collect data?
49. Build a Web Application •Create a web application with the VS template •Add Serilog and Seq as a sink •Configure Serilog in Application_Start
50. Modify Application_Start (Global.asax) void Application_Start(object sender, EventArgs e) { // Create Logger Log.Logger = new LoggerConfiguration() .WriteTo.Seq("http://localhost:5341") .CreateLogger(); Log.Information("Start at {startup}", DateTime.Now); //Other code … }
51. Modify web.config <browserCaps> <use var="HTTP_USER_AGENT" /> <filter> <case match="Windows NT 6.1"> platform="Windows 7" </case> <case match="Windows XP|Windows NT 5\.1|WinNT 5\.1"> platform="Windows XP" </case> </filter> </browserCaps> </system.web>
52. Track successful Log-in protected void LogIn(object sender, EventArgs e) { if (IsValid) { // Validate the user password var manager = new UserManager(); ApplicationUser user = manager.Find(UserName.Text, Password.Text); if (user != null) { IdentityHelper.SignIn(manager, user, RememberMe.Checked); LogSuccessfulLogin(user); IdentityHelper.RedirectToReturnUrl(…);
53. Track successful Log-in private void LogSuccessfulLogin(ApplicationUser user) { // Minimalistic log message => only the platform is logged Log.Information("login using {os}", Context.Request.Browser.Platform); // More information at login => UserName and platform Log.Information("login by {user} using {os}", user.UserName, Context.Request.Browser.Platform); // Even more information at login => UserName, platform and session Log.Information("login by {user} using {os} in session {session}", user.UserName, Context.Request.Browser.Platform, Context.Session.SessionID); }
54. Check Events in Seq Back