Marshall Simmonds - Log File Analysis

At Define Media Group, Marshall and his team have been spearheading deep reviews of some of the biggest publishers’ raw data files. It’s intense work but what they’ve learned is insightful and applicable to all industries. Marshall’s going to be sharing those insights with you.

Transcript

1. Marshall Simmonds Define Media Group, Inc. mdsimmonds Log File Analysis
2. mdsimmonds www.DefineMG.com
3. mdsimmonds www.DefineMG.com
4. mdsimmonds www.DefineMG.com
5. mdsimmonds www.DefineMG.com
6. mdsimmonds www.DefineMG.com Why Do We Do This?
7. mdsimmonds www.DefineMG.com
8. mdsimmonds www.DefineMG.com About.com Project 2000
9. mdsimmonds www.DefineMG.com About.com Project 2000 • Big 12 Search Engines
10. mdsimmonds www.DefineMG.com About.com Project 2000 • Big 12 Search Engines • Most Active
11. mdsimmonds www.DefineMG.com About.com Project 2000 • Big 12 Search Engines • Most Active • Pages Requested
12. mdsimmonds www.DefineMG.com About.com Project 2000 • Big 12 Search Engines • Most Active • Pages Requested • Pages NOT Requested
13. mdsimmonds www.DefineMG.com About.com Project 2000 • Big 12 Search Engines • Most Active • Pages Requested • Pages NOT Requested • Too L A R G E
14. mdsimmonds www.DefineMG.com
15. mdsimmonds www.DefineMG.com Server Logs Answer • IP – Who • User Agent – Who • Method – What • Response – What • URI_Request – Where • Date – When
16. mdsimmonds www.DefineMG.com Boot Strapping
17. mdsimmonds www.DefineMG.com Boot Strapping • Excel
18. mdsimmonds www.DefineMG.com Boot Strapping • Excel
19. mdsimmonds www.DefineMG.com AnnieLytics.com • Annie Cushing • SeerInteractive.com
20. mdsimmonds www.DefineMG.com • http://moz.com/blog/server-log-essentials-for-seo
21. mdsimmonds www.DefineMG.com • Never seen it before? • Never know what’s in the deep recesses
22. mdsimmonds www.DefineMG.com Case Study #1 • Bridal Site • Requested:
23. mdsimmonds www.DefineMG.com Case Study #1 • Bridal Site • Requested: • Started with one day- be very very careful – Status Codes – Number of URLs crawled per directories – Type of URL “/[whatever]” • Be kind and ask for just Bing • Nawww – Ask for anything Google • .CSV is fine • ~100K Pages
24. mdsimmonds www.DefineMG.com
25. mdsimmonds www.DefineMG.com
26. mdsimmonds www.DefineMG.com
27. mdsimmonds www.DefineMG.com
28. mdsimmonds www.DefineMG.com
29. mdsimmonds www.DefineMG.com Most Popular? Nope.
30. mdsimmonds www.DefineMG.com Most Popular? Nope.
31. mdsimmonds www.DefineMG.com Most Popular? Nope.
32. mdsimmonds www.DefineMG.com Most Popular? Nope.
33. mdsimmonds www.DefineMG.com Most Popular? Nope.
34. mdsimmonds www.DefineMG.com Most Popular? Nope. ?????
35. mdsimmonds www.DefineMG.com Post Mortem
36. mdsimmonds www.DefineMG.com Post Mortem • Definitely compare against crawl data • Not perfect for auditing and is very rabbit-holey
37. mdsimmonds www.DefineMG.com Post Mortem • Definitely compare against crawl data • Not perfect for auditing and is very rabbit-holey • Googlebot spent a lot of time in /Search
38. mdsimmonds www.DefineMG.com Post Mortem • Definitely compare against crawl data • Not perfect for auditing and is very rabbit-holey • Googlebot spent a lot of time in /Search • IP addresses of Spiders are interesting but won’t reveal much so avoid that rabbit hole • The actionable item is, that isn’t actionable
39. mdsimmonds www.DefineMG.com Post Mortem • Definitely compare against crawl data • Not perfect for auditing and is very rabbit-holey • Googlebot spent a lot of time in /Search • IP addresses of Spiders are interesting but won’t reveal much so avoid that rabbit hole • The actionable item is, that isn’t actionable • Crawl budget?
40. mdsimmonds www.DefineMG.com What Learned?
41. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different
42. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different Crawl ≠ Popularity
43. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different Crawl ≠ Popularity Be careful about the data pull size
44. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different Crawl ≠ Popularity Be careful about the data pull size Previously unknown directory getting hit
45. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different Crawl ≠ Popularity Be careful about the data pull size Previously unknown directory getting hit Saw that Robots.txt got hit 1,394 times
46. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different Crawl ≠ Popularity Be careful about the data pull size Previously unknown directory getting hit Saw that Robots.txt got hit 1,394 times Small data sets yield small trends
47. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different Crawl ≠ Popularity Be careful about the data pull size Previously unknown directory getting hit Saw that Robots.txt got hit 1,394 times Small data sets yield small trends Googs like /search
48. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different Crawl ≠ Popularity Be careful about the data pull size Previously unknown directory getting hit Saw that Robots.txt got hit 1,394 times Small data sets yield small trends Googs like /search Identified that nothing tremendously wrong
49. mdsimmonds www.DefineMG.com What Learned? What Googlebot sees What Browsers see = different Crawl ≠ Popularity Be careful about the data pull size Previously unknown directory getting hit Saw that Robots.txt got hit 1,394 times Small data sets yield small trends Googs like /search Identified that nothing tremendously wrong Saw cool stuff, increased value, learned
50. mdsimmonds www.DefineMG.com
51. mdsimmonds www.DefineMG.com Case Study #2 • Enterprise level site covering many categories • Month’s worth of data • Requested: – Everything from Googlebot (and some other stuff) • ~1,000,000 pages
52. mdsimmonds www.DefineMG.com
53. mdsimmonds www.DefineMG.com • Google accounts for more than 96% of month’s spider visits. • Bing is a distant second • Few visits from Baidu, Yandex, Yahoo Slurp • Some User Agents get tagged both as regular browsers (Chrome, Firefox, etc.) and as spiders
54. mdsimmonds www.DefineMG.com Et tu Marissa?
55. mdsimmonds www.DefineMG.com Et tu Marissa?
56. mdsimmonds www.DefineMG.com Daily Time Trend • Average daily spider visits:~1.3million
57. mdsimmonds www.DefineMG.com Daily Time Trend • Average daily spider visits:~1.3million
58. mdsimmonds www.DefineMG.com
59. mdsimmonds www.DefineMG.com Hourly Trend: Heat map • Lots of consistency, some sharp drops and recoveries • No obvious hourly time trend: most hours show 40‐70,000 hits
60. mdsimmonds www.DefineMG.com Hourly Trend • Messy, with consistent range of visits • Morning/evening drops are clear
61. mdsimmonds www.DefineMG.com Google Dance?
62. mdsimmonds www.DefineMG.com Google Dance?
63. mdsimmonds www.DefineMG.com Spider Visits by Asset Type • Most spider visits did not favor an asset • Homepage visits were rare!
64. mdsimmonds www.DefineMG.com Distinct Sections & Visits by Spider
65. mdsimmonds www.DefineMG.com Distinct Sections & Visits by Spider
66. mdsimmonds www.DefineMG.com Distinct Sections & Visits by Spider
67. mdsimmonds www.DefineMG.com Spider Variation • Spiders show some distinct behavior • Bing IPs appear to index less than Google • There are two distinct classes of Google IP addresses. • It’s possible spiders function significantly differently across companies
68. mdsimmonds www.DefineMG.com Distinct Sections & Visits by Spider
69. mdsimmonds www.DefineMG.com
70. mdsimmonds www.DefineMG.com
71. mdsimmonds www.DefineMG.com Distinct Sections & Visits by Spider
72. mdsimmonds www.DefineMG.com
73. mdsimmonds www.DefineMG.com Search Pages Section Front City Page Home Page International Page
74. mdsimmonds www.DefineMG.com Want to Know How to Cascade PageRank? Section Fronts
75. mdsimmonds www.DefineMG.com Want to Know How to Cascade PageRank? Section Fronts Business Sports Movies Style
76. mdsimmonds www.DefineMG.com The Most Crawled Pages On Your Site
77. mdsimmonds www.DefineMG.com
78. mdsimmonds www.DefineMG.com
79. mdsimmonds www.DefineMG.com Tools
80. mdsimmonds www.DefineMG.com More Tools • ExactTrend.com
81. mdsimmonds www.DefineMG.com Visualization Tools
82. mdsimmonds www.DefineMG.com Takeaways • Crawl Barriers not Budget • Google likes /search • IP Addresses aren’t worth the time for small data sets • There are preferred crawl times, section and pages • Crawl popularity isn’t traffic popularity • Home page activity should be low • GWT will only show you a fraction of activity • Excel is nice but get some tools
83. mdsimmonds www.DefineMG.com Want more? DefineMG.com/blog
84. mdsimmonds www.DefineMG.com
85. Marshall Simmonds Founder / CEO - Define Media Group, Inc. www.DefineMG.com Marshall@DefineMG.com mdsimmonds