Rand Fishkin - Why Great Marketers Must Be Great Skeptics

The world of web marketing is filled with common wisdom, best practices, and tactics based on past experiences, but some of these industry standards may actually be holding you back. Watch as Rand dives into stories and examples of those who've broken patterns and found value through counterintuitive marketing. This presentation will cover a wide variety of marketing channels, including SEO, social media, content marketing, CRO, and more.

Transcript

1. Rand Fishkin, Wizard of Moz | @randfish | rand@moz.com Why Great Marketers Must Be Great Skeptics
2. This Presentation Is Online Here: bit.ly/mozskeptics
3. Great Skepticism Defining
4. I have some depressing news…
7. Does anyone in this room believe that the Earth doesn’t revolve around the Sun?
9. The Earth (and everything in the solar system, including the Sun) revolves around our system’s gravitational Barycenter, which is only sometimes near the center of the Sun.
10. Let’s try a more marketing-centric example...
11. In 2009, Conversion Rate Experts built us a new landing page, and increased our subscribers by nearly 25%. What did they do? Via CRE’s Case Study
12. One of the most commonly cited facts about CRE’s work is the “long landing page.”
14. The Crap Skeptic Let’s change our landing page to be a long one right now!
15. The Crap Skeptic The Good Skeptic Let’s change our landing page to be a long one right now! We should A/B test a long landing page in our conversion funnel.
16. The Crap Skeptic The Good Skeptic The Great Skeptic Let’s change our landing page to be a long one right now! We should A/B test a long landing page in our conversion funnel. How do we know page length was responsible? What else changed?
17. The Crap Skeptic The Good Skeptic The Great Skeptic
18. The Crap Skeptic The Good Skeptic The Great Skeptic “I do believe sadly it’s going to take some diseases coming back to realize that we need to change and develop vaccines that are safe.”
19. The Crap Skeptic The Good Skeptic The Great Skeptic “I do believe sadly it’s going to take some diseases coming back to realize that we need to change and develop vaccines that are safe.” “Listen, all magic is scientific principals presented like "mystical hoodoo" which is fun, but it's sort of irresponsible.”
20. The Crap Skeptic The Good Skeptic The Great Skeptic “I do believe sadly it’s going to take some diseases coming back to realize that we need to change and develop vaccines that are safe.” “Listen, all magic is scientific principals presented like "mystical hoodoo" which is fun, but it's sort of irresponsible.” "The good thing about science is that it's true whether or not you believe in it."
21. In fact, we’ve changed our landing pages numerous times to shorter versions and seen equal success. Length, it would seem, was not the primary factor in this page’s success.
22. What separates the crap, good, & great?
23. Assumes one belief-reinforcing data point is evidence enough Doesn’t question what’s truly causal vs. merely correlated Doesn’t seek to validate
24. Doesn’t make assumptions about why a result occurred Knows that correlation isn’t necessarily causal Validates assumptions w/ data
25. Seeks to discover the reasons underlying the results Knows that correlation doesn’t imply causality Thoroughly validates, but doesn’t let imperfect knowledge stop progress
26. Will more conversion tests lead to better results? Testing
27. Obviously the more tests we run, the better we can optimize our pages. We need to build a “culture of testing” around here.
28. Via Wordstream’s What is a Good Conversion Rate?
29. Via Wordstream’s What is a Good Conversion Rate? Do Those Who Test More Really Perform Better?
30. Hmm… There’s no correlation between those who run more tests across more pages and those who have higher conversion rates. Maybe the number of tests isn’t the right goal.
31. Via Factors That Drive How Quickly You Can Run New Online Tests
32. CONVERSION DECISION (it’s a complex process)
33. Trust Word of Mouth Likability Design Associations Word of Mouth Amount of Pain CTAs UX Effort Required Process Historical Experiences Social Proof Copywriting CONVERSION DECISION Timing Discovery Path Branding Price (it’s a complex process)
34. How do we know where our conversion problems lie?
35. Ask Smart Questions to the Right People Potential Customers Who Didn’t Buy Those Who Tried/Bought But Didn’t Love It Customers Who Bought & Loved It Professional, demographic, & psychographic characteristics Professional, demographic, & psychographic characteristics Professional, demographic, & psychographic characteristics What objections did you have to buying? What objections did you have; how did you overcome them? What objections did you overcome; how? What would have made you stay/love the product? What would have made you overcome them? What do you love most? Can we share?
36. We can start by targeting the right kinds of customers. Trying to please everyone is a recipe for disaster.
37. Our tests should be focused around overcoming the objections of the people who best match our customer profiles
38. Testing button colors
39. Testing headlines, copy, visuals, & form fields
40. Designing for how customers think about their problems & your solution
42. THIS!
43. Does telling users we encrypt data scare them? Security
44. Via Visual Website Optimizer Could this actually HURT conversion?
45. Via Visual Website Optimizer
46. Via Visual Website Optimizer A/B Test Results They found that without the secure icon had over 400% improvement on conversions as compared to having the image. [Note: results ARE statistically significant]
47. We need to remove the security messages on our site ASAP!
48. We should test this.
49. Is this the most meaningful test we can perform right now? (I’m not saying it isn’t, just that we should prioritize intelligently)
50. Via Kayak’s Most Interesting A/B Test vs.
51. Via Kayak’s Most Interesting A/B Test A/B Test Results “So we decided to do our own experiment about this and we actually found the opposite that when we removed the messaging, people tended to book less.” - Vinayak Ranade, Director of Engineering for Mobile, KAYAK
52. Good thing we tested! Good thing we tested! Your evidence is no match for my ignorance!
53. What should we expect from sharing our content on social media? Social CTR
54. Just find the average social CTRs and then try to match them or do better. No brainer.
55. Via Signup.to’s Analysis of CTR on Twitter
56. Via Signup.to’s Analysis of CTR on Twitter
59. 306/701 = 43.6%... WTF??
61. Phew! We’re not alone. Via Chartbeat
62. Assuming social metrics and engagement correlate was a flawed assumption. We need to find a better way to measure and improve social sharing.
66. OK. We can create some benchmarks based on these numbers and their averages, then work to improve them over time.
67. That is an insane amount of variability!
68. There are other factors at work here. We need to understand them before we can create smart metrics or useful expectations
69. Reach Traffic Engagement
70. Timing Source Audience Affinity Formatting Network-Created Limitations to Visibility Brand Reach Traffic Engagement
71. Let’s start by examining the data and impacts of timing.
72. Via Facebook Insights
73. Via Followerwonk
74. Via Google Analytics
75. There’s a lot of nuance, but we can certainly see how messages sent at certain times reach different sizes and populations of our audience.
76. Comparing a tweet or share sent at 9am Pacific against tweets and shares sent at 11pm Pacific will give us misleading data.
77. But, we now know three things:
78. But, we now know three things: #1 - When our audience is online
79. But, we now know three things: #1 - When our audience is online #2 – Sharing just once is suboptimal
80. But, we now know three things: #1 - When our audience is online #2 – Sharing just once is suboptimal #3 – To be a great skeptic (and marketer), we should attempt to understand each of these inputs with similar rigorousness
81. Do they work? Can we make them more effective? Share Buttons
82. After relentless testing, OKTrends found that the following share buttons worked best:
85. OKTrends found that removing all but a single button (the “like” on Facebook) had the most positive effect.
86. And that waiting until the visitor had scrolled to the bottom of the article produced the highest number of actions
87. We should remove all our social sharing buttons and replace them with a single slide-over social CTA for Facebook likes!
88. Buzzfeed has also done a tremendous amount of social button testing & optimization…
89. And sometimes they do this…
90. And sometimes this…
91. Is Buzzfeed still in testing mode?
92. Nope. They’ve found it’s best to show different buttons based on both the type of content and how you reached the site.
93. OK… Well, then let’s do that… Do it now!
94. Testing a small number of the most impactful social button changes should produce enough evidence to give us a direction to pursue.
95. Buzzfeed & OKTrends share several unique qualities: 1) They have huge amounts of social traffic 2) Social shares are integral to their business model 3) The content they create is optimized for social sharing
96. Unless we also fit a number of these criteria, I have to ask again: Is this the most meaningful test we can perform right now?
97. BTW – it is true that testing social buttons can coincide with a lot of other tests (since it’s on content vs. the funnel), but dev resources and marketing bandwidth probably are not infinite 
98. Does it still work better than standard link text? Anchor Text
99. Psh. Anchor text links obviously work. Otherwise Google wouldn’t be penalizing all these sites for getting them.
100. It has been a while since we’ve seen a public test of anchor text. And there’s no way to know for sure how powerful it still is.
101. Testing in Google is very, very hard. There’s so many confounding variables – we’d have to choose our criteria carefully and repeat the test multiple times to feel confident of any result.
102. 1) Three word, informational keyword phrase with relatively light competition and stable rankings Test Conditions: 2) We selected two results (“A” and “B”), ranking #13 (“A”) and #20 ( “B”) in logged-out, non-personalized results 3) We pointed links from 20 pages on 20 unique, high-DA, high- trust, off-topic sites at both “A” and “B”
103. A) We pointed 20 links from 20 domains at this result with anchor text exactly matching the query phrase #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 B) We pointed 20 links from the same 20 pages as “A” to this URL with anchor text that did not contain any words in the query
104. #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 After 20 days, all of the links had been indexed by Google. “A” and “B” both moved up 4 positions. None of the other results moved more than 2 positions.
105. See? Told you it works.
106. While both results moved up the same number of positions, it’s almost certainly the case that #13 to #9 was against more serious challengers, and thus anchor text would seem to make a difference. That said, I’d want to repeat this a few times.
107. Princess Bubblegum and I are in agreement. We should do the test at least 2-3 more times keeping as many variables as possible the same.
108. 1) Three word, informational keyword phrase with relatively light competition and stable rankings Early Results from a Second Test: 2) We selected two results (“A” and “B”), ranking #20 (“A”) and #14 ( “B”) in logged-out, non-personalized results 3) We pointed links from 20 pages on 20 unique, high-DA, high- trust, off-topic sites at both “A” and “B”
109. B) We pointed 20 links from 20 domains to this URL with anchor text that did not contain any words in the query #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 A) We pointed 20 links from the same pages/domains at this result with anchor text exactly matching the query phrase
110. #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 After 16 days, all of the links had been indexed by Google. “A” moved up 19 positions to #1! B moved up 5 positions to #9. None of the other results moved more than 2 positions.
111. Good thing we tested! This is looking more conclusive, but we should run at least one more test. Anchor text = rankings. Stick a fork in it!
112. Does it influence Google’s non- personalized search rankings? Google+
113. Good discussion about Google+ correlations in this post Google+ is just too damn high.
114. Good discussion about Google+ correlations in this post From a comment Matt Cutts left on the blog post: “Most of the initial discussion on this thread seemed to take from the blog post the idea that more Google +1s led to higher web ranking. I wanted to preemptively tackle that perception.”
115. Good discussion about Google+ correlations in this post To me, that’s Google working really hard to NOT say “we don’t use any data from Google+ (directly or indirectly) at all in our ranking algorithms.” I would be very surprised if they said that.
116. Google explicitly SAID +1s don’t affect rankings. You think they’d lie so blatantly? As if.
117. The correlations are surprisingly high for something with no connection. There have been several tests showing no result, but if all it takes is a Google+ post, let’s do it!
118. First, remember how hard it is to prove causality with a public test like this. And second, don’t let anything but consistent, repeatable, provable results sway your opinion.
120. #21 #22 #23 #24 #25 #26 At 10:50am, the test URL ranked #26 in logged-out, non- personalized, non-geo-biased, Google US results.
121. 42 minutes later, after ~30 shares, 40 +1s, and several other G+ accounts posting the link, the target moved up to position #23 #21 #22 #23 #24 #25 #26
122. #21 #22 #23 #24 #25 #26 48 hours later, after 100 shares of the post, 95 +1s, and tons of additional posts, the result was back down to #25
123. At least we proved one thing – the Google+ community is awesome. Nearly 50 people shared the URL in their own posts on G+!
124. Many G+ users personalized results, however, were clearly affected.
125. #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 Something very strange is happening in relation to the test URL in my personalized results, though. It’s actually ranking LOWER than in non-personalized results.
126. Could Google be donking up the test? Sadly, it’s impossible to know.
127. GASP!!! The posts did move the result up, then someone from Google must have seen it and is messing with you!!!
128. Sigh… It’s possible that Jenny’s right, but impossible to prove. We don’t know for sure what caused the initial movement, nor can we say what’s causing the weird personalized results.
129. More testing is needed, but how you do it without any potential monkey wrenches is going to be a big challenge. That said, remember this:
130. Phew! We’re not alone. Via Chartbeat
131. If I were Google, I wouldn’t use Google+ activity by itself to rank anything, but I would connect G+ to my other data sources and potentially increase a page’s rankings if many pieces of data told a story of engagement & value for visitors.
132. Ready to Be Your Own Skeptic?
133. Rand Fishkin, Wizard of Moz | @randfish | rand@moz.com bit.ly/mozskeptics