Hacker News vs. Slashdot—Reputation Systems in Crowdsourced Technology News

Comparing the reputation systems of Slashdot (slashdot.org) and Hacker News (news.ycombinator.com), highlighting details, differences, pros and cons and presenting possible changes.

Christoph Matthies (@chrisma0), Robert Lehmann (@rlehmann)

Published under CC BY-NC 3.0

Transcript

1. Hacker News vs. Slashdot Reputation Systems in Crowdsourced Technology News Robert Lehmann, Christoph Matthies (@rlehmann) (@chrisma0) Nov 2012
2. 2011 2012 1000 2000 3000 Alexa Daily Traffic Rank slashdot.org ycombinator.com
4. http://xkcd.com/802/ by Randall Munroe (CC BY-NC 2.5)
6. Slashdot • "http colon slash slash slash dot dot org" • founded 1997 by Rob "CmdrTaco" Malda • owned by Dice Holdings (formerly Geeknet) • CMS freely licensed ◦ slashcode.com • 5,300 comments per day • origin of the Slashdot Effect • Wil Wheaton uses it ◦ CleverNickName www.flickr.com/photos/redjar/123436539/ by redjar, 2006 (CC BY-SA 2.0)
7. Slashdot users older average guys alexa.com, Nov 25th 2012 Education Gender Age Has children Browsing Location
8. News for Nerds, Stuff that Matters
9. Comments can be hidden (and are by default)
10. Range: -1 to +5
11. Users Anonymous Submit stories Editor
12. Users Anonymous Submit stories $ $ (paid) Editor
13. Users publish discuss Stories Comments Anonymous Users Editor
14. Users Metamoderators publish discuss Editor evaluate Stories Comments +1, Insightful control Moderators M1 M2 Unfair Anonymous Users
15. Users Moderators granted moderation points Moderators randomly chosen (proportionate to comments): • Logged In User • Regular Slashdot Readers • Long Time Readers • Willing to Serve ('unwilling' not checked) • Positive Contributors (positive karma) Exclusively, participate either as a moderator or a poster in a single discussion Moderation points expire after 3 days
16. Users Moderators M1 evaluate +1, Insightful Result: +1 Result: -1 discuss Comments
17. Metamoderators Users oldest 7.5% of users “ remove bad moderators from the M1 (moderator) eligibility pool and reward good moderators with more delicious mod points — Malda, 2003 ▪ once a day ▪ revoked if abused
18. Metamoderators control M2 Unfair +1, Insightful M1 Ratings Use in an Online Discussion System: The Slashdot Case. Lampe, Clifford A. dissertation, 2006
19. Stories Comments +1 / -1 M1 M2 Unfair metamoderated published moderated + karm a + / - karma - karma (private & normalized) Users
20. Karma determines Comment starting score: ◦ "Anonymous Coward" ◦ default ◦ "Excellent" karma ◦ "Terrible" karma Terrible Bad Neutral Positive Good Excellent -10 -1 0 12 25 Defaults in Slash CMS (slashcode/sql/mysql/defaults.sql) 0 + 1 + 2 - 1
21. User karma not shown in thread. User ID is shown: "CmdrTaco" is #1.
23. • founded 2007 • Paul Graham ◦ of YCombinator fame • 3,300 stories per week • avoid Eternal September à la Reddit • print digest "Hacker Monthly" • written in Arc (pg's Lisp dialect) “ Hacker News If you apply for YC and get rejected, pg [Paul Graham] may follow-up with an explanation just because you have a high score. http://www.quora.com/Hacker-News/How-valuable-is-karma, Jan 13 2013 commons.wikimedia.org/wiki/File:Paulgraham_240x320.jpg by Sarah Harlin, 2007 (pd)
24. Hacker News users basically: yuppies alexa.com, Nov 25th 2012 Education Gender Age Has children Browsing Location
25. “Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
27. Upvotes Upvotes
28. Users publish discuss upvote Stories Comments
29. no duplicate votes once you have voted
30. kinda bookmarks not only endorsements
31. Karma = comment upvotes + story upvotes
32. upwards trend 2009 John Graham-Cumming, http://blog.jgc.org/2009/09/forty-two-days-of-hacker-news-karma.html
33. scale napkin math: 4.4M total karma (AUC) by about 300 users
34. rewards The following limits are adjusted every once in a while.
35. report reply flag inappropriate content (spam or offtopic, for stories too) 200 karma
36. polls 200 karma new story type:
37. 250 karma top color change (yay! \o/)
38. downvotes 500 karma for comments only
39. Top 100 leaderboard
40. comment score below zero ranked down and greyed out (def comment-color (c) (if (> c!score 0) black (grayrange c!score))) default sort order: comment score high average score = better sort order
42. Community Nature Caring Collaborative Cordial Competitive Combative
43. Four design goals for distributed moderation systems Limited influence Timeliness Minimal effort Accuracy
44. Four design goals for distributed moderation systems Limited influence Timeliness Minimal effort Accuracy not considered
45. Four design goals for distributed moderation systems Limited influence Timeliness Minimal effort Accuracy not considered attaching a specific label, only if mod points available simply upvote
46. Four design goals for distributed moderation systems Limited influence Timeliness Minimal effort Accuracy little influence, one of many not considered larger influence, due to only 7 ratings and filter attaching a specific label, only if mod points available simply upvote
47. hard to form (due to randomness of selection), but can demote a comment to the bottom (and zzzing, filtering kicks in!) limited impact if there are enough voters (5 vs 5000) Imagine: five moderator cabal Aside: Limited influence
48. Four design goals for distributed moderation systems Limited influence Timeliness Minimal effort Accuracy little influence, one of many not considered larger influence, due to only 7 ratings and filter no evaluation, opposite vote by other party only evidence suggests: M2 doesn't work optimally attaching a specific label, only if mod points available simply upvote
49. Effectiveness of metamoderation on Slashdot Interpretation: Even if the metamoderation disagreed with the moderation, it did not lead to an immediate reversal Metamoderation's agreement with fairness of moderation % of moderations reversed Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space. Lampe, Resnick, 2004
50. Other considerations display of comment karma trust towards new user display of user karma dealing with "unqualified" comments
51. Other considerations display of comment karma trust towards new user display of user karma dealing with "unqualified" comments no downvoting lower starting comment karma, no moderation at all
52. Other considerations display of comment karma trust towards new user display of user karma dealing with "unqualified" comments numerical total & avg. on profile, Top 100 leaderboard private, only normalized no downvoting lower starting comment karma, no moderation at all
53. Other considerations display of comment karma trust towards new user display of user karma dealing with "unqualified" comments only relative ranking numerical total & avg. on profile, Top 100 leaderboard private, only normalized both numeric value and adjective no downvoting lower starting comment karma, no moderation at all
54. Other considerations display of comment karma trust towards new user display of user karma dealing with "unqualified" comments only relative ranking numerical total & avg. on profile, Top 100 leaderboard private, only normalized both numeric value and adjective greyed out, displayed at bottom, no choice folded (customizable via filter) no downvoting lower starting comment karma, no moderation at all
55. political discussion • Meritocracy? • Cronyism ◦ partiality to long- standing friends • Gerontocracy ◦ rule of the oldest • Demarchy ◦ rule of randomly selected • In-group bias • Meritocracy • Involuntary servitude? they both share: lack of citizen-driven impeachment process
56. buried treasures comments that should have high scores but do not www.flickr.com/photos/pedrovezini/5795835389/ by Pedro Vezini, 2011 (CC BY-NC-SA 2.0)
57. Is this a problem?
58. Metamoderation's agreement with fairness of moderation % of moderations reversed Interpretation: Only about a third of all wrong moderations get reversed. Once your treasure is buried, no one is digging it back up again.
59. How? Attention Deficit • low scoring comments ◦ filters, sorting, masking 30% 29% 25% 9% -1 0 +1 +2 Starting scores % of moderated comments bad karma anon neutral/ good karma excellent karma
60. • low scoring comments ◦ filters, sorting, masking • response comments ◦ moderators lose interest ◦ many more responses 48% moderations received in a thread 22% How? Attention Deficit
61. X II • low scoring comments ◦ filters, sorting, masking • response comments ◦ moderators lose interest ◦ many more responses • late comments ◦ people read fresh news X II How? Attention Deficit X II first ⅕ of comments middle comments last ⅕ of comments 59% moderated 22 minutes 25% moderated 79 minutes 7% moderated 288 minutes median delay of moderation
62. How? Information Cascade Effect Moderators influenced by previous moderations (to remain silent or contribute another moderation in the same direction.) [1] [1] Bikhchandani, S.: A Theory of Fads, Fashion, Custom and Cultural Change as Informational Cascades. (1989) www.flickr.com/photos/vitodens/23506394/ by John Steadman, 2010 (CC BY-NC 2.0)
63. explicitly visible comment scores Information Cascade Effect +3, Insightful as applicable to Slashdot
64. Remember, though All these effects don't have to be negative. e.g. Information Cascade Effect can strengthen good comments and further demote bad ones. Remember, though All these effects don't have to be negative! Yin and Yang, e.g. Information Cascade Effect can strengthen good comments and further demote bad ones. www.flickr.com/photos/bcii/4499830063/ by Juho William Tauriainen, 2010 (CC BY-NC-SA 2.0)
65. Lampe’s Solution [1] Adjust the damn comment filter. -1 0 +5 +4 +3 +1 0 +5 -1 -1 most recent 0 0 -1 0 +5 +4 +3 +1 0 +5 -1 -1 0 0 visible hidden recently written recently moderated [1] Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space (2004, part of his dissertation) www.flickr.com/photos/mr_t_in_dc/4249886990/ by Mr. T in DC, 2010 (CC BY-ND 2.0) after before Lack of moderation indicates comments are okay.
66. Timeliness Minimal effort Limited influence Accuracy The four design goals in Lampe’s proposal Slashdot Lampe’s proposal
67. Lampe’s Solution as applied to Hacker News recent comments missing context
68. Power to the people M2 metamoderators anonymous users paid editors moderators Reversal rate is bad? Make metamoderation more powerful! Timeliness Minimal effort Limited influence Accuracy* * for a biased definition of accuracy www.flickr.com/photos/donshall/1357949216/, Don Shall, 2007 (CC BY-NC-ND 2.0)
69. Encourage constant greatness Spray & Pray works too well? Let karma decay! * * shifting the effort to the broader community Timeliness Minimal effort Limited influence Accuracy www.flickr.com/photos/isthis4real/4150784567/, by A QUIVERFUL OF FOTOS, 2009 (CC BY 2.0)
70. protection mechanisms http://commons.wikimedia.org/wiki/File:Riot_Police.jpg by Chris Brown, 2009 (CC BY 2.0) And now for something completely different...
71. (def vote-for (user i (o dir 'up)) (unless (or ((votes user) i!id) (and (~live i) (isnt user i!by))) (withs (ip (logins* user) vote (list (seconds) ip user dir i!score)) (unless (or (and (or (ignored user) (check-key user 'novote)) (isnt user i!by)) (and (is dir 'down) (~editor user) (or (check-key user 'nodowns) (> (downvote-ratio user) downvote-ratio-limit*) ; prevention of karma-bombing (just-downvoted user i!by))) (and (~legit-user user) (isnt user i!by) (find [is (cadr _) ip] i!votes)) (and (isnt i!type 'pollopt) (biased-voter i vote))) (++ i!score (case dir up 1 down -1)) cheating, the secret sauce • karma bombing prevention ◦ prevent cabal against individual users • downvote ratio ◦ cannot cast too many downvotes • sockpuppet protection [https://github.com/nex3/arc/blob/master/lib/news.arc]
72. if ($constants->{comments_portscan} && ( $constants->{comments_portscan} == 2 || $constants->{comments_portscan} == 1 && $user->{is_anon} ) ) { my $is_trusted = $slashdb->checkAL2($user->{srcids}, 'trusted'); if (!$is_trusted) { my $is_proxy = $slashdb->checkForOpenProxy ($user->{hostip}); if ($is_proxy) { $$error_message = getError('open proxy', { unencoded_ip => $ENV{REMOTE_ADDR}, port => $is_proxy, }); $form_success = 0; return; } } } • HTTP Proxy • maximum number of posts per day • "trollishness" ◦ half-lived amount of downvotes • compresses too well (with Zlib) ◦ ASCII art "nobody should be using that many wide characters in the Standard English alphabet" Anonymous Cowards punished harder. and now a word from CmdrTaco rigorous checks for legit posts [slashcode/Slash/Utility/Comments/Comments.pm]
73. if ($constants->{comments_portscan} && ( $constants->{comments_portscan} == 2 || $constants->{comments_portscan} == 1 && $user->{is_anon} ) ) { my $is_trusted = $slashdb->checkAL2($user->{srcids}, 'trusted'); if (!$is_trusted) { my $is_proxy = $slashdb->checkForOpenProxy ($user->{hostip}); if ($is_proxy) { $$error_message = getError('open proxy', { unencoded_ip => $ENV{REMOTE_ADDR}, port => $is_proxy, }); $form_success = 0; return; } } } and now a word from CmdrTaco • 7 whitespaces in the subject lots of space in the subject ... lots of space in the head. • more than one character one character. Hmmm. Gee, might this be a troll? • words longer than 80 letters that's an awful long string of letters there! • 50% caps PLEASE DON'T USE SO MANY CAPS. USING CAPS IS LIKE YELLING! • "feces", "goatse.cx" too smelly to submit rigorous filters for legit posts Error messages taken from Slashcode [slashcode/sql/mysql/datadump.sql]
74. banhammer or: how to deal with cheating or: how to kill a troll www.flickr.com/photos/dunechaser/6042982955/, Andrew Becraft, 2011 (CC BY-NC-SA 2.0)
75. • Karma: Terrible (-10) • default comment score of -1 • set all comment scores to Offtopic (-1) • Karma: Terrible (-10) • revoke all moderation points warning if a user has more than 10 Karma Use -f to forcibly slap bitchslap modslap
76. nuke blast blast ban the site + kill submission ban user + http://commons.wikimedia.org/wiki/File:HMS_Barham_explodes.jpg, Imperial War Museums (public domain)
77. hellbanning hellbanning • invisible to all other users, but not himself • participating normally, but no one ever responds • don't feed the troll • accounts on bugmenot.com hellbanned “a hellbanned user is likely to get bored or frustrated and leave [...] the silent treatment is the cruelest punishment of them all. [1] — Jeff Atwood [1] http://www.codinghorror.com/blog/2011/06/suspension-ban-or-hellban.html www.flickr.com/photos/ul_marga/455693907/ by Daniele Margaroli, 2007 (CC BY-NC-ND 2.0)
78. smallest system that could possibly work (secret sauce hidden on the inside) many different techniques (to counter specific problems), grown complex system
79. secret sauce secret sauce complex system complex system www.flickr.com/photos/mac_users_guide/3719186889/ by Carl Berkeley, 2009 (CC BY-ND 2.0) www.flickr.com/photos/kim_scarborough/147973192/, by Kim Scarborough,2006 (CC BY-SA 2.0)
80. Thanks for listening, have some xkcd art. http://xkcd.com/301/ by Randall Munroe (CC BY-NC 2.5)
81. Backup slides (additional information)
82. • repost top content from Reddit • tell people about your submission • good grammar and no speling mistakes • repost comments that got good score • exploit (coding) flaws • use lots of sock puppet accounts General cheating (this always works) http://commons.wikimedia.org/wiki/File:Cheating.JPG by Hariadhi (GNU FDL)
83. Gaming the System • subscriptions for money: see each story 10-20 minutes before it goes live. • have a low user ID • promote viewpoints which align with popular sentiment, e.g Lunix [sic] • copy comment from lower down the page • be informative (google, anyone?) • submit stories more than comments (more karma earned on average) • submit stories between 9:00 and 12:00 UTC • make quality comments www.flickr.com/photos/jomenager/7622610418/ by Georges Ménager, 2012 (CC BY-NC-ND 2.0)
84. http://slashdot.org/~Steve%20%27Rim%27%20Jobs/journal/
85. “ if you try cheating, PG himself will come down and smack you on the head, and you sure don't want that! — http://www.kirillzubovsky.com/getting-featured-on-hacker-news
86. unless ($opts{'f'}) { die <<ERR if $user->{karma} > 10; User $_ has a karma of $user->{karma}. Slap aborted. Use -f to forcibly slap. ERR } print "Bitch Slapping #$_...\n"; print "Karma Reduced\n"; print "Default Score Reduced\n"; $slashdb->setUser($_, { -karma => $constants->{badkarma}, -defaultpoints => -1, }); print "ALL live comments moderated down\n\n"; $slashdb->sqlDo("UPDATE comments SET points=-1,reason=1 WHERE uid=$_"); Exhibit A — source of bitchslap (Slashdot)
87. Moderation adjectives Normal The default setting attached to every comment when you have moderation privileges. Offtopic A comment which has nothing to do with the story it's linked to (song lyrics, obscene ascii art, etc). Flamebait Comments whose sole purpose is to insult and enrage. Troll A Troll is similar to Flamebait, but slightly more refined. This is a prank comment intended to provoke indignant (or just confused) responses. Redundant Redundant posts add no new information; they take up space with information either in the original post, the attached links, or lots of previous comments. Insightful An Insightful comment makes you think, or puts a new spin on a given story. Examples: an analogy you hadn't thought of, or a telling counterexample. Interesting If you believe a comment to be Interesting (and on-topic), it is. Informative Informative comments add new information to explain the circumstances hinted at by a particular story, fill in "The Other Side" of an argument, etc. Funny Choose "Funny" if you think the comment is actually funny, not just because it seems intended to be. Overrated Sometimes comments are disproportionately up-moderated—this probably means several moderators saw it at nearly the same time, and their cumulative scores exaggerated its merit. (Example: A knock- knock joke at +5, Funny.) Such a comment is Overrated. Underrated Likewise, some comments get smashed lower than they might deserve. Choosing "Underrated" means you think it should be read by more people.
88. Tweaking options on Slashdot
89. Basic Social Media
90. Submit-Publish
91. Slashdot submission overview
92. Hacker News submission overview
93. Comment overview on Slashdot
94. Hacker News comment overview
95. http://slashdot.org/pollBooth.pl?qid=1805&aid=-1
96. Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space. Lampe, Resnick, 2004
97. Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space. Lampe, Resnick, 2004
98. Distribution of comment scores on Slashdot Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space. Lampe, Resnick, 2004