Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hacker News vs. Slashdot—Reputation Systems in Crowdsourced Technology News

Hacker News vs. Slashdot—Reputation Systems in Crowdsourced Technology News

Comparing the reputation systems of Slashdot (slashdot.org) and Hacker News (news.ycombinator.com), highlighting details, differences, pros and cons and presenting possible changes.

Christoph Matthies (@chrisma0), Robert Lehmann (@rlehmann)

Published under CC BY-NC 3.0

4ab2845fa4f5ab860bc3476c9f11b759?s=128

Christoph Matthies

September 20, 2013
Tweet

More Decks by Christoph Matthies

Other Decks in Technology

Transcript

  1. Hacker News vs. Slashdot Reputation Systems in Crowdsourced Technology News

    Robert Lehmann, Christoph Matthies (@rlehmann) (@chrisma0) Nov 2012
  2. 2011 2012 1000 2000 3000 Alexa Daily Traffic Rank slashdot.org

    ycombinator.com
  3. None
  4. http://xkcd.com/802/ by Randall Munroe (CC BY-NC 2.5)

  5. None
  6. Slashdot • "http colon slash slash slash dot dot org"

    • founded 1997 by Rob "CmdrTaco" Malda • owned by Dice Holdings (formerly Geeknet) • CMS freely licensed ◦ slashcode.com • 5,300 comments per day • origin of the Slashdot Effect • Wil Wheaton uses it ◦ CleverNickName www.flickr.com/photos/redjar/123436539/ by redjar, 2006 (CC BY-SA 2.0)
  7. Slashdot users older average guys alexa.com, Nov 25th 2012 Education

    Gender Age Has children Browsing Location
  8. News for Nerds, Stuff that Matters

  9. Comments can be hidden (and are by default)

  10. Range: -1 to +5

  11. Users Anonymous Submit stories Editor

  12. Users Anonymous Submit stories $ $ (paid) Editor

  13. Users publish discuss Stories Comments Anonymous Users Editor

  14. Users Metamoderators publish discuss Editor evaluate Stories Comments +1, Insightful

    control Moderators M1 M2 Unfair Anonymous Users
  15. Users Moderators granted moderation points Moderators randomly chosen (proportionate to

    comments): • Logged In User • Regular Slashdot Readers • Long Time Readers • Willing to Serve ('unwilling' not checked) • Positive Contributors (positive karma) Exclusively, participate either as a moderator or a poster in a single discussion Moderation points expire after 3 days
  16. Users Moderators M1 evaluate +1, Insightful Result: +1 Result: -1

    discuss Comments
  17. Metamoderators Users oldest 7.5% of users “ remove bad moderators

    from the M1 (moderator) eligibility pool and reward good moderators with more delicious mod points — Malda, 2003 ▪ once a day ▪ revoked if abused
  18. Metamoderators control M2 Unfair +1, Insightful M1 Ratings Use in

    an Online Discussion System: The Slashdot Case. Lampe, Clifford A. dissertation, 2006
  19. Stories Comments +1 / -1 M1 M2 Unfair metamoderated published

    moderated + karm a + / - karma - karma (private & normalized) Users
  20. Karma determines Comment starting score: ◦ "Anonymous Coward" ◦ default

    ◦ "Excellent" karma ◦ "Terrible" karma Terrible Bad Neutral Positive Good Excellent -10 -1 0 12 25 Defaults in Slash CMS (slashcode/sql/mysql/defaults.sql) 0 + 1 + 2 - 1
  21. User karma not shown in thread. User ID is shown:

    "CmdrTaco" is #1.
  22. None
  23. • founded 2007 • Paul Graham ◦ of YCombinator fame

    • 3,300 stories per week • avoid Eternal September à la Reddit • print digest "Hacker Monthly" • written in Arc (pg's Lisp dialect) “ Hacker News If you apply for YC and get rejected, pg [Paul Graham] may follow-up with an explanation just because you have a high score. http://www.quora.com/Hacker-News/How-valuable-is-karma, Jan 13 2013 commons.wikimedia.org/wiki/File:Paulgraham_240x320.jpg by Sarah Harlin, 2007 (pd)
  24. Hacker News users basically: yuppies alexa.com, Nov 25th 2012 Education

    Gender Age Has children Browsing Location
  25. “Anything that good hackers would find interesting. That includes more

    than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
  26. None
  27. Upvotes Upvotes

  28. Users publish discuss upvote Stories Comments

  29. no duplicate votes once you have voted

  30. kinda bookmarks not only endorsements

  31. Karma = comment upvotes + story upvotes

  32. upwards trend 2009 John Graham-Cumming, http://blog.jgc.org/2009/09/forty-two-days-of-hacker-news-karma.html

  33. scale napkin math: 4.4M total karma (AUC) by about 300

    users
  34. rewards The following limits are adjusted every once in a

    while.
  35. report reply flag inappropriate content (spam or offtopic, for stories

    too) 200 karma
  36. polls 200 karma new story type:

  37. 250 karma top color change (yay! \o/)

  38. downvotes 500 karma for comments only

  39. Top 100 leaderboard

  40. comment score below zero ranked down and greyed out (def

    comment-color (c) (if (> c!score 0) black (grayrange c!score))) default sort order: comment score high average score = better sort order
  41. None
  42. Community Nature Caring Collaborative Cordial Competitive Combative

  43. Four design goals for distributed moderation systems Limited influence Timeliness

    Minimal effort Accuracy
  44. Four design goals for distributed moderation systems Limited influence Timeliness

    Minimal effort Accuracy not considered
  45. Four design goals for distributed moderation systems Limited influence Timeliness

    Minimal effort Accuracy not considered attaching a specific label, only if mod points available simply upvote
  46. Four design goals for distributed moderation systems Limited influence Timeliness

    Minimal effort Accuracy little influence, one of many not considered larger influence, due to only 7 ratings and filter attaching a specific label, only if mod points available simply upvote
  47. hard to form (due to randomness of selection), but can

    demote a comment to the bottom (and zzzing, filtering kicks in!) limited impact if there are enough voters (5 vs 5000) Imagine: five moderator cabal Aside: Limited influence
  48. Four design goals for distributed moderation systems Limited influence Timeliness

    Minimal effort Accuracy little influence, one of many not considered larger influence, due to only 7 ratings and filter no evaluation, opposite vote by other party only evidence suggests: M2 doesn't work optimally attaching a specific label, only if mod points available simply upvote
  49. Effectiveness of metamoderation on Slashdot Interpretation: Even if the metamoderation

    disagreed with the moderation, it did not lead to an immediate reversal Metamoderation's agreement with fairness of moderation % of moderations reversed Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space. Lampe, Resnick, 2004
  50. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments
  51. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments no downvoting lower starting comment karma, no moderation at all
  52. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments numerical total & avg. on profile, Top 100 leaderboard private, only normalized no downvoting lower starting comment karma, no moderation at all
  53. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments only relative ranking numerical total & avg. on profile, Top 100 leaderboard private, only normalized both numeric value and adjective no downvoting lower starting comment karma, no moderation at all
  54. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments only relative ranking numerical total & avg. on profile, Top 100 leaderboard private, only normalized both numeric value and adjective greyed out, displayed at bottom, no choice folded (customizable via filter) no downvoting lower starting comment karma, no moderation at all
  55. political discussion • Meritocracy? • Cronyism ◦ partiality to long-

    standing friends • Gerontocracy ◦ rule of the oldest • Demarchy ◦ rule of randomly selected • In-group bias • Meritocracy • Involuntary servitude? they both share: lack of citizen-driven impeachment process
  56. buried treasures comments that should have high scores but do

    not www.flickr.com/photos/pedrovezini/5795835389/ by Pedro Vezini, 2011 (CC BY-NC-SA 2.0)
  57. Is this a problem?

  58. Metamoderation's agreement with fairness of moderation % of moderations reversed

    Interpretation: Only about a third of all wrong moderations get reversed. Once your treasure is buried, no one is digging it back up again.
  59. How? Attention Deficit • low scoring comments ◦ filters, sorting,

    masking 30% 29% 25% 9% -1 0 +1 +2 Starting scores % of moderated comments bad karma anon neutral/ good karma excellent karma
  60. • low scoring comments ◦ filters, sorting, masking • response

    comments ◦ moderators lose interest ◦ many more responses 48% moderations received in a thread 22% How? Attention Deficit
  61. X II • low scoring comments ◦ filters, sorting, masking

    • response comments ◦ moderators lose interest ◦ many more responses • late comments ◦ people read fresh news X II How? Attention Deficit X II first ⅕ of comments middle comments last ⅕ of comments 59% moderated 22 minutes 25% moderated 79 minutes 7% moderated 288 minutes median delay of moderation
  62. How? Information Cascade Effect Moderators influenced by previous moderations (to

    remain silent or contribute another moderation in the same direction.) [1] [1] Bikhchandani, S.: A Theory of Fads, Fashion, Custom and Cultural Change as Informational Cascades. (1989) www.flickr.com/photos/vitodens/23506394/ by John Steadman, 2010 (CC BY-NC 2.0)
  63. explicitly visible comment scores Information Cascade Effect +3, Insightful as

    applicable to Slashdot
  64. Remember, though All these effects don't have to be negative.

    e.g. Information Cascade Effect can strengthen good comments and further demote bad ones. Remember, though All these effects don't have to be negative! Yin and Yang, e.g. Information Cascade Effect can strengthen good comments and further demote bad ones. www.flickr.com/photos/bcii/4499830063/ by Juho William Tauriainen, 2010 (CC BY-NC-SA 2.0)
  65. Lampe’s Solution [1] Adjust the damn comment filter. -1 0

    +5 +4 +3 +1 0 +5 -1 -1 most recent 0 0 -1 0 +5 +4 +3 +1 0 +5 -1 -1 0 0 visible hidden recently written recently moderated [1] Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space (2004, part of his dissertation) www.flickr.com/photos/mr_t_in_dc/4249886990/ by Mr. T in DC, 2010 (CC BY-ND 2.0) after before Lack of moderation indicates comments are okay.
  66. Timeliness Minimal effort Limited influence Accuracy The four design goals

    in Lampe’s proposal Slashdot Lampe’s proposal
  67. Lampe’s Solution as applied to Hacker News recent comments missing

    context
  68. Power to the people M2 metamoderators anonymous users paid editors

    moderators Reversal rate is bad? Make metamoderation more powerful! Timeliness Minimal effort Limited influence Accuracy* * for a biased definition of accuracy www.flickr.com/photos/donshall/1357949216/, Don Shall, 2007 (CC BY-NC-ND 2.0)
  69. Encourage constant greatness Spray & Pray works too well? Let

    karma decay! * * shifting the effort to the broader community Timeliness Minimal effort Limited influence Accuracy www.flickr.com/photos/isthis4real/4150784567/, by A QUIVERFUL OF FOTOS, 2009 (CC BY 2.0)
  70. protection mechanisms http://commons.wikimedia.org/wiki/File:Riot_Police.jpg by Chris Brown, 2009 (CC BY 2.0)

    And now for something completely different...
  71. (def vote-for (user i (o dir 'up)) (unless (or ((votes

    user) i!id) (and (~live i) (isnt user i!by))) (withs (ip (logins* user) vote (list (seconds) ip user dir i!score)) (unless (or (and (or (ignored user) (check-key user 'novote)) (isnt user i!by)) (and (is dir 'down) (~editor user) (or (check-key user 'nodowns) (> (downvote-ratio user) downvote-ratio-limit*) ; prevention of karma-bombing (just-downvoted user i!by))) (and (~legit-user user) (isnt user i!by) (find [is (cadr _) ip] i!votes)) (and (isnt i!type 'pollopt) (biased-voter i vote))) (++ i!score (case dir up 1 down -1)) cheating, the secret sauce • karma bombing prevention ◦ prevent cabal against individual users • downvote ratio ◦ cannot cast too many downvotes • sockpuppet protection [https://github.com/nex3/arc/blob/master/lib/news.arc]
  72. if ($constants->{comments_portscan} && ( $constants->{comments_portscan} == 2 || $constants->{comments_portscan} ==

    1 && $user->{is_anon} ) ) { my $is_trusted = $slashdb->checkAL2($user->{srcids}, 'trusted'); if (!$is_trusted) { my $is_proxy = $slashdb->checkForOpenProxy ($user->{hostip}); if ($is_proxy) { $$error_message = getError('open proxy', { unencoded_ip => $ENV{REMOTE_ADDR}, port => $is_proxy, }); $form_success = 0; return; } } } • HTTP Proxy • maximum number of posts per day • "trollishness" ◦ half-lived amount of downvotes • compresses too well (with Zlib) ◦ ASCII art "nobody should be using that many wide characters in the Standard English alphabet" Anonymous Cowards punished harder. and now a word from CmdrTaco rigorous checks for legit posts [slashcode/Slash/Utility/Comments/Comments.pm]
  73. if ($constants->{comments_portscan} && ( $constants->{comments_portscan} == 2 || $constants->{comments_portscan} ==

    1 && $user->{is_anon} ) ) { my $is_trusted = $slashdb->checkAL2($user->{srcids}, 'trusted'); if (!$is_trusted) { my $is_proxy = $slashdb->checkForOpenProxy ($user->{hostip}); if ($is_proxy) { $$error_message = getError('open proxy', { unencoded_ip => $ENV{REMOTE_ADDR}, port => $is_proxy, }); $form_success = 0; return; } } } and now a word from CmdrTaco • 7 whitespaces in the subject lots of space in the subject ... lots of space in the head. • more than one character one character. Hmmm. Gee, might this be a troll? • words longer than 80 letters that's an awful long string of letters there! • 50% caps PLEASE DON'T USE SO MANY CAPS. USING CAPS IS LIKE YELLING! • "feces", "goatse.cx" too smelly to submit rigorous filters for legit posts Error messages taken from Slashcode [slashcode/sql/mysql/datadump.sql]
  74. banhammer or: how to deal with cheating or: how to

    kill a troll www.flickr.com/photos/dunechaser/6042982955/, Andrew Becraft, 2011 (CC BY-NC-SA 2.0)
  75. • Karma: Terrible (-10) • default comment score of -1

    • set all comment scores to Offtopic (-1) • Karma: Terrible (-10) • revoke all moderation points warning if a user has more than 10 Karma Use -f to forcibly slap bitchslap modslap
  76. nuke blast blast ban the site + kill submission ban

    user + http://commons.wikimedia.org/wiki/File:HMS_Barham_explodes.jpg, Imperial War Museums (public domain)
  77. hellbanning hellbanning • invisible to all other users, but not

    himself • participating normally, but no one ever responds • don't feed the troll • accounts on bugmenot.com hellbanned “a hellbanned user is likely to get bored or frustrated and leave [...] the silent treatment is the cruelest punishment of them all. [1] — Jeff Atwood [1] http://www.codinghorror.com/blog/2011/06/suspension-ban-or-hellban.html www.flickr.com/photos/ul_marga/455693907/ by Daniele Margaroli, 2007 (CC BY-NC-ND 2.0)
  78. smallest system that could possibly work (secret sauce hidden on

    the inside) many different techniques (to counter specific problems), grown complex system
  79. secret sauce secret sauce complex system complex system www.flickr.com/photos/mac_users_guide/3719186889/ by

    Carl Berkeley, 2009 (CC BY-ND 2.0) www.flickr.com/photos/kim_scarborough/147973192/, by Kim Scarborough,2006 (CC BY-SA 2.0)
  80. Thanks for listening, have some xkcd art. http://xkcd.com/301/ by Randall

    Munroe (CC BY-NC 2.5)
  81. Backup slides (additional information)

  82. • repost top content from Reddit • tell people about

    your submission • good grammar and no speling mistakes • repost comments that got good score • exploit (coding) flaws • use lots of sock puppet accounts General cheating (this always works) http://commons.wikimedia.org/wiki/File:Cheating.JPG by Hariadhi (GNU FDL)
  83. Gaming the System • subscriptions for money: see each story

    10-20 minutes before it goes live. • have a low user ID • promote viewpoints which align with popular sentiment, e.g Lunix [sic] • copy comment from lower down the page • be informative (google, anyone?) • submit stories more than comments (more karma earned on average) • submit stories between 9:00 and 12:00 UTC • make quality comments www.flickr.com/photos/jomenager/7622610418/ by Georges Ménager, 2012 (CC BY-NC-ND 2.0)
  84. http://slashdot.org/~Steve%20%27Rim%27%20Jobs/journal/

  85. “ if you try cheating, PG himself will come down

    and smack you on the head, and you sure don't want that! — http://www.kirillzubovsky.com/getting-featured-on-hacker-news
  86. unless ($opts{'f'}) { die <<ERR if $user->{karma} > 10; User

    $_ has a karma of $user->{karma}. Slap aborted. Use -f to forcibly slap. ERR } print "Bitch Slapping #$_...\n"; print "Karma Reduced\n"; print "Default Score Reduced\n"; $slashdb->setUser($_, { -karma => $constants->{badkarma}, -defaultpoints => -1, }); print "ALL live comments moderated down\n\n"; $slashdb->sqlDo("UPDATE comments SET points=-1,reason=1 WHERE uid=$_"); Exhibit A — source of bitchslap (Slashdot)
  87. Moderation adjectives Normal The default setting attached to every comment

    when you have moderation privileges. Offtopic A comment which has nothing to do with the story it's linked to (song lyrics, obscene ascii art, etc). Flamebait Comments whose sole purpose is to insult and enrage. Troll A Troll is similar to Flamebait, but slightly more refined. This is a prank comment intended to provoke indignant (or just confused) responses. Redundant Redundant posts add no new information; they take up space with information either in the original post, the attached links, or lots of previous comments. Insightful An Insightful comment makes you think, or puts a new spin on a given story. Examples: an analogy you hadn't thought of, or a telling counterexample. Interesting If you believe a comment to be Interesting (and on-topic), it is. Informative Informative comments add new information to explain the circumstances hinted at by a particular story, fill in "The Other Side" of an argument, etc. Funny Choose "Funny" if you think the comment is actually funny, not just because it seems intended to be. Overrated Sometimes comments are disproportionately up-moderated—this probably means several moderators saw it at nearly the same time, and their cumulative scores exaggerated its merit. (Example: A knock- knock joke at +5, Funny.) Such a comment is Overrated. Underrated Likewise, some comments get smashed lower than they might deserve. Choosing "Underrated" means you think it should be read by more people.
  88. Tweaking options on Slashdot

  89. Basic Social Media

  90. Submit-Publish

  91. Slashdot submission overview

  92. Hacker News submission overview

  93. Comment overview on Slashdot

  94. Hacker News comment overview

  95. http://slashdot.org/pollBooth.pl?qid=1805&aid=-1

  96. Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation

    Space. Lampe, Resnick, 2004
  97. Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation

    Space. Lampe, Resnick, 2004
  98. Distribution of comment scores on Slashdot Slash(dot) and Burn: Distributed

    Moderation in a Large Online Conversation Space. Lampe, Resnick, 2004