Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hacker News vs. Slashdot—Reputation Systems in Crowdsourced Technology News

Hacker News vs. Slashdot—Reputation Systems in Crowdsourced Technology News

Comparing the reputation systems of Slashdot (slashdot.org) and Hacker News (news.ycombinator.com), highlighting details, differences, pros and cons and presenting possible changes.

Christoph Matthies (@chrisma0), Robert Lehmann (@rlehmann)

Published under CC BY-NC 3.0

Christoph Matthies

September 20, 2013
Tweet

More Decks by Christoph Matthies

Other Decks in Technology

Transcript

  1. Hacker News vs. Slashdot Reputation Systems in Crowdsourced Technology News

    Robert Lehmann, Christoph Matthies (@rlehmann) (@chrisma0) Nov 2012
  2. Slashdot • "http colon slash slash slash dot dot org"

    • founded 1997 by Rob "CmdrTaco" Malda • owned by Dice Holdings (formerly Geeknet) • CMS freely licensed ◦ slashcode.com • 5,300 comments per day • origin of the Slashdot Effect • Wil Wheaton uses it ◦ CleverNickName www.flickr.com/photos/redjar/123436539/ by redjar, 2006 (CC BY-SA 2.0)
  3. Users Moderators granted moderation points Moderators randomly chosen (proportionate to

    comments): • Logged In User • Regular Slashdot Readers • Long Time Readers • Willing to Serve ('unwilling' not checked) • Positive Contributors (positive karma) Exclusively, participate either as a moderator or a poster in a single discussion Moderation points expire after 3 days
  4. Metamoderators Users oldest 7.5% of users “ remove bad moderators

    from the M1 (moderator) eligibility pool and reward good moderators with more delicious mod points — Malda, 2003 ▪ once a day ▪ revoked if abused
  5. Metamoderators control M2 Unfair +1, Insightful M1 Ratings Use in

    an Online Discussion System: The Slashdot Case. Lampe, Clifford A. dissertation, 2006
  6. Stories Comments +1 / -1 M1 M2 Unfair metamoderated published

    moderated + karm a + / - karma - karma (private & normalized) Users
  7. Karma determines Comment starting score: ◦ "Anonymous Coward" ◦ default

    ◦ "Excellent" karma ◦ "Terrible" karma Terrible Bad Neutral Positive Good Excellent -10 -1 0 12 25 Defaults in Slash CMS (slashcode/sql/mysql/defaults.sql) 0 + 1 + 2 - 1
  8. • founded 2007 • Paul Graham ◦ of YCombinator fame

    • 3,300 stories per week • avoid Eternal September à la Reddit • print digest "Hacker Monthly" • written in Arc (pg's Lisp dialect) “ Hacker News If you apply for YC and get rejected, pg [Paul Graham] may follow-up with an explanation just because you have a high score. http://www.quora.com/Hacker-News/How-valuable-is-karma, Jan 13 2013 commons.wikimedia.org/wiki/File:Paulgraham_240x320.jpg by Sarah Harlin, 2007 (pd)
  9. “Anything that good hackers would find interesting. That includes more

    than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
  10. comment score below zero ranked down and greyed out (def

    comment-color (c) (if (> c!score 0) black (grayrange c!score))) default sort order: comment score high average score = better sort order
  11. Four design goals for distributed moderation systems Limited influence Timeliness

    Minimal effort Accuracy not considered attaching a specific label, only if mod points available simply upvote
  12. Four design goals for distributed moderation systems Limited influence Timeliness

    Minimal effort Accuracy little influence, one of many not considered larger influence, due to only 7 ratings and filter attaching a specific label, only if mod points available simply upvote
  13. hard to form (due to randomness of selection), but can

    demote a comment to the bottom (and zzzing, filtering kicks in!) limited impact if there are enough voters (5 vs 5000) Imagine: five moderator cabal Aside: Limited influence
  14. Four design goals for distributed moderation systems Limited influence Timeliness

    Minimal effort Accuracy little influence, one of many not considered larger influence, due to only 7 ratings and filter no evaluation, opposite vote by other party only evidence suggests: M2 doesn't work optimally attaching a specific label, only if mod points available simply upvote
  15. Effectiveness of metamoderation on Slashdot Interpretation: Even if the metamoderation

    disagreed with the moderation, it did not lead to an immediate reversal Metamoderation's agreement with fairness of moderation % of moderations reversed Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space. Lampe, Resnick, 2004
  16. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments
  17. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments no downvoting lower starting comment karma, no moderation at all
  18. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments numerical total & avg. on profile, Top 100 leaderboard private, only normalized no downvoting lower starting comment karma, no moderation at all
  19. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments only relative ranking numerical total & avg. on profile, Top 100 leaderboard private, only normalized both numeric value and adjective no downvoting lower starting comment karma, no moderation at all
  20. Other considerations display of comment karma trust towards new user

    display of user karma dealing with "unqualified" comments only relative ranking numerical total & avg. on profile, Top 100 leaderboard private, only normalized both numeric value and adjective greyed out, displayed at bottom, no choice folded (customizable via filter) no downvoting lower starting comment karma, no moderation at all
  21. political discussion • Meritocracy? • Cronyism ◦ partiality to long-

    standing friends • Gerontocracy ◦ rule of the oldest • Demarchy ◦ rule of randomly selected • In-group bias • Meritocracy • Involuntary servitude? they both share: lack of citizen-driven impeachment process
  22. buried treasures comments that should have high scores but do

    not www.flickr.com/photos/pedrovezini/5795835389/ by Pedro Vezini, 2011 (CC BY-NC-SA 2.0)
  23. Metamoderation's agreement with fairness of moderation % of moderations reversed

    Interpretation: Only about a third of all wrong moderations get reversed. Once your treasure is buried, no one is digging it back up again.
  24. How? Attention Deficit • low scoring comments ◦ filters, sorting,

    masking 30% 29% 25% 9% -1 0 +1 +2 Starting scores % of moderated comments bad karma anon neutral/ good karma excellent karma
  25. • low scoring comments ◦ filters, sorting, masking • response

    comments ◦ moderators lose interest ◦ many more responses 48% moderations received in a thread 22% How? Attention Deficit
  26. X II • low scoring comments ◦ filters, sorting, masking

    • response comments ◦ moderators lose interest ◦ many more responses • late comments ◦ people read fresh news X II How? Attention Deficit X II first ⅕ of comments middle comments last ⅕ of comments 59% moderated 22 minutes 25% moderated 79 minutes 7% moderated 288 minutes median delay of moderation
  27. How? Information Cascade Effect Moderators influenced by previous moderations (to

    remain silent or contribute another moderation in the same direction.) [1] [1] Bikhchandani, S.: A Theory of Fads, Fashion, Custom and Cultural Change as Informational Cascades. (1989) www.flickr.com/photos/vitodens/23506394/ by John Steadman, 2010 (CC BY-NC 2.0)
  28. Remember, though All these effects don't have to be negative.

    e.g. Information Cascade Effect can strengthen good comments and further demote bad ones. Remember, though All these effects don't have to be negative! Yin and Yang, e.g. Information Cascade Effect can strengthen good comments and further demote bad ones. www.flickr.com/photos/bcii/4499830063/ by Juho William Tauriainen, 2010 (CC BY-NC-SA 2.0)
  29. Lampe’s Solution [1] Adjust the damn comment filter. -1 0

    +5 +4 +3 +1 0 +5 -1 -1 most recent 0 0 -1 0 +5 +4 +3 +1 0 +5 -1 -1 0 0 visible hidden recently written recently moderated [1] Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space (2004, part of his dissertation) www.flickr.com/photos/mr_t_in_dc/4249886990/ by Mr. T in DC, 2010 (CC BY-ND 2.0) after before Lack of moderation indicates comments are okay.
  30. Timeliness Minimal effort Limited influence Accuracy The four design goals

    in Lampe’s proposal Slashdot Lampe’s proposal
  31. Power to the people M2 metamoderators anonymous users paid editors

    moderators Reversal rate is bad? Make metamoderation more powerful! Timeliness Minimal effort Limited influence Accuracy* * for a biased definition of accuracy www.flickr.com/photos/donshall/1357949216/, Don Shall, 2007 (CC BY-NC-ND 2.0)
  32. Encourage constant greatness Spray & Pray works too well? Let

    karma decay! * * shifting the effort to the broader community Timeliness Minimal effort Limited influence Accuracy www.flickr.com/photos/isthis4real/4150784567/, by A QUIVERFUL OF FOTOS, 2009 (CC BY 2.0)
  33. (def vote-for (user i (o dir 'up)) (unless (or ((votes

    user) i!id) (and (~live i) (isnt user i!by))) (withs (ip (logins* user) vote (list (seconds) ip user dir i!score)) (unless (or (and (or (ignored user) (check-key user 'novote)) (isnt user i!by)) (and (is dir 'down) (~editor user) (or (check-key user 'nodowns) (> (downvote-ratio user) downvote-ratio-limit*) ; prevention of karma-bombing (just-downvoted user i!by))) (and (~legit-user user) (isnt user i!by) (find [is (cadr _) ip] i!votes)) (and (isnt i!type 'pollopt) (biased-voter i vote))) (++ i!score (case dir up 1 down -1)) cheating, the secret sauce • karma bombing prevention ◦ prevent cabal against individual users • downvote ratio ◦ cannot cast too many downvotes • sockpuppet protection [https://github.com/nex3/arc/blob/master/lib/news.arc]
  34. if ($constants->{comments_portscan} && ( $constants->{comments_portscan} == 2 || $constants->{comments_portscan} ==

    1 && $user->{is_anon} ) ) { my $is_trusted = $slashdb->checkAL2($user->{srcids}, 'trusted'); if (!$is_trusted) { my $is_proxy = $slashdb->checkForOpenProxy ($user->{hostip}); if ($is_proxy) { $$error_message = getError('open proxy', { unencoded_ip => $ENV{REMOTE_ADDR}, port => $is_proxy, }); $form_success = 0; return; } } } • HTTP Proxy • maximum number of posts per day • "trollishness" ◦ half-lived amount of downvotes • compresses too well (with Zlib) ◦ ASCII art "nobody should be using that many wide characters in the Standard English alphabet" Anonymous Cowards punished harder. and now a word from CmdrTaco rigorous checks for legit posts [slashcode/Slash/Utility/Comments/Comments.pm]
  35. if ($constants->{comments_portscan} && ( $constants->{comments_portscan} == 2 || $constants->{comments_portscan} ==

    1 && $user->{is_anon} ) ) { my $is_trusted = $slashdb->checkAL2($user->{srcids}, 'trusted'); if (!$is_trusted) { my $is_proxy = $slashdb->checkForOpenProxy ($user->{hostip}); if ($is_proxy) { $$error_message = getError('open proxy', { unencoded_ip => $ENV{REMOTE_ADDR}, port => $is_proxy, }); $form_success = 0; return; } } } and now a word from CmdrTaco • 7 whitespaces in the subject lots of space in the subject ... lots of space in the head. • more than one character one character. Hmmm. Gee, might this be a troll? • words longer than 80 letters that's an awful long string of letters there! • 50% caps PLEASE DON'T USE SO MANY CAPS. USING CAPS IS LIKE YELLING! • "feces", "goatse.cx" too smelly to submit rigorous filters for legit posts Error messages taken from Slashcode [slashcode/sql/mysql/datadump.sql]
  36. banhammer or: how to deal with cheating or: how to

    kill a troll www.flickr.com/photos/dunechaser/6042982955/, Andrew Becraft, 2011 (CC BY-NC-SA 2.0)
  37. • Karma: Terrible (-10) • default comment score of -1

    • set all comment scores to Offtopic (-1) • Karma: Terrible (-10) • revoke all moderation points warning if a user has more than 10 Karma Use -f to forcibly slap bitchslap modslap
  38. nuke blast blast ban the site + kill submission ban

    user + http://commons.wikimedia.org/wiki/File:HMS_Barham_explodes.jpg, Imperial War Museums (public domain)
  39. hellbanning hellbanning • invisible to all other users, but not

    himself • participating normally, but no one ever responds • don't feed the troll • accounts on bugmenot.com hellbanned “a hellbanned user is likely to get bored or frustrated and leave [...] the silent treatment is the cruelest punishment of them all. [1] — Jeff Atwood [1] http://www.codinghorror.com/blog/2011/06/suspension-ban-or-hellban.html www.flickr.com/photos/ul_marga/455693907/ by Daniele Margaroli, 2007 (CC BY-NC-ND 2.0)
  40. smallest system that could possibly work (secret sauce hidden on

    the inside) many different techniques (to counter specific problems), grown complex system
  41. secret sauce secret sauce complex system complex system www.flickr.com/photos/mac_users_guide/3719186889/ by

    Carl Berkeley, 2009 (CC BY-ND 2.0) www.flickr.com/photos/kim_scarborough/147973192/, by Kim Scarborough,2006 (CC BY-SA 2.0)
  42. • repost top content from Reddit • tell people about

    your submission • good grammar and no speling mistakes • repost comments that got good score • exploit (coding) flaws • use lots of sock puppet accounts General cheating (this always works) http://commons.wikimedia.org/wiki/File:Cheating.JPG by Hariadhi (GNU FDL)
  43. Gaming the System • subscriptions for money: see each story

    10-20 minutes before it goes live. • have a low user ID • promote viewpoints which align with popular sentiment, e.g Lunix [sic] • copy comment from lower down the page • be informative (google, anyone?) • submit stories more than comments (more karma earned on average) • submit stories between 9:00 and 12:00 UTC • make quality comments www.flickr.com/photos/jomenager/7622610418/ by Georges Ménager, 2012 (CC BY-NC-ND 2.0)
  44. “ if you try cheating, PG himself will come down

    and smack you on the head, and you sure don't want that! — http://www.kirillzubovsky.com/getting-featured-on-hacker-news
  45. unless ($opts{'f'}) { die <<ERR if $user->{karma} > 10; User

    $_ has a karma of $user->{karma}. Slap aborted. Use -f to forcibly slap. ERR } print "Bitch Slapping #$_...\n"; print "Karma Reduced\n"; print "Default Score Reduced\n"; $slashdb->setUser($_, { -karma => $constants->{badkarma}, -defaultpoints => -1, }); print "ALL live comments moderated down\n\n"; $slashdb->sqlDo("UPDATE comments SET points=-1,reason=1 WHERE uid=$_"); Exhibit A — source of bitchslap (Slashdot)
  46. Moderation adjectives Normal The default setting attached to every comment

    when you have moderation privileges. Offtopic A comment which has nothing to do with the story it's linked to (song lyrics, obscene ascii art, etc). Flamebait Comments whose sole purpose is to insult and enrage. Troll A Troll is similar to Flamebait, but slightly more refined. This is a prank comment intended to provoke indignant (or just confused) responses. Redundant Redundant posts add no new information; they take up space with information either in the original post, the attached links, or lots of previous comments. Insightful An Insightful comment makes you think, or puts a new spin on a given story. Examples: an analogy you hadn't thought of, or a telling counterexample. Interesting If you believe a comment to be Interesting (and on-topic), it is. Informative Informative comments add new information to explain the circumstances hinted at by a particular story, fill in "The Other Side" of an argument, etc. Funny Choose "Funny" if you think the comment is actually funny, not just because it seems intended to be. Overrated Sometimes comments are disproportionately up-moderated—this probably means several moderators saw it at nearly the same time, and their cumulative scores exaggerated its merit. (Example: A knock- knock joke at +5, Funny.) Such a comment is Overrated. Underrated Likewise, some comments get smashed lower than they might deserve. Choosing "Underrated" means you think it should be read by more people.
  47. Distribution of comment scores on Slashdot Slash(dot) and Burn: Distributed

    Moderation in a Large Online Conversation Space. Lampe, Resnick, 2004