Sweetie Bot, Part 3 by Cloud Hop (#PonyDev @ BronyCon 2018)

A 10 minute lightning talk at BronyCon 2018's Pony Programming panel. Find more slides at http://developer.horse

Presented by Cloud Hop

Transcript

1. Sweetie  Bot I  MADE  AN  ANTI-­‐SPAM  BOT  FOR  DISCORD NOW  I  REGRET  EVERYTHING
2. Introduction Erik  McClure  aka  Cloud  Hop From  Redmond,  Washington Works  At  Microsoft  (again) Made  Pony  Clicker  (for  some  reason) Sometimes  writes  music  and  fanfiction http://cloudhop.horse twitter:  @blackhole0173
3. Discord § A  new  chat  and  voice  client  for  gamers  that  emerged  in  late  2015. § Way  better  than  Skype  in  almost  every  way § Honestly  this  slide  should  probably  be  removed  because  everyone  knows  what  Discord  is.
4. Bot  Origins § When  Discord  first  emerged,  My  Little  Pony  chats  were  targeted  by  trolls § Building  an  anti-­‐spam  bot  required  using  a  reversed  engineered,  unsupported  api § The  API  itself  would  often  throw  spurious  errors  or  simply  fail  to  perform  actions § Sweetie  Bot  exploded  to  1000+  servers  with  hundreds  of  thousands  of  users § She  is  still  the  top  google  result  for  “discord  anti-­‐spam  bot”,  for  some  reason. § The  old  code  intended  to  deal  with  a  few  hundred  servers  simply  couldn’t  deal  with  this § Last  time,  I  talked  about  the  database  deadlock  issues.
5. Then  This  Happened
6. Debugging  With  Cloud  Hop § This  time,  let’s  talk  about  Go  deadlocks. § Discordgo is  either  single-­‐threaded  or  launches  a  new  goroutine for  every  event. § Single-­‐threaded  is  bad  because  of  a  rate-­‐limiting  edge  case. § Multi-­‐threaded  is  bad  because  discordgo is  awful. § Sweetie’s  method  of  silencing  requires  solving  a  race-­‐condition § In  this  case,  doing  things  the  right  way  was  a  bad  idea.   § Sweetie’s  deadlocking  was  so  bad  I  built  a  deadlock  detector § It  literally  turns  the  bot  off  and  on  again  if  something  goes  wrong
7. Fix  Code  By  Introducing  Race  Conditions! § For  the  love  of  all  that  is  holy,  NEVER  ACTUALLY  DO  THIS. § The  deadlock  is  caused  by  DiscordGo’s terrible  architecture. § One  state  object  holds  state  for  all  servers  and  has  only  one  lock § Trying  to  run  discordgo using  Go’s  data  race  detector  tells  you  that  everything  is  a  data  race § Discordgo assumes  that  you  never  take  write  lock  on  the  state  in  an  event  handler. § This  isn’t  actually  mentioned  anywhere § Fixing  this  requires  rewriting  the  entire  library  from  the  ground  up § Even  just  fixing  this  specific  subcase  would  require  a  pull  request. § So  I  just  assign  the  new  slice  to  the  old  slice  without  a  lock  and  hope  it  doesn’t  crash § It  worked
8. Lessons  Learned § Discordgo is  a  bad  library § Making  this  bot  was  a  mistake § The  only  way  to  fix  this  is  to  either  build  a   new  Go  API  library  or  use  another  language § But  the  discord  rate-­‐limit  edge  case  makes  this  a   giant  pain  in  the  ass  in  any  language. § Most  of  this  could  probably  have  been   avoided  with  more  granular  locking. § Unfortunately  discordgo has  more  fundamental   architectural  issues.