Evolving  IGN’s  new  APIs  with   Scala   Manish  Pandit   Oct  4th,  2012   

About  me       Manish  Pandit   Director  of  Engineering,  IGN   @lobster1234   linkedin.com/in/mpandit   

About  IGN   We  are  a  leading  online  media  and  services  company     obsessed  with  gaming  and  entertainment.     56MM  Monthly  Unique  Views   737MM  Monthly  Page  Views   20MM  Monthly  Video  Views     

My  ObjecTve   Is:   To  get  you  excited  about  building  APIs  in  Scala     Isn't:   To  incite  a  funcTonal  vs.  imperaTve,  or  Scala  vs.   ____  debate     

Agenda   •  APIs  at  IGN   •  API  EvoluTon   •  Why  Scala   •  API  Development  with  Scala   •  Monitoring,  Maintaining,  Evolving   •  Performance  Management       

APIs  at  IGN   •  Offer  SOA  advantages   •  Isolate  the  back  end  systems  from  the  FE   •  IGN  is  going  where  the  audience  is,  fast   – Mobile  Devices,  Consoles,  TVs,  Aggregators   •  APIs  are  a  specialized  skillset   •  Expand  the  development  to  outside   developers   

APIs  at  IGN  :  Numbers   •  ~5  Billion  requests  a  month   •  Average  Response  Tme  of  under  20ms  on   cache  misses   •  17  APIs  between  Social,  OAuth,  and  Content   •  Hardest  hit  APIs  doing  about  25K  RPM  at  peak   

EvoluTon   

The  API  EvoluTon  :  V1  [2006]   •  The  CMS  and  the  desktop  site  calling  Java   Services,  which  wrapped  Oracle  DB  calls   behind  Hibernate.   •  Caching  with  Memcache   – Near-­‐far  Cache  Architecture  to  address  hot  spots   •  Not  all  content  types  were  supported   •  Schema  was  sTll  too  complex   

The  API  EvoluTon  :  V2  [2009]   •  Rails  based  Content  API,  Java  based  Social  API   –  Standards  around  request/response   –  Automated  TesTng   –  MongoDB  NoSQL  storage  to  simplify  the  schema   –  Solr   •  Video  and  Blogs  served  by  V2  API   •  Games  and  ArTcles  sTll  used  V1   •  Mixed-­‐infrastructure  with  Rails  and  Java   echosystem   •  Performance  sucked  royally   

The  API  EvoluTon  :  V3  [2012]   •  A  fresh  look  at  the  APIs  and  IGN’s  direcTon   •  First  ajempt  to     – Integrate  all  content  types   – Open  up  the  APIs  to  external  consumers   – Evolve  the  CMS,  tradiTonally  a  monolithic  system   – ElasTcSearch   – Varnish  Response  Caching   •  A  learning  opportunity  for  tradiTonal  Java   stack  engineers   

API  Development   •  MVC  Model  for  APIs   – View,  which  marshals/unmarshals  request  and   response  data   – Controller,  which  picks  the  right  model  to  serve   the  request   – Model,  which  has  the  necessary  logic  to  talk  to  the   database,  or  other  APIs  to  fetch  data   

Why  Scala   •  Performance  Ted  to  cores  than  processor   speed,  i.e.  concurrency   •  Ideal  for  API  development  –  (relaTvely)  simple   to  handle  concurrency  and  immutability   •  JVM  based  =  Performance   •  Re-­‐use  Java  toolkits  and  libraries   •  Concise,  expressive  code  (1:2.5  with  Java)   •  Actor  model  makes  the  difficult  parts  easy   

Why  Scala   •  Richer  CollecTons  API   •  Traits  enabling  Mix-­‐Ins  and  behavior  re-­‐use   •  StaTcally  Typed  w/Type  inference   •  FuncTonal   •  Growing  developer  community   •  REPL   •  Culture  –  learning,  hacking  and  achieving       

The  Biggest  Reason   •  Just  makes  sense   

The  Biggest  Reason   •  Just  makes  sense   – Pajern  matching   – Implicit  conversions   – Control  abstracTons  for  collecTon  mapping,   filtering,  iteraTng   – Guaranteed  Singletons  at  the  classloader  level   – Almost  non-­‐existent  boilerplate   ….ProducTve,  and  fun!   

Picking  up  Scala  :  Week  1   •  Scala  for  Java  Developers  fast-­‐tracked  the   learning   •  Syntax  Familiarity   •  Java  without  semicolons?   def total(nums: List[Int]) :Int = {! var sum:Int=0! for(num<-nums) {! sum+=num! }! sum! }! 

Picking  up  Scala  :  Week  4   •  Reduce  the  lines  of  code,  embrace   immutability,  use  the  funcTonal  paradigms   built  into  the  language     •  Unmystrify  the  implicits  magic   nums.foldLeft(0)((n,c) => n+c)! 

Picking  up  Scala  :  Today   •  Focus  on  immutability   •  Think  funcTonal  –  with  the  knowledge  gained   from  using  the  funcTonal  aspects  of  the   language,  build  control  abstracTons   •  Keep  learning  and  applying   

API  Components   •  Scalatra  (View,  Controller)   •  Lit-­‐MongoRecord  for  MongoDB  (Model)   •  Casbah  for  MongoDB   •  Play!  Anorm  for  MySQL   •  Actors  for  tasks  like  syndicaTon   •  Maven  for  Builds   •  MongoDB  for  persistance   

Other  API  Components   •  Varnish  Cache     •  ElasTcsearch   •  Yammer  Metrics  (ping,  healthcheck)   •  Swagger  (self-­‐documenTng  RESTful  APIs)   •  3Scale  for  Partner  APIs   •  IntelliJ  IDEA  and  Eclipse  with  ScalaIDE   

Challenges  with  Scala   •  Learning  Curve  depends  on  the  culture   •  TPS  Cover  Sheets,  vs.  Hack  Nights   •  Scala  is  what  you  make  of  it     

Challenges  with  Scala   •  Scala  is  what  you  make  of  it   – Middle  ground  –  simple  is  beauTful   •  Tooling  not  as  rich  as  Java     – Changing  with  Typesafe  invesTng  in  ScalaIDE     

Awesome..?       hjp://scalaz.github.com/scalaz/scalaz-­‐2.9.1-­‐6.0.4/doc.sxr/scalaz/BKTree.scala.html   

Challenges  with  Scala   •  Slow  compilaTon  based  on  the  source   – How  well  the  code  matches  up  to  the  JVM   constructs   •  No  (binary)  compaTbility  of  dependencies  for   the  major  releases  (2.8,  2.9,  2.10(?))   •  Monitoring   – No  naTve  instrumentaTon  agents  as  they  exist  for   Java   

API  Development  :  Deploying   •  ConTnuous  Deployment   •  Jenkins  hooks  to  github   •  Pushes  arTfact(s)  to  Nexus     •  Fabric  to  push  code  to  producTon  (to  be   automated)     

API  Performance  Management   •  As  far  as  Scala  goes,  the  Tps  are  not  much   different  than  Java   – GC  Tuning   – Avoid  shared  state   – Parallel  collecTons   

API  Performance  Management   •  For  the  overall  APIs,  the  performance  tuning  has   many  knobs  to  work  around  with   –  Stateless  services   –  Async  calls  with  Akka  Agents   –  Response  level  cache   •  Keep  an  eye  on  the  database  and  network   –  Control  the  payload  size   –  Use  reasonable  defaults     –  Allow  field  filtering   –  Paginate  your  response   

Monitoring  and  Maintenance   •  Newrelic   – JVM  stats,  SQL  trace   – Supports  Scala  via  annotaTons,  but  not  as  rich  as   Java   •  Typesafe  Console  for  Akka   

Future  of  APIs  at  IGN   •  Opening  up  the  APIs   •  Tests,  tests,  tests..   •  Move  towards  simplifying  the  stack   – sbt  vs.  maven   – Typesafe  stack       

The  IGN  Tech  Stack   

Resources   •  Do  NOT  use  scaladoc  as  your  1st  reference,  at   least  iniTally   •  StackOverflow   •  Twijer’s  Scala  School   •  Online  blogs  and  tutorials   •  Book:  Scala  for  the  ImpaTent   •  Scala  Meetup  Groups  in  your  area   •  Coursera  course  on  FP  by  MarTn  Odersky     

QuesTons