Evolving IGN's New APIs on Scala

Slides from my talk at JavaOne 2012 on how we evolved IGN's new APIs on Scala and Frameworks related to Scala.

1. Evolving  IGN’s  new  APIs  with   Scala   Manish  Pandit   Oct  4th,  2012  
2. About  me       Manish  Pandit   Director  of  Engineering,  IGN   @lobster1234   linkedin.com/in/mpandit  
3. About  IGN   We  are  a  leading  online  media  and  services  company     obsessed  with  gaming  and  entertainment.     56MM  Monthly  Unique  Views   737MM  Monthly  Page  Views   20MM  Monthly  Video  Views    
4. My  ObjecTve   Is:   To  get  you  excited  about  building  APIs  in  Scala     Isn't:   To  incite  a  funcTonal  vs.  imperaTve,  or  Scala  vs.   ____  debate    
5. Agenda   •  APIs  at  IGN   •  API  EvoluTon   •  Why  Scala   •  API  Development  with  Scala   •  Monitoring,  Maintaining,  Evolving   •  Performance  Management      
6. APIs  at  IGN   •  Offer  SOA  advantages   •  Isolate  the  back  end  systems  from  the  FE   •  IGN  is  going  where  the  audience  is,  fast   – Mobile  Devices,  Consoles,  TVs,  Aggregators   •  APIs  are  a  specialized  skillset   •  Expand  the  development  to  outside   developers  
7. APIs  at  IGN  :  Numbers   •  ~5  Billion  requests  a  month   •  Average  Response  Tme  of  under  20ms  on   cache  misses   •  17  APIs  between  Social,  OAuth,  and  Content   •  Hardest  hit  APIs  doing  about  25K  RPM  at  peak  
8. EvoluTon  
9. The  API  EvoluTon  :  V1  [2006]   •  The  CMS  and  the  desktop  site  calling  Java   Services,  which  wrapped  Oracle  DB  calls   behind  Hibernate.   •  Caching  with  Memcache   – Near-­‐far  Cache  Architecture  to  address  hot  spots   •  Not  all  content  types  were  supported   •  Schema  was  sTll  too  complex  
10. The  API  EvoluTon  :  V2  [2009]   •  Rails  based  Content  API,  Java  based  Social  API   –  Standards  around  request/response   –  Automated  TesTng   –  MongoDB  NoSQL  storage  to  simplify  the  schema   –  Solr   •  Video  and  Blogs  served  by  V2  API   •  Games  and  ArTcles  sTll  used  V1   •  Mixed-­‐infrastructure  with  Rails  and  Java   echosystem   •  Performance  sucked  royally  
11. The  API  EvoluTon  :  V3  [2012]   •  A  fresh  look  at  the  APIs  and  IGN’s  direcTon   •  First  ajempt  to     – Integrate  all  content  types   – Open  up  the  APIs  to  external  consumers   – Evolve  the  CMS,  tradiTonally  a  monolithic  system   – ElasTcSearch   – Varnish  Response  Caching   •  A  learning  opportunity  for  tradiTonal  Java   stack  engineers  
12. API  Development   •  MVC  Model  for  APIs   – View,  which  marshals/unmarshals  request  and   response  data   – Controller,  which  picks  the  right  model  to  serve   the  request   – Model,  which  has  the  necessary  logic  to  talk  to  the   database,  or  other  APIs  to  fetch  data  
13. Why  Scala   •  Performance  Ted  to  cores  than  processor   speed,  i.e.  concurrency   •  Ideal  for  API  development  –  (relaTvely)  simple   to  handle  concurrency  and  immutability   •  JVM  based  =  Performance   •  Re-­‐use  Java  toolkits  and  libraries   •  Concise,  expressive  code  (1:2.5  with  Java)   •  Actor  model  makes  the  difficult  parts  easy  
14. Why  Scala   •  Richer  CollecTons  API   •  Traits  enabling  Mix-­‐Ins  and  behavior  re-­‐use   •  StaTcally  Typed  w/Type  inference   •  FuncTonal   •  Growing  developer  community   •  REPL   •  Culture  –  learning,  hacking  and  achieving      
15. The  Biggest  Reason   •  Just  makes  sense  
16. The  Biggest  Reason   •  Just  makes  sense   – Pajern  matching   – Implicit  conversions   – Control  abstracTons  for  collecTon  mapping,   filtering,  iteraTng   – Guaranteed  Singletons  at  the  classloader  level   – Almost  non-­‐existent  boilerplate   ….ProducTve,  and  fun!  
17. Picking  up  Scala  :  Week  1   •  Scala  for  Java  Developers  fast-­‐tracked  the   learning   •  Syntax  Familiarity   •  Java  without  semicolons?   def total(nums: List[Int]) :Int = {! var sum:Int=0! for(num<-nums) {! sum+=num! }! sum! }!
18. Picking  up  Scala  :  Week  4   •  Reduce  the  lines  of  code,  embrace   immutability,  use  the  funcTonal  paradigms   built  into  the  language     •  Unmystrify  the  implicits  magic   nums.foldLeft(0)((n,c) => n+c)!
19. Picking  up  Scala  :  Today   •  Focus  on  immutability   •  Think  funcTonal  –  with  the  knowledge  gained   from  using  the  funcTonal  aspects  of  the   language,  build  control  abstracTons   •  Keep  learning  and  applying  
20. API  Components   •  Scalatra  (View,  Controller)   •  Lit-­‐MongoRecord  for  MongoDB  (Model)   •  Casbah  for  MongoDB   •  Play!  Anorm  for  MySQL   •  Actors  for  tasks  like  syndicaTon   •  Maven  for  Builds   •  MongoDB  for  persistance  
21. Other  API  Components   •  Varnish  Cache     •  ElasTcsearch   •  Yammer  Metrics  (ping,  healthcheck)   •  Swagger  (self-­‐documenTng  RESTful  APIs)   •  3Scale  for  Partner  APIs   •  IntelliJ  IDEA  and  Eclipse  with  ScalaIDE  
22. Challenges  with  Scala   •  Learning  Curve  depends  on  the  culture   •  TPS  Cover  Sheets,  vs.  Hack  Nights   •  Scala  is  what  you  make  of  it    
23. Challenges  with  Scala   •  Scala  is  what  you  make  of  it   – Middle  ground  –  simple  is  beauTful   •  Tooling  not  as  rich  as  Java     – Changing  with  Typesafe  invesTng  in  ScalaIDE    
24. Awesome..?       hjp://scalaz.github.com/scalaz/scalaz-­‐2.9.1-­‐6.0.4/doc.sxr/scalaz/BKTree.scala.html  
25. Challenges  with  Scala   •  Slow  compilaTon  based  on  the  source   – How  well  the  code  matches  up  to  the  JVM   constructs   •  No  (binary)  compaTbility  of  dependencies  for   the  major  releases  (2.8,  2.9,  2.10(?))   •  Monitoring   – No  naTve  instrumentaTon  agents  as  they  exist  for   Java  
26. API  Development  :  Deploying   •  ConTnuous  Deployment   •  Jenkins  hooks  to  github   •  Pushes  arTfact(s)  to  Nexus     •  Fabric  to  push  code  to  producTon  (to  be   automated)    
27. API  Performance  Management   •  As  far  as  Scala  goes,  the  Tps  are  not  much   different  than  Java   – GC  Tuning   – Avoid  shared  state   – Parallel  collecTons  
28. API  Performance  Management   •  For  the  overall  APIs,  the  performance  tuning  has   many  knobs  to  work  around  with   –  Stateless  services   –  Async  calls  with  Akka  Agents   –  Response  level  cache   •  Keep  an  eye  on  the  database  and  network   –  Control  the  payload  size   –  Use  reasonable  defaults     –  Allow  field  filtering   –  Paginate  your  response  
29. Monitoring  and  Maintenance   •  Newrelic   – JVM  stats,  SQL  trace   – Supports  Scala  via  annotaTons,  but  not  as  rich  as   Java   •  Typesafe  Console  for  Akka  
30. Future  of  APIs  at  IGN   •  Opening  up  the  APIs   •  Tests,  tests,  tests..   •  Move  towards  simplifying  the  stack   – sbt  vs.  maven   – Typesafe  stack      
31. The  IGN  Tech  Stack  
32. Resources   •  Do  NOT  use  scaladoc  as  your  1st  reference,  at   least  iniTally   •  StackOverflow   •  Twijer’s  Scala  School   •  Online  blogs  and  tutorials   •  Book:  Scala  for  the  ImpaTent   •  Scala  Meetup  Groups  in  your  area   •  Coursera  course  on  FP  by  MarTn  Odersky    
33. QuesTons