Evolving IGN's New APIs on Scala

Transcript

1. Evolving  IGN’s  new  APIs  with   Scala   Manish  Pandit

     Oct  4th,  2012  

2. About  me       Manish  Pandit   Director  of

    Engineering,  IGN   @lobster1234   linkedin.com/in/mpandit  

3. About  IGN   We  are  a  leading  online  media  and

    services  company     obsessed  with  gaming  and  entertainment.     56MM  Monthly  Unique  Views   737MM  Monthly  Page  Views   20MM  Monthly  Video  Views    

4. My  ObjecTve   Is:   To  get  you  excited  about

    building  APIs  in  Scala     Isn't:   To  incite  a  funcTonal  vs.  imperaTve,  or  Scala  vs.   ____  debate    

5. Agenda   •  APIs  at  IGN   •  API  EvoluTon

     •  Why  Scala   •  API  Development  with  Scala   •  Monitoring,  Maintaining,  Evolving   •  Performance  Management      

6. APIs  at  IGN   •  Offer  SOA  advantages   • 

   Isolate  the  back  end  systems  from  the  FE   •  IGN  is  going  where  the  audience  is,  fast   – Mobile  Devices,  Consoles,  TVs,  Aggregators   •  APIs  are  a  specialized  skillset   •  Expand  the  development  to  outside   developers  

7. APIs  at  IGN  :  Numbers   •  ~5  Billion  requests

    a  month   •  Average  Response  Tme  of  under  20ms  on   cache  misses   •  17  APIs  between  Social,  OAuth,  and  Content   •  Hardest  hit  APIs  doing  about  25K  RPM  at  peak  

8. EvoluTon  

9. The  API  EvoluTon  :  V1  [2006]   •  The  CMS

    and  the  desktop  site  calling  Java   Services,  which  wrapped  Oracle  DB  calls   behind  Hibernate.   •  Caching  with  Memcache   – Near-­‐far  Cache  Architecture  to  address  hot  spots   •  Not  all  content  types  were  supported   •  Schema  was  sTll  too  complex  

10. The  API  EvoluTon  :  V2  [2009]   •  Rails  based

     Content  API,  Java  based  Social  API   –  Standards  around  request/response   –  Automated  TesTng   –  MongoDB  NoSQL  storage  to  simplify  the  schema   –  Solr   •  Video  and  Blogs  served  by  V2  API   •  Games  and  ArTcles  sTll  used  V1   •  Mixed-­‐infrastructure  with  Rails  and  Java   echosystem   •  Performance  sucked  royally  

11. The  API  EvoluTon  :  V3  [2012]   •  A  fresh

     look  at  the  APIs  and  IGN’s  direcTon   •  First  ajempt  to     – Integrate  all  content  types   – Open  up  the  APIs  to  external  consumers   – Evolve  the  CMS,  tradiTonally  a  monolithic  system   – ElasTcSearch   – Varnish  Response  Caching   •  A  learning  opportunity  for  tradiTonal  Java   stack  engineers  

12. API  Development   •  MVC  Model  for  APIs   – View,

     which  marshals/unmarshals  request  and   response  data   – Controller,  which  picks  the  right  model  to  serve   the  request   – Model,  which  has  the  necessary  logic  to  talk  to  the   database,  or  other  APIs  to  fetch  data  

13. Why  Scala   •  Performance  Ted  to  cores  than  processor

      speed,  i.e.  concurrency   •  Ideal  for  API  development  –  (relaTvely)  simple   to  handle  concurrency  and  immutability   •  JVM  based  =  Performance   •  Re-­‐use  Java  toolkits  and  libraries   •  Concise,  expressive  code  (1:2.5  with  Java)   •  Actor  model  makes  the  difficult  parts  easy  

14. Why  Scala   •  Richer  CollecTons  API   •  Traits

     enabling  Mix-­‐Ins  and  behavior  re-­‐use   •  StaTcally  Typed  w/Type  inference   •  FuncTonal   •  Growing  developer  community   •  REPL   •  Culture  –  learning,  hacking  and  achieving      

15. The  Biggest  Reason   •  Just  makes  sense  

16. The  Biggest  Reason   •  Just  makes  sense   – Pajern

     matching   – Implicit  conversions   – Control  abstracTons  for  collecTon  mapping,   filtering,  iteraTng   – Guaranteed  Singletons  at  the  classloader  level   – Almost  non-­‐existent  boilerplate   ….ProducTve,  and  fun!  

17. Picking  up  Scala  :  Week  1   •  Scala  for

     Java  Developers  fast-­‐tracked  the   learning   •  Syntax  Familiarity   •  Java  without  semicolons?   def total(nums: List[Int]) :Int = {! var sum:Int=0! for(num<-nums) {! sum+=num! }! sum! }!

18. Picking  up  Scala  :  Week  4   •  Reduce  the

     lines  of  code,  embrace   immutability,  use  the  funcTonal  paradigms   built  into  the  language     •  Unmystrify  the  implicits  magic   nums.foldLeft(0)((n,c) => n+c)!

19. Picking  up  Scala  :  Today   •  Focus  on  immutability

      •  Think  funcTonal  –  with  the  knowledge  gained   from  using  the  funcTonal  aspects  of  the   language,  build  control  abstracTons   •  Keep  learning  and  applying  

20. API  Components   •  Scalatra  (View,  Controller)   •  Lit-­‐MongoRecord

     for  MongoDB  (Model)   •  Casbah  for  MongoDB   •  Play!  Anorm  for  MySQL   •  Actors  for  tasks  like  syndicaTon   •  Maven  for  Builds   •  MongoDB  for  persistance  

21. Other  API  Components   •  Varnish  Cache     • 

    ElasTcsearch   •  Yammer  Metrics  (ping,  healthcheck)   •  Swagger  (self-­‐documenTng  RESTful  APIs)   •  3Scale  for  Partner  APIs   •  IntelliJ  IDEA  and  Eclipse  with  ScalaIDE  

22. Challenges  with  Scala   •  Learning  Curve  depends  on  the

     culture   •  TPS  Cover  Sheets,  vs.  Hack  Nights   •  Scala  is  what  you  make  of  it    

23. Challenges  with  Scala   •  Scala  is  what  you  make

     of  it   – Middle  ground  –  simple  is  beauTful   •  Tooling  not  as  rich  as  Java     – Changing  with  Typesafe  invesTng  in  ScalaIDE    

24. Awesome..?       hjp://scalaz.github.com/scalaz/scalaz-­‐2.9.1-­‐6.0.4/doc.sxr/scalaz/BKTree.scala.html  

25. Challenges  with  Scala   •  Slow  compilaTon  based  on  the

     source   – How  well  the  code  matches  up  to  the  JVM   constructs   •  No  (binary)  compaTbility  of  dependencies  for   the  major  releases  (2.8,  2.9,  2.10(?))   •  Monitoring   – No  naTve  instrumentaTon  agents  as  they  exist  for   Java  

26. API  Development  :  Deploying   •  ConTnuous  Deployment   • 

    Jenkins  hooks  to  github   •  Pushes  arTfact(s)  to  Nexus     •  Fabric  to  push  code  to  producTon  (to  be   automated)    

27. API  Performance  Management   •  As  far  as  Scala  goes,

     the  Tps  are  not  much   different  than  Java   – GC  Tuning   – Avoid  shared  state   – Parallel  collecTons  

28. API  Performance  Management   •  For  the  overall  APIs,  the

     performance  tuning  has   many  knobs  to  work  around  with   –  Stateless  services   –  Async  calls  with  Akka  Agents   –  Response  level  cache   •  Keep  an  eye  on  the  database  and  network   –  Control  the  payload  size   –  Use  reasonable  defaults     –  Allow  field  filtering   –  Paginate  your  response  

29. Monitoring  and  Maintenance   •  Newrelic   – JVM  stats,  SQL

     trace   – Supports  Scala  via  annotaTons,  but  not  as  rich  as   Java   •  Typesafe  Console  for  Akka  

30. Future  of  APIs  at  IGN   •  Opening  up  the

     APIs   •  Tests,  tests,  tests..   •  Move  towards  simplifying  the  stack   – sbt  vs.  maven   – Typesafe  stack      

31. The  IGN  Tech  Stack  

32. Resources   •  Do  NOT  use  scaladoc  as  your  1st

     reference,  at   least  iniTally   •  StackOverflow   •  Twijer’s  Scala  School   •  Online  blogs  and  tutorials   •  Book:  Scala  for  the  ImpaTent   •  Scala  Meetup  Groups  in  your  area   •  Coursera  course  on  FP  by  MarTn  Odersky    

33. QuesTons