Pros and cons of Scala as a server-side programming language at ShiftForward

ShiftForward has been using Scala almost since its inception, in 2011. During the past few years, both the language and the way ShiftForward uses it has evolved. This talk aims to explain ShiftForward's experiences with Scala: why we decided to go with it in the first place, what benefits we take from it, how it affects our programming style and what issues we have encountered in using it.

Transcript

1. PROS AND CONS OF SCALA AS A SERVER-SIDE PROGRAMMING LANGUAGE AT SHIFTFORWARD / Joao Azevedo @jcazevedo http://jcazevedo.net/commit2016
2. ABOUT ME Graduated from FEUP in 2010; Worked on a crew-scheduling application for a railway company at SISCOG from 2010 to 2011; Worked on mobile application development and signal analysis tools at Fraunhofer Portugal from 2011 to 2012; At Shi Forward since 2012, working on distributed, responsive and scalable systems for the online advertising industry.
3. SHIFTFORWARD: WHY SCALA IN THE FIRST PLACE?
4. STATICALLY TYPED WITH DECENT TYPE INFERENCE Specifying program invariants at compile time: Increases development speed; Removes the need for some classes of tests. Not having to declare every single type removes excessive verbosity.
5. JVM Platform independence (predictable runtime environment) eases the deployment of services; The ecosystem is very rich: libraries and tools.
6. CONCURRENCY Thread-based concurrency (from Java); Actor-based concurrency (inspired by Erlang, available in Akka); Parallel collections; Controllable immutability.
7. SCALA PROS: THE TYPE SYSTEM
8. SCALA CLASS HIERARCHY AND UNIFIED TYPES
9. PARAMETRIC POLYMORPHISM d e f d u p [ T ] ( x : T , n : I n t ) : L i s t [ T ] = i f ( n = = 0 ) N i l e l s e x : : d u p ( x , n - 1 ) s c a l a > d u p ( 3 , 3 ) r e s 0 : L i s t [ I n t ] = L i s t ( 3 , 3 , 3 ) s c a l a > d u p ( " t h r e e " , 3 ) r e s 1 : L i s t [ S t r i n g ] = L i s t ( t h r e e , t h r e e , t h r e e )
10. TYPE INFERENCE d e f i d [ T ] ( x : T ) = x s c a l a > v a l x = i d ( 3 2 2 ) x : I n t = 3 2 2 s c a l a > v a l x = i d ( " h e y " ) x : S t r i n g = h e y
11. FUNCTIONS AS TYPES d e f f o o [ A , B ] ( l : L i s t [ A ] , f : A = > B ) : L i s t [ B ] = i f ( l . l e n g t h = = 0 ) N i l e l s e f ( l . h e a d ) : : f o o ( l . t a i l , f ) s c a l a > f o o ( L i s t ( 1 , 2 , 3 ) , { x : I n t = > x * 2 } ) r e s 0 : L i s t [ I n t ] = L i s t ( 2 , 4 , 6 )
12. TRAITS Collection of fields and behaviors that can be extended or mixed in to classes. t r a i t C a r { v a l b r a n d : S t r i n g d e f e m i t B r a n d : S t r i n g = " M y b r a n d i s " + b r a n d } t r a i t S h i n y { v a l s h i n e R e f l e c t i o n : I n t } c l a s s B M W e x t e n d s C a r w i t h S h i n y { v a l b r a n d = " B M W " v a l s h i n e R e f l e c t i o n = 1 2 } s c a l a > v a l b m w = n e w B M W b m w : B M W = B M W @ 2 d a 8 e d 8 0 s c a l a > b m w . e m i t B r a n d r e s 0 : S t r i n g = M y b r a n d i s B M W
13. SCALA PROS: DEFINING DOMAIN OBJECTS
14. CASE CLASSES c a s e c l a s s U s e r ( i d : I n t , n a m e : S t r i n g , a g e : I n t ) Can be constructed without using n e w ; Automatically have equality and hash code methods; Nice t o S t r i n g methods; Pattern matching.
15. CASE CLASSES c a s e c l a s s U s e r ( i d : I n t , n a m e : S t r i n g , a g e : I n t ) c a s e c l a s s S i t e ( u s e r s : L i s t [ U s e r ] ) v a l s i t e 1 = S i t e ( L i s t ( U s e r ( 0 , " a l i c e " , 3 7 ) , U s e r ( 1 , " b o b " , 4 6 ) ) ) v a l s i t e 2 = S i t e ( L i s t ( U s e r ( 0 , " a l i c e " , 3 7 ) , U s e r ( 1 , " b o b " , 4 6 ) ) ) s c a l a > s i t e 1 . t o S t r i n g r e s 0 : S t r i n g = S i t e ( L i s t ( U s e r ( 0 , a l i c e , 3 7 ) , U s e r ( 1 , b o b , 4 6 ) ) ) s c a l a > s i t e 1 = = s i t e 2 r e s 1 : B o o l e a n = t r u e
16. PATTERN MATCHING d e f a v e r a g e A g e ( s i t e : S i t e ) = { d e f a g e S u m ( u s e r s : L i s t [ U s e r ] ) : I n t = u s e r s m a t c h { c a s e U s e r ( _ , _ , a g e ) : : r e s t = > a g e + a g e S u m ( r e s t ) c a s e N i l = > 0 } a g e S u m ( s i t e . u s e r s ) / s i t e . u s e r s . l e n g t h } v a l s i t e 1 = S i t e ( L i s t ( U s e r ( 0 , " a l i c e " , 3 7 ) , U s e r ( 1 , " b o b " , 4 6 ) ) ) s c a l a > a v e r a g e A g e ( s i t e 1 ) r e s 0 : I n t = 4 1
17. EXAMPLE: MODELING JSON s e a l e d t r a i t J s V a l u e { d e f t o J s o n S t r i n g : S t r i n g } c a s e c l a s s J s N u m b e r ( n : I n t ) e x t e n d s J s V a l u e { d e f t o J s o n S t r i n g = n . t o S t r i n g } c a s e c l a s s J s S t r i n g ( s : S t r i n g ) e x t e n d s J s V a l u e { d e f t o J s o n S t r i n g = " \ " " + s + " \ " " } c a s e c l a s s J s A r r a y ( v a l u e s : A r r a y [ J s V a l u e ] ) e x t e n d s J s V a l u e { d e f t o J s o n S t r i n g = { v a l e l e m e n t s S t r i n g = v a l u e s . m a p ( _ . t o J s o n S t r i n g ) . m k S t r i n g ( " , " ) " [ " + e l e m e n t s S t r i n g + " ] " } }
18. EXAMPLE: MODELING JSON c a s e c l a s s J s O b j e c t ( v a l u e s : M a p [ S t r i n g , J s V a l u e ] ) e x t e n d s J s V a l u e { d e f t o J s o n S t r i n g = { v a l e l e m e n t s = v a l u e s . m a p { c a s e ( k , v ) = > " \ " " + k + " \ " : " + v . t o J s o n S t r i n g } " { " + e l e m e n t s . m k S t r i n g ( " , " ) + " } " } } v a l u s e r J s o n = J s O b j e c t ( M a p ( " i d " - > J s N u m b e r ( 0 ) , " n a m e " - > J s S t r i n g ( " a l i c e " ) , " a g e " - > J s N u m b e r ( 3 7 ) ) ) s c a l a > u s e r J s o n . t o J s o n S t r i n g r e s 0 : S t r i n g = { " i d " : 0 , " n a m e " : " a l i c e " , " a g e " : 3 7 }
19. SCALA PROS: MONADS
20. A WAY TO ABSTRACT COMPUTATIONS t r a i t M [ A ] d e f u n i t [ A ] : A = > M [ A ] = ? ? ? d e f f l a t M a p [ A , B ] : M [ A ] = > ( A = > M [ B ] ) = > M [ B ] = ? ? ?
21. MONADIC OPERATIONS EVERYWHERE s c a l a > L i s t ( 1 , 2 , 3 ) . f l a t M a p { x = > L i s t ( x - 1 , x , x + 1 ) } r e s 0 : L i s t [ I n t ] = L i s t ( 0 , 1 , 2 , 1 , 2 , 3 , 2 , 3 , 4 ) s c a l a > L i s t ( 1 , 2 , 3 ) . m a p { x = > x * 2 } r e s 1 : L i s t [ I n t ] = L i s t ( 2 , 4 , 6 )
22. (FLAT)MAP IS THE GLUE v a l l 1 = L i s t ( 1 , 2 ) v a l l 2 = L i s t ( 8 , 9 ) l 1 f l a t M a p { v 1 = > l 2 m a p { v 2 = > v 1 * v 2 } } r e s 0 : L i s t [ I n t ] = L i s t ( 8 , 9 , 1 6 , 1 8 )
23. FOR COMPREHENSIONS v a l l 1 = L i s t ( 1 , 2 ) v a l l 2 = L i s t ( 8 , 9 ) f o r { v 1 < - l 1 v 2 < - l 2 } y i e l d v 1 * v 2 r e s 0 : L i s t [ I n t ] = L i s t ( 8 , 9 , 1 6 , 1 8 )
24. REUSABLE COMPONENTS d e f f ( v : I n t ) = v * 2 v a l l = L i s t ( 1 , 2 , 3 , 4 ) v a l o = S o m e ( 4 ) v a l t = T r y ( 5 ) s c a l a > l . m a p ( f ) r e s 0 : L i s t [ I n t ] = L i s t ( 2 , 4 , 6 , 8 ) s c a l a > o . m a p ( f ) r e s 1 : O p t i o n [ I n t ] = S o m e ( 8 ) s c a l a > t . m a p ( f ) r e s 2 : s c a l a . u t i l . T r y [ I n t ] = S u c c e s s ( 1 0 )
25. SCALA PROS: TYPE CLASSES
26. PREREQUISITE: IMPLICIT PARAMETERS d e f a d d e r ( a : I n t ) ( i m p l i c i t b : I n t ) = a + b s c a l a > a d d e r ( 2 ) < c o n s o l e > : 1 3 : e r r o r : c o u l d n o t f i n d i m p l i c i t v a l u e f o r p a r a m e t e r b : I n t a d d e r ( 2 ) ^ s c a l a > i m p l i c i t v a l x = 5 x : I n t = 5 s c a l a > a d d e r ( 2 ) r e s 0 : I n t = 7
27. PREREQUISITE: IMPLICIT CONVERSIONS d e f d o u b l e ( v : I n t ) = v * 2 i m p l i c i t d e f s t r i n g T o I n t ( s : S t r i n g ) = s . t o I n t s c a l a > d o u b l e ( " 1 2 3 4 " ) r e s 0 : I n t = 2 4 6 8
28. AD HOC POLYMORPHISM t r a i t J s o n F o r m a t [ A ] { d e f w r i t e ( v a l u e : A ) : J s V a l u e d e f r e a d ( j s o n : J s V a l u e ) : A } d e f t o J s o n [ A ] ( a : A ) ( i m p l i c i t f o r m a t : J s o n F o r m a t [ A ] ) : S t r i n g = { f o r m a t . w r i t e ( a ) . t o J s o n S t r i n g } i m p l i c i t o b j e c t I n t J s o n F o r m a t e x t e n d s J s o n F o r m a t [ I n t ] { d e f w r i t e ( v a l u e : I n t ) = J s N u m b e r ( v a l u e ) d e f r e a d ( j s o n : J s V a l u e ) = j s o n m a t c h { c a s e J s N u m b e r ( v a l u e ) = > v a l u e c a s e _ = > t h r o w n e w E x c e p t i o n ( " U n e x p e c t e d J S O N t y p e " ) } } s c a l a > t o J s o n ( 1 ) r e s 7 : S t r i n g = 1 s c a l a > t o J s o n ( " I ' m a s t r i n g ! " ) < c o n s o l e > : 1 7 : e r r o r : c o u l d n o t f i n d i m p l i c i t v a l u e f o r p a r a m e t e r f o r m a t : J s o n F o r m a t [ S t r i n g ] t o J s o n ( " I ' m a s t r i n g ! " ) ^
29. EXTENSION METHODS t r a i t J s o n W r i t a b l e [ A ] { d e f t o J s o n : S t r i n g } i m p l i c i t d e f t o J s o n W r i t e a b l e [ A ] ( v : A ) ( i m p l i c i t f o r m a t : J s o n F o r m a t [ A ] ) = n e w J s o n W r i t a b l e [ A ] { d e f t o J s o n = f o r m a t . w r i t e ( v ) . t o J s o n S t r i n g } s c a l a > 1 . t o J s o n r e s 0 : S t r i n g = 1 s c a l a > U s e r ( 0 , " a l i c e " , 3 7 ) . t o J s o n r e s 1 : S t r i n g = { " i d " : 0 , " n a m e " : " a l i c e " , " a g e " : 3 7 }
30. SCALA PROS: CONCURRENCY
31. FUTURE s c a l a > F u t u r e { / * e x p e n s i v e c o m p u t a t i o n * / 2 } r e s 0 : s c a l a . c o n c u r r e n t . F u t u r e [ I n t ] = s c a l a . c o n c u r r e n t . i m p l . P r o m i s e $ D e f a u l t P r o m i s e @ 2 c c 4 4 a d s c a l a > r e s 0 . o n C o m p l e t e ( p r i n t l n ) S u c c e s s ( 2 )
32. COMPOSING FUTURES
33. FUTURE IS A MONAD v a l u s e r F u t u r e : F u t u r e [ U s e r ] = u s e r I n f o ( r e q u e s t . u s e r n a m e ) v a l g e o F u t u r e : F u t u r e [ G e o ] = u s e r G e o ( r e q u e s t . i p ) v a l t p D a t a : F u t u r e [ D a t a ] = u s e r F u t u r e . f l a t M a p ( u s e r = > t h i r d P a r t y D a t a ( u s e r . i d ) ) v a l i n c o m e I n f o : F u t u r e [ D o u b l e ] = g e o F u t u r e . f l a t M a p ( g e o = > a v e r a g e I n c o m e ( g e o ) ) f o r { u s e r < - u s e r F u t u r e d a t a < - t p D a t a a v g I n c o m e < - i n c o m e I n f o } y i e l d A u g m e n t e d U s e r ( u s e r , d a t a , a v g I n c o m e )
34. AKKA c l a s s M y A c t o r e x t e n d s A c t o r { d e f r e c e i v e = { c a s e v a l u e : S t r i n g = > p r i n t l n ( " r e c e i v e d " + v a l u e ) c a s e _ = > p r i n t l n ( " r e c e i v e d u n k n o w n m e s s a g e " ) } } s c a l a > v a l a c t o r = s y s t e m . a c t o r O f ( P r o p s ( n e w M y A c t o r ) ) a c t o r : a k k a . a c t o r . A c t o r R e f = A c t o r [ a k k a : / / d e f a u l t / u s e r / $ b # 1 1 3 7 2 1 1 9 8 4 ] s c a l a > a c t o r ! " h e l l o " r e c e i v e d h e l l o s c a l a > a c t o r ! 1 r e c e i v e d u n k n o w n m e s s a g e
35. AKKA c l a s s D o u b l e r e x t e n d s A c t o r { d e f r e c e i v e = { c a s e v : I n t = > s e n d e r ! ( v * 2 ) c a s e _ = > p r i n t l n ( " r e c e i v e d u n k n o w n m e s s a g e " ) } } s c a l a > v a l r e s = a c t o r ? 3 r e s : s c a l a . c o n c u r r e n t . F u t u r e [ A n y ] = S u c c e s s ( 6 )
36. AKKA: WHAT ELSE? The message-driven programming model promotes asynchronous interfaces; Supervision strategies enable fault tolerance; Location transparency promotes scalability; Persistency enables internal state recovery.
37. LARGE-SCALE DATA PROCESSING: APACHE SPARK R D D is another monad! v a l t e x t F i l e = s c . t e x t F i l e ( " h d f s : / / . . . " ) v a l c o u n t s = t e x t F i l e . f l a t M a p ( l i n e = > l i n e . s p l i t ( " " ) ) . m a p ( w o r d = > ( w o r d , 1 ) ) . r e d u c e B y K e y ( _ + _ ) c o u n t s . s a v e A s T e x t F i l e ( " h d f s : / / . . . " )
38. OTHER SCALA PROS Power to create DSLs; Macros; Higher kinded types.
39. SCALA CONS: LEARNING CURVE
40. QUICK TO START USING, HARD TO MASTER When coming from an OO-background, it's easy to write OO code in Scala.
41. FUNCTIONAL IN THE SMALL, OO IN THE LARGE Pure functions, referential transparency and immutability are applied in small, isolated areas.
42. MORE FUNCTIONAL ABSTRACTIONS ; ; . Scalaz Shapeless Cats
43. OO OR FUNCTIONAL DESIGN? Do I design it using typeclasses and ad hoc polymorphism or using subtyping and parametric polymorphism?
44. SCALA CONS: COMPILATION TIMES
45. STARTUP OVERHEAD Scalac has lots of classes which have to be loaded and jit- compiled. The long startup time is mitigated by using incremental compilation.
46. COMPILATION SPEED Type inference; Implicit resolution; Functional idioms generate many more classes per given file size than Java.
47. COMPILATION SPEED
48. COMPILATION SPEED
49. COMPILATION SPEED
50. SCALA CONS: BUILD SYSTEM
51. SBT IS WIDELY USED Simple (?) build tool: a type-safe programmable build configuration.
52. SLIGHTLY OVERENGINEERED AND CONVOLUTED A build definition consists of one or more S e t t i n g s; A S e t t i n g describes a transformation of the build description; S e t t i n g s are scoped: project, configuration and task axis; A T a s k is a computation of values or side effects; T a s k s are also assigned to keys and can depend on settings and other tasks; T a s k K e y [ T ] returns S e t t i n g [ T a s k [ T ] ] .
53. DEPENDENCY RESOLUTION IS SLOW The dependency resolution (ivy) is single threaded, and not cached by default; There are alternatives: coursier, a pure Scala dependency resolver, is around 5 times faster resolving dependencies in a multi-project (28 projects) build (5 vs 25 minutes).
54. SCALA CONS: IDE AND TOOLING
55. IDES IntelliJ IDEA; Scala IDE (for Eclipse); ENSIME (for Emacs, Atom, Vim and Sublime).
56. RAPIDLY EVOLVING ECOSYSTEM IDE plugins are not as sophisticated (particularly when compared with Java), so developers might get frustrated. Work on the Scala presentation compiler (used by Scala IDE and ENSIME) and other tools makes us look with optimism to the future.
57. QUESTIONS?