Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microservices with Clojure

Microservices with Clojure

Microservices are especially characterized by their autonomy: the services are as loosely coupled as possible. For this reason, a microservice architecture allows different technologies to be deployed that are exactly suitable for the problem which needs to be solved and the skill set of the software engineer. In such an environment, the benefits of a modern language like Clojure can be invaluable: short and precise code, concurrent processing, and direct support for asynchronous programming.

However, there are also particular challenges when dealing with microservices: How can a project be created with as little overhead as possible? How can services communicate with each other? How can I protect my service when other services fail? And once a service has been created, how can I deploy and monitor it?

In this talk we will introduce the participants to Clojure step by step and, with the help of an example, introduce the Clojure libraries which can be used in order to tackle the previously stated challenges.

Michael Vitz

April 14, 2016
Tweet

More Decks by Michael Vitz

Other Decks in Programming

Transcript

  1. Easier to understand $ m a n u n i

    q . . . D E S C R I P T I O N T h e u n i q u t i l i t y r e a d s t h e s p e c i f i e d i n p u t _ f i l e c o m p a r i n g a d j a c e n t l i n e s , a n d w r i t e s a c o p y o f e a c h u n i q u e i n p u t l i n e t o t h e o u t p u t _ f i l e . . . .
  2. Understanding a monolith... $ m a n m y -

    h u g e - m o n o l i t h < e n d l e s s t e x t >
  3. Composability p s - e f | g r e

    p p r o g r a m - i - d o n t - l i k e | g r e p - v g r e p | a w k ' { p r i n t $ 2 } ' | k i l l
  4. Don’t build thinks like “frontend”, “api” and “database access” –

    instead, build “fulfillment” and “payment”.
  5. Rules & conventions Project setup is done more than once

     has to be easy Deployment is done all the time  must be automated Many apps are configured and write logs  should be simple for every app, common format, Splunk or ELK
  6. Rules & conventions Many services are talking to each other

     standardize on protocols and data formats The network is not reliable  asynchronous programming and stability patterns for decoupling and resilience It’s easy to lose track of what’s actually happening  gather metrics for monitoring
  7. Clojure Crash Course ( p r i n t l

    n " H e l l o B e r l i n ! " ) Lisp + JVM Functional programming language Simple programming model
  8. Clojure Crash Course ( p r i n t l

    n " H e l l o B e r l i n ! " ) Lisp + JVM Functional programming language Simple programming model
  9. Clojure Crash Course { : n a m e "

    C l o j u r e " : f e a t u r e s [ : f u n c t i o n a l : j v m : p a r e n s ] : c r e a t o r " R i c h H i c k e y " : s t a b l e - v e r s i o n { : n u m b e r " 1 . 8 . 0 " : r e l e a s e " 2 0 1 6 / 0 1 / 1 9 " } }
  10. Clojure Crash Course ( d e f n m a

    i n [ a r g s ] ( p r i n t l n " H e l l o W o r l d ! " ) ) vs. p u b l i c s t a t i c v o i d m a i n ( S t r i n g [ ] a r g s ) { S y s t e m . o u t . p r i n t l n ( " H e l l o W o r l d ! " ) ; }
  11. Clojure Crash Course ( + 1 2 3 ) >

    6 ( : c i t y { : n a m e " i n n o Q " : c i t y " B e r l i n " } ) > " B e r l i n " ( m a p i n c [ 1 2 3 ] ) > ( 2 3 4 )
  12. Leiningen Alternative to Maven Describes Clojure project with generic data

    structures Maven repository compatibility Offers plugin system
  13. Leiningen ( d e f p r o j e

    c t s i m p l e - c a l e n d a r " 0 . 1 . 0 - S N A P S H O T " : d e s c r i p t i o n " s i m p l e c a l e n d a r a p p " : u r l " h t t p s : / / g i t h u b . c o m / i n n o q / s i m p l e - c a l e n d a r " : d e p e n d e n c i e s [ [ o r g . c l o j u r e / c l o j u r e " 1 . 8 . 0 " ] [ c o m p o j u r e " 1 . 4 . 0 " ] [ r i n g " 1 . 4 . 0 " ] [ r i n g / r i n g - j s o n " 0 . 4 . 0 " ] . . . [ e n v i r o n " 1 . 0 . 2 " ] ] : p l u g i n s [ [ l e i n - r i n g " 0 . 9 . 7 " ] [ l e i n - e n v i r o n " 1 . 0 . 2 " ] ] : r i n g { : h a n d l e r s i m p l e - c a l e n d a r . c o r e / w e b a p p : i n i t s i m p l e - c a l e n d a r . c o r e / i n i t } : p r o f i l e s { : u b e r j a r { : a o t : a l l } } )
  14. Automated deployment Not really a language thing ... ... but

    good tooling can help a lot Leiningen makes building fat JARs easy WAR files can be generated as well
  15. Logging Don’t let every app handle log files Just write

    everything to stdout Let some external tool handle storage Standardize on log format
  16. org.clojure/tools.logging Macros delegating to different implementations Supports slf4j, Apache commons-logging,

    log4j and java.util.logging Logging levels: : t r a c e : d e b u g : i n f o : w a r n : e r r o r : f a t a l
  17. org.clojure/tools.logging ( n s s i m p l e

    - c a l e n d a r . c o r e ( : r e q u i r e [ c l o j u r e . t o o l s . l o g g i n g : a s l o g ] ) ( d e f n u p d a t e - c o n t a c t ! [ u r l ] . . . ( l o g / i n f o " U p d a t e d u s e r " i d n e w - e m a i l ) ) p r o j e c t . c l j : : d e p e n d e n c i e s [ [ o r g . c l o j u r e / t o o l s . l o g g i n g " 0 . 3 . 1 " ] [ l o g 4 j " 1 . 2 . 1 7 " : e x c l u s i o n s [ j a v a x . m a i l / m a i l j a v a x . j m s / j m s c o m . s u n . j d m k / j m x t o o l s c o m . s u n . j m x / j m x r i ] ] [ o r g . s l f 4 j / s l f 4 j - l o g 4 j 1 2 " 1 . 7 . 1 8 " ] . . . ]
  18. org.clojure/tools.logging l o g 4 j . p r o

    p e r t i e s : l o g 4 j . r o o t L o g g e r = I N F O , s t a n d a r d l o g 4 j . a p p e n d e r . s t a n d a r d = o r g . a p a c h e . l o g 4 j . C o n s o l e A p p e n d e r l o g 4 j . a p p e n d e r . s t a n d a r d . T a r g e t = S y s t e m . o u t l o g 4 j . a p p e n d e r . s t a n d a r d . l a y o u t = o r g . a p a c h e . l o g 4 j . P a t t e r n L a y o u t l o g 4 j . a p p e n d e r . s t a n d a r d . l a y o u t . C o n v e r s i o n P a t t e r n = % d { y y y y - m m - d d H H : m m : s s , S S S } [ % p ] % c - % m % n s t d o u t : 2 0 1 5 - 1 1 - 1 8 1 3 : 1 1 : 5 4 , 4 6 8 [ I N F O ] s i m p l e - c a l e n d a r . c o r e - U p d a t e d u s e r 5 f 5 6 5 0 4 0 e v e @ e x a m p l e . o r g 2 0 1 5 - 1 1 - 1 8 1 3 : 1 1 : 5 4 , 4 7 6 [ I N F O ] s i m p l e - c a l e n d a r . c o r e - U p d a t e d u s e r 7 8 6 4 9 4 e f b o b @ e x a m p l e . o r g
  19. Configuration Store configuration in the environment, not in the codebase

    Use mechanism that works for every application, e.g., environment variables
  20. environ Manages environment settings following 12factor Reads values from Java

    system properties Environment variables . l e i n - e n v (created via Leiningen plugin from p r o f i l e s . c l j )
  21. environ ( d e f c o n t a

    c t s - f e e d ( e n v : c o n t a c t s - f e e d ) ) > j a v a - D c o n t a c t s . f e e d = h t t p : / / c o n t a c t s . e x a m p l e . o r g / f e e d - j a r s t a n d a l o n e . j a r > C O N T A C T S _ F E E D = h t t p : / / c o n t a c t s . e x a m p l e . o r g / f e e d l e i n r i n g s e r v e r - h e a d l e s s > l e i n r i n g s e r v e r - h e a d l e s s p r o f i l e s . c l j : { : d e v { : e n v { : c o n t a c t s - f e e d " h t t p : / / c o n t a c t s . e x a m p l e . o r g / f e e d " } } }
  22. HTTP server Ring for HTTP basics Compojure for routing Request

    & response are data A web app is a function which takes a request and returns a response https://github.com/ring-clojure/ring/blob/master/SPEC
  23. Ring ( d e f e x a m p

    l e - r e q u e s t { : u r i " / c o n t a c t s " : r e q u e s t - m e t h o d : g e t : h e a d e r s { " A c c e p t " " t e x t / p l a i n " } } ) ( d e f n e x a m p l e - a p p [ r e q ] { : s t a t u s 2 0 0 : b o d y ( s t r " H e l l o a t " ( : u r i r e q ) ) } ) ( e x a m p l e - a p p e x a m p l e - r e q u e s t ) > { : s t a t u s 2 0 0 : b o d y " H e l l o a t / c o n t a c t s " }
  24. Compojure ( d e f r o u t e

    s c o n t a c t s ( G E T " / c o n t a c t s / : i d " [ i d ] ( g e t - c o n t a c t i d ) ) ( P U T " / c o n t a c t s / : i d " [ i d : a s r e q u e s t ] ( u p d a t e - c o n t a c t ! i d ( : b o d y r e q u e s t ) ) ) ( P O S T " / c o n t a c t s " r e q u e s t ( a d d - c o n t a c t ! ( : b o d y r e q u e s t ) ) ) ( G E T " / f e e d " [ ] { : s t a t u s 2 0 0 : h e a d e r s { " C o n t e n t - T y p e " " a p p l i c a t i o n / a t o m + x m l " } : b o d y ( c o n t a c t s - f e e d ) } ) ) )
  25. Frameworks Quickly get a web app up and running. Duct

    Luminus Pedestal Modularity tesla-microservices
  26. org.clojure/data.json Clojure data structures are JSON superset ( j s

    o n / w r i t e - s t r { : n a m e " A l i c e M i l l e r " : t e a m s [ " T e a m A " , " T e a m B " ] } ) > " { \ " n a m e \ " : \ " A l i c e M i l l e r \ " , \ " t e a m s \ " : [ \ " T e a m A \ " , \ " T e a m B \ " ] } " ( j s o n / r e a d - s t r " { \ " i d \ " : 1 2 3 , \ " n a m e \ " : \ " A l i c e M i l l e r \ " } " : k e y - f n k e y w o r d ) > { : i d 1 2 3 , : n a m e " A l i c e M i l l e r " }
  27. HTML HTML is represented with generic data structures This allows

    processing HTML with standard Clojure functions There are different formats, a popular one is called “hiccup”
  28. HTML < h t m l > < b o

    d y > < h 1 > h e l l o ! < / h 1 > < i m g s r c = " s o m e . j p g " / > < i m g s r c = " a n o t h e r . j p g " / > < / b o d y > < / h t m l > [ : h t m l { } [ : b o d y { } [ : h 1 { } " h e l l o ! " ] [ : i m g { : s r c " s o m e . j p g " } ] [ : i m g { : s r c " a n o t h e r . j p g " } ] ] ]
  29. Creating Atom Feeds ( d e f n e n

    t r y [ e v e n t ] [ : e n t r y [ : t i t l e ( - > e v e n t : t y p e n a m e ) ] [ : u p d a t e d ( : t i m e s t a m p e v e n t ) ] [ : a u t h o r [ : n a m e " c o n t a c t s s e r v i c e " ] ] [ : i d ( s t r " u r n : c o n t a c t s : f e e d : e v e n t : " ( : i d e v e n t ) ) ] [ : c o n t e n t { : t y p e " j s o n " } ( j s o n / g e n e r a t e - s t r i n g e v e n t ) ] ] ) ( d e f n a t o m - f e e d [ e v e n t s u r l ] ( c l o j u r e . d a t a . x m l / e m i t - s t r ( c l o j u r e . d a t a . x m l / s e x p - a s - e l e m e n t [ : f e e d { : x m l n s " h t t p : / / w w w . w 3 . o r g / 2 0 0 5 / A t o m " } [ : i d " u r n : c o n t a c t s : f e e d " ] [ : u p d a t e d ( - > e v e n t s l a s t : t i m e s t a m p ) ] [ : t i t l e { : t y p e " t e x t " } " c o n t a c t s e v e n t s " ] [ : l i n k { : r e l " s e l f " : h r e f u r l } ] ( m a p e n t r y e v e n t s ) ] ) ) )
  30. adamwynne/feedparser-clj Retrieves and Parses RSS/Atom feeds ( d e f

    f ( f e e d p a r s e r / p a r s e - f e e d " h t t p s : / / w w w . i n n o q . c o m / d e / p o d c a s t . r s s " ) ) ( : t i t l e f ) > " i n n o Q P o d c a s t " ( c o u n t ( : e n t r i e s f ) ) > 1 8 Library for consuming feeds based on feedparser: Feedworker
  31. Asynchronous communication Helps to decouple and separate different application parts

    E-mail notifications do not need to be sent synchronously
  32. org.clojure/core.async Supports asynchronous programming and communications Messages can be sent

    to and read from channels Channels can be buffered or unbuffered Blocking and non-blocking operations possible
  33. org.clojure/core.async ( d e f n o t i f

    i c a t i o n s ( c h a n 1 0 0 0 ) )
  34. org.clojure/core.async ( d e f n o t i f

    i c a t i o n s ( c h a n 1 0 0 0 ) )
  35. org.clojure/core.async ( d e f n o t i f

    i c a t i o n s ( c h a n 1 0 0 0 ) ) ( d e f n s e n d - n o t i f i c a t i o n [ e m a i l e v e n t - l i n k ] ( g o ( > ! n o t i f i c a t i o n s { : e m a i l e m a i l : e v e n t - l i n k e v e n t - l i n k } ) ) )
  36. org.clojure/core.async ( d e f n o t i f

    i c a t i o n s ( c h a n 1 0 0 0 ) ) ( d e f n s e n d - n o t i f i c a t i o n [ e m a i l e v e n t - l i n k ] ( g o ( > ! n o t i f i c a t i o n s { : e m a i l e m a i l : e v e n t - l i n k e v e n t - l i n k } ) ) )
  37. org.clojure/core.async ( d e f n o t i f

    i c a t i o n s ( c h a n 1 0 0 0 ) ) ( d e f n s e n d - n o t i f i c a t i o n [ e m a i l e v e n t - l i n k ] ( g o ( > ! n o t i f i c a t i o n s { : e m a i l e m a i l : e v e n t - l i n k e v e n t - l i n k } ) ) )
  38. org.clojure/core.async ( d e f n o t i f

    i c a t i o n s ( c h a n 1 0 0 0 ) ) ( d e f n s e n d - n o t i f i c a t i o n [ e m a i l e v e n t - l i n k ] ( g o ( > ! n o t i f i c a t i o n s { : e m a i l e m a i l : e v e n t - l i n k e v e n t - l i n k } ) ) ) ( d e f n n o t i f y - u s e r [ e m a i l e v e n t - l i n k ] . . . ) ( d e f n s t a r t - n o t i f i e r [ ] ( g o - l o o p [ m e s s a g e ( < ! n o t i f i c a t i o n s ) ] ( n o t i f y - u s e r ( : e m a i l m e s s a g e ) ( : e v e n t - l i n k m e s s a g e ) ) ( r e c u r ( < ! n o t i f i c a t i o n s ) ) ) )
  39. org.clojure/core.async ( d e f n o t i f

    i c a t i o n s ( c h a n 1 0 0 0 ) ) ( d e f n s e n d - n o t i f i c a t i o n [ e m a i l e v e n t - l i n k ] ( g o ( > ! n o t i f i c a t i o n s { : e m a i l e m a i l : e v e n t - l i n k e v e n t - l i n k } ) ) ) ( d e f n n o t i f y - u s e r [ e m a i l e v e n t - l i n k ] . . . ) ( d e f n s t a r t - n o t i f i e r [ ] ( g o - l o o p [ m e s s a g e ( < ! n o t i f i c a t i o n s ) ] ( n o t i f y - u s e r ( : e m a i l m e s s a g e ) ( : e v e n t - l i n k m e s s a g e ) ) ( r e c u r ( < ! n o t i f i c a t i o n s ) ) ) )
  40. org.clojure/core.async ( d e f n o t i f

    i c a t i o n s ( c h a n 1 0 0 0 ) ) ( d e f n s e n d - n o t i f i c a t i o n [ e m a i l e v e n t - l i n k ] ( g o ( > ! n o t i f i c a t i o n s { : e m a i l e m a i l : e v e n t - l i n k e v e n t - l i n k } ) ) ) ( d e f n n o t i f y - u s e r [ e m a i l e v e n t - l i n k ] . . . ) ( d e f n s t a r t - n o t i f i e r [ ] ( g o - l o o p [ m e s s a g e ( < ! n o t i f i c a t i o n s ) ] ( n o t i f y - u s e r ( : e m a i l m e s s a g e ) ( : e v e n t - l i n k m e s s a g e ) ) ( r e c u r ( < ! n o t i f i c a t i o n s ) ) ) )
  41. com.netflix.hystrix/hystrix-clj Idiomatic Clojure wrapper for Hystrix ( h y s

    t r i x / d e f c o m m a n d n o t i f y - u s e r [ e m a i l e v e n t - l i n k ] ( c l i e n t / p o s t n o t i f i c a t i o n - s e r v i c e { : c o n t e n t - t y p e : j s o n : b o d y . . . } ) ) Will throw exception in case of timeout or other failure
  42. com.netflix.hystrix/hystrix-clj ; r e t u r n s t

    r u e i f s u c e s s f u l , f a l s e i f c i r c u i t - b r e a k e r o p e n ( h y s t r i x / d e f c o m m a n d n o t i f y - u s e r { : h y s t r i x / f a l l b a c k - f n n o t i f y - f a l l b a c k } [ e m a i l e v e n t - l i n k ] ( c l i e n t / p o s t n o t i f i c a t i o n - s e r v i c e { : c o n t e n t - t y p e : j s o n : b o d y . . . } ) t r u e ) ; r e t u r n s f a l s e i f c i r c u i t - b r e a k e r o p e n ( d e f n n o t i f y - f a l l b a c k [ e m a i l e v e n t - l i n k ] ( l e t [ i s O p e n ( . i s C i r c u i t B r e a k e r O p e n h y s t r i x / * c o m m a n d * ) ] ; a d d m e s s a g e t o q u e u e a g a i n ( s e n d - n o t i f i c a t i o n e m a i l e v e n t - l i n k ) ( n o t i s O p e n ) ) )
  43. com.netflix.hystrix/hystrix-clj ( d e f n s t a r

    t - n o t i f i e r [ ] ( g o - l o o p [ m e s s a g e ( < ! n o t i f i c a t i o n s ) ] ( i f - n o t ( n o t i f y - u s e r ( : e m a i l m e s s a g e ) ( : e v e n t - l i n k m e s s a g e ) ) ( d o ( l o g / e r r o r " C a n n o t r e a c h n o t i f i c a t i o n s e r v i c e " " - w i l l w a i t u n t i l n e x t t r y " ) ( < ! ( t i m e o u t 5 0 0 0 ) ) ) ) ( r e c u r ( < ! n o t i f i c a t i o n s ) ) ) ) hystrix-event-stream-clj available as well
  44. Questions How many HTTP errors are occurring? Are database queries

    failing? Is that backend service slow again? How many jobs are in the queue?
  45. Metrics Logging provides a stream of events Metrics provide aggregated

    state Popular library: Dropwizard Metrics Two steps: collect metrics, then publish them
  46. Gauges Current state of one single value Number of jobs

    in the queue Some configured value Ratio of cache hits to misses
  47. Gauges ( d e f m e t r i

    c s - r e g i s t r y ( n e w - r e g i s t r y ) ) ( g a u g e - f n m e t r i c s - r e g i s t r y " j o b s - r e a d y " # ( q u e r y d a t a b a s e " s e l e c t c o u n t ( * ) f r o m j o b s w h e r e s t a t u s = ' r e a d y ' " ) )
  48. Counters derivative of by “The US National Debt clock /

    counter, New York” Ben Sutherland (CC BY 2.0)
  49. Counters ( d e f l o g g e

    d - i n - u s e r s ( c o u n t e r m e t r i c s - r e g i s t r y " l o g g e d - i n - u s e r s " ) ) ( P O S T " / l o g i n " [ n a m e p w d ] . . . ( i n c ! l o g g e d - i n - u s e r s ) . . . ) ( P O S T " / l o g o u t " [ n a m e ] . . . ( d e c ! l o g g e d - i n - u s e r s ) . . . )
  50. Histograms derivative of by “Number of cat posts on Metafilter

    per month, as a percentage of the total number of posts” Steven Taschuk (CC BY 2.0)
  51. Histograms Distribution of numerical data (min, max, mean, standard deviation,

    quantiles) For example, number of search results Different value “reservoirs”, e.g., Entire application lifetime Last N searches Last N minutes
  52. Histograms ( d e f n u m b e

    r - o f - r e s u l t s ( h i s t o g r a m m e t r i c s - r e g i s t r y " n u m b e r - o f - s e a r c h - r e s u l t s " ) ) ( d e f n s e a r c h [ q u e r y ] ( l e t [ r e s u l t s ( e x e c u t e q u e r y ) ] ( u p d a t e ! n u m b e r - o f - r e s u l t s ( c o u n t r e s u l t s ) ) r e s u l t s ) )
  53. Meters Rate of an event (per second) Total count of

    events Average rate over application lifetime Rate in the last 1, 5 and 15 minutes For example, incoming requests
  54. Meters ( d e f i n c - r

    e q u e s t s - m e t e r ( m e t e r s / m e t e r m e t r i c s - r e g i s t r y " i n c o m i n g - r e q u e s t s - m e t e r " ) ) ( d e f n r e q u e s t s - m e t e r [ h a n d l e r ] ( f n [ r e q ] ( m e t e r s / m a r k ! i n c - r e q u e s t s - m e t e r ) ( h a n d l e r r e q ) ) )
  55. Timers Histogram & meter Histogram of duration of an activity,

    meter of occurrence of the activity For example, specific type of database query
  56. Timers ( d e f i n c - r

    e q u e s t s - t i m e r ( t i m e r s / t i m e r ! m e t r i c s - r e g i s t r y " i n c o m i n g - r e q u e s t s - t i m e r " ) ) ( d e f n r e q u e s t s - t i m e r [ h a n d l e r ] ( f n [ r e q ] ( t i m e r s / t i m e ! i n c - r e q u e s t s - t i m e r ( h a n d l e r r e q ) ) ) )
  57. Reporting Metrics stores current state in its registry Should be

    reported to external tool Visualization Historical data Reporters for console, JMX, Ganglia, Graphite, CSV etc. included
  58. Conclusion Organize around business capabilities Rules & conventions needed Standardize

    interfaces, logging and configuration Different solutions to improve stability Monitor your systems Good support in Clojure