Workshop. Building Scala Microservice

Building Scala micros-service using Typelevel stack and other libraries

Transcript

1. Building Scala Microservices using Typelevel-Stack LXScala Lisbon Alexey Novakov, Ultra Tendency
2. Prerequisite Install: SBT Git Docker & Docker-compose IDE for Scala git clone https://github.com/novakov-alexey/workshop-scala-msi-tpl.git sbt compile test:compile
3. About me - 4yrs in Scala, 10yrs in Java - a fan of Data Processing, - Distributed Systems, - Functional Programming Olá, I am Alexey! I am working at Ultra Tendency
4. Plan for today 1. Intro to Scala stack and Project requirements 2. HTTP Server 3. HTTP Routes, JSON Codecs 4. Service Logic 5. Configuration 6. Data Layer 7. Testing 8. Dependency Injection 9. SBT release, native-packger (Docker) 10.Q & A exercises exercises
5. Nice to know what is: 1. Functional Patterns: Functor, Applicative, Monad, MonadError (Cats), IO Monad 2. Referencial transparency in FP 3. Pure vs. Impure function 4. Encoding of Type Classes in Scala 5. Lazy vs. Eager evaluation
6. G o a l o f t h e Workshop - inspire you to write functional code in Scala - be familiar with some of the Typelevel libraries as your building blocks Not a goal: - learn how to write micro services - complete all exercises
7. Typelevel Others Lightbend Origin of the libraries
8. Cats, Shapeless
9. (★ GitHub Stars) ★
17. Let’s build a typical microservice
19. REST API Service Architecture Domain Model Postgres SQL HTTP Client
20. Add: POST /api/v1/trips, body = JSON Update: PUT /api/v1/trips/<id>, body = JSON Delete: DELETE /api/v1/trips/<id> Select all: GET /api/v1/trips?sort=id&page=1&pageSize=100 Select one: GET /api/v1/trips/<id> Ride booking API
21. Models (Protocol) final case class Trip( id: Int, city: String, vehicle: Vehicle, price: Int, completed: Boolean, distance: Option[Int], endDate: Option[LocalDate] ) final case class Trips(trips: Seq[Trip]) object Vehicle extends Enumeration { type Vehicle = Value val Bike, Taxi, Car = Value } final case class CommandResult(count: Int)
22. Task #1 http4s server, logging
23. HTTP Server - http4s - Components for the workshop: - http4s is built with cats, cats-effect, fs2-io "org.http4s" %% "http4s-blaze-server" % Version "org.http4s" %% "http4s-blaze-client" % Version % Test “org.http4s" %% "http4s-circe" % Version "org.http4s" %% "http4s-dsl" % Version blaze project is NIO micro framework
24. http4s-dsl to define a route val routes: HttpRoutes[F] = HttpRoutes.of[F] { case GET -> Root / IntVar(id) => … Ok(Trip(id, "lisbon", Vehicle.Bike, 2000, completed = false, None, None)) } // Select one: GET /api/v1/trips/<id> class MyRoutes[F[_]: Sync] extends Http4sDsl[F] … import cats.effect.Sync, cats.implicits._ import org.http4s.HttpRoutes, org.http4s.dsl.Http4sDsl
25. routes: pattern matching val routes: HttpRoutes[F] = HttpRoutes.of[F] { case GET -> Root / IntVar(id) => // http4s-dsl provides a shortcut to create an F[Response] for Status Code Ok(…) case req @ POST -> Root / “some_string” / someVal => // do something with request object req.as[MyCaseClass].flatMap( _ => Ok(…)) case PUT -> Root / “some_string” / someVal => Ok(…) } … sequence of functions Docs: https://http4s.org/v0.20/dsl/
26. Cats: IOApp import cats.effect._, cats.implicits._ object Main extends IOApp { override def run(args: List[String]): IO[ExitCode] = ??? } Docs: https://typelevel.org/cats-effect/datatypes/ioapp.html - IOApp - is an entry point to a pure FP program - brings ContextShift[IO] and Timer[IO] - provides interruption handler
27. start server import cats.effect._, cats.implicits._ import fs2.Stream import org.http4s.implicits._ import org.http4s.server.blaze.BlazeServerBuilder import org.http4s.server.middleware.Logger object Main extends IOApp { override def run(args: List[String]): IO[ExitCode] = stream[IO].compile.drain.as[ExitCode.Success] def stream[F[_]: ConcurrentEffect: Timer]: Stream[F, ExitCode] = { val httpApp = new MyRoutes[F]().routes.orNotFound val finalHttpApp = Logger.httpApp(logHeaders = true, logBody = true)(httpApp) BlazeServerBuilder[F] .bindHttp(8080, "localhost") .withHttpApp(finalHttpApp) .serve } } takes our routes starts a server process converts to IO, which never ends
28. Routes - An HttpRoutes[F] is a type alias for: Kleisli[OptionT[F, ?], Request[F], Response[F]] Kleisli: A => F[B] def of[F[_]](pf: PartialFunction[Request[F], F[Response[F]]])( implicit F: Sync[F]): HttpRoutes[F] - HttpRoutes.of definition: - Eventually, we will use cats.effect.IO as effect type for F
29. Cats Effect: IO - Describes async and sync computation - Returns exactly one result - Can end either in success or failure - Can be canceled* Docs: https://typelevel.org/cats- effect/datatypes/io.html scala> import cats.effect.IO import cats.effect.IO scala> val ioa = IO { println("hey!") } ioa: cats.effect.IO[Unit] = IO$307547358 scala> val program: IO[Unit] = for { _ <- ioa _ <- ioa } yield () program: cats.effect.IO[Unit] = IO$243763162 scala> program.unsafeRunSync() hey! hey!
30. Cats Effect: Sync trait Sync[F[_]] extends Bracket[F, Throwable] with Defer[F] { def suspend[A](thunk: => F[A]): F[A] def delay[A](thunk: => A): F[A] = suspend(pure(thunk)) } A Monad that can suspend the execution of side effects in the F[_] context.
31. Scala Logging - It wraps slf4j and requires logging backend library like logback - Usage: class MyClass extends LazyLogging { logger.debug("This is very convenient ;-)”) … } “com.typesafe.scala-logging” %% scala-logging” if (logger.isDebugEnabled) logger.debug(s"Some $expensive message!") - check-enabled-idiom is applied automatically by Scala macros
32. Task #1 1.1. implement HTTP routes • in org.scalamsi.http.QueryRoutes add a route for GET /ping to Ok(“pong”) and leave it assigned to val routes: HttpRoutes[F]. See examples in the code. • instantiate QueryRoutes in org.scalamsi.Module and assign it to val routes: HttpRoutes[F] using such code to combine Uri prefix with a route: Router(apiPrefix -> (qr.routes))
33. Task #1 cont. 1.2. In Main.scala, implement run function 1.3 start HTTP server via SBT 1.4. use scala-logging in your Main or Module classes This can be done by extending or mixing-in StrictLogging or LazyLogging traits into your class Once mixed in, a logger instance is in your scope, so just use it to write some useful info message First, extend Main object from IOApp. Override its run method. Implement it by calling stream function with IO as effect type and then call .compile.drain.as(ExitCode.Success) Check Task1 via test: sbt "testOnly *Task1Test” - run shell command: sbt run. Console output should report that web-server is started - (optional check) use curl to call the ping route to get pong reply back: curl localhost:8080/api/v1/trips/ping - stop sbt via Ctrl+C
34. Task #2 http4s all routes, json codecs
35. Add: POST /api/v1/trips, body = JSON Update: PUT /api/v1/trips/<id>, body = JSON Delete: DELETE /api/v1/trips/<id> Select all: GET /api/v1/trips?sort=id&page=1&pageSize=100 Select one: GET /api/v1/trips/<id> Ride booking API
36. Circe - JSON library for Scala - built with Cats - supports JVM and Scala.js - allows to parse JSON text into JSON AST, Traverse JSON - Codecs can be derived automatically, semi-automatically or be written manually
37. https://blog.softwaremill.com/scalar-2019-whiteboard-voting-40b31e4f7f7
38. Task #2 2.1. Add Circe codecs for protocol case classes • in org.scalamsi.json.CirceJsonCodecs, add implicit Encoder & Decoder instances - Trip, Trips and CommandResult In order to add these instances just leverage circe functions import io.circe.generic.semiauto.{deriveDecoder, deriveEncoder} • mixin org.scalamsi.json.CirceJsonCodecs into QueryRoutes & CommandRoutes Docs: https://circe.github.io/circe/codecs/semiauto-derivation.html
39. Task #2 cont. 2.2. implement 5 routes of the Ride API • in - org.scalamsi.http.QueryRoutes (2 routes) - org.scalamsi.http.CommandRoutes (3 routes) assign them to val routes::HttpRoutes[F]. See examples in the code • in org.scalamsi.Module, instantiate QueryRoutes and CommandRoutes • assign both to val routes: HttpRoutes[F] using below code to combine them: Router(apiPrefix -> (qr.routes <+> cr.routes)) Check Task2 via test: sbt "testOnly *Task2Test" Docs: https://typelevel.org/cats/typeclasses/semigroupk.html
40. Task #3 service logic
41. Service Algebra trait TripServiceAlg[F[_]] { def selectAll(page: Option[Int], pageSize: Option[Int], sort: Option[String]): F[Trips] def select(id: Int): F[Option[Trip]] def insert(trip: Trip): F[Int] def update(id: Int, trip: Trip): F[Int] def delete(id: Int): F[Int] } F[_] - can be later set to scala.Future*, monix.Task, my.IO, Id, etc.
42. Generic Types: Why? RequestContext 㱺 Future[RouteResult] TripServiceAlg[F[_]] Repository[F[_]] TripRepo[Future] TripService[Future] Slick Akka HTTP, Play Implementation Abstraction layer: framework free, library free, Request 㱺 IO[Response] TripService[IO] TripRepo[IO] Doobie http4s
43. Service Interpreter def selectAll(page: Option[Int], pageSize: Option[Int], sort: Option[String]): F[Trips] = { ….. repo .selectAll(pageN, size, sort) // <— F[Seq[Trip]] .map(Trips) } import cats.Functor import cats.syntax.functor._ class TripService[F[_]](repo: Repository[F])(implicit F: Functor[F]) { Docs: https://typelevel.org/cats/typeclasses/functor.html
44. Algebra definition or DSL as parameterised trait Interpreter as class/object implementation This technique is called - Tagless Final See more in this short and nice blogpost: https:// www.basementcrowd.com/2019/01/17/an-introduction-to-tagless-final-in- scala/
45. Task #3 implement TripService Service logic is mainly delegating everything to the Repository level and doing a few input parameter validation. 1. Complete selectAll method implementation by flatMapping the sorting field and calling the selectAll on the repository. Try to use for-comprehension here 2. Implement insert and update method: use existing validate method and then call Repository method to insert or update, if the input data is valid 3. in Module.scala, instantiate real TripService and assign it to val service. Check Task3 via test: sbt "testOnly *Task3Test”
46. Task 4 Configuration
47. Configuration: HOCON foo { bar = 10 baz = 12 } foo.bar =10 foo.baz =12 Or Example: Human-Optimized Config Object Notation Typesafe Config library import com.typesafe.config.ConfigFactory val conf = ConfigFactory.load() val bar1 = conf.getInt("foo.bar") val foo = conf.getConfig("foo") val bar2 = foo.getInt("bar")
48. server { host = localhost port = 8080 } storage { host = localhost port = 5432 dbName = trips url = "jdbc:postgresql://"${storage.host}":"${storage.port}"/"${storage.dbName} driver = "org.postgresql.Driver" user = “trips" password = "trips" } application.conf
49. import pureconfig.loadConfig import pureconfig.generic.auto._ val path = … // path to application.conf def load: Either[ConfigReaderFailures, Server] = { loadConfig[Server](Paths.get(path), “server”) } “com.github.pureconfig:pureconfig" final case class Server( host: String, port: Int ) server { host = localhost port = 8080 } application.conf: Docs: https://pureconfig.github.io/docs/index.html
50. “eu.timepit:refined-pureconfig" import eu.timepit.refined.types.net.UserPortNumber // from 1024 to 49151 import eu.timepit.refined.types.string.NonEmptyString import eu.timepit.refined.pureconfig._ [info] Compiling 2 Scala sources to /Users/aa/dev/git/akka-crud-service/target/scala-2.12/classes ... [error] /Users/aa/dev/git/akka-crud-service/src/main/scala/org/alexeyn/configs.scala:39:84: Left predicate of (!(808 < 1024) && !(808 > 49151)) failed: Predicate (808 < 1024) did not fail. [error] final case class Server(host: NonEmptyString = "localhost", port: UserPortNumber = 808) final case class Server( host: NonEmptyString = "localhost", port: UserPortNumber = 808 ) invalid literal value Docs: https://github.com/fthomas/refined
51. Task #4 implement refined types for JdbcConfig a) in configs.scala, define two more types inside the object refined: 1. A type for JdbcConfig#maximumPoolSize as Int range from 1 to 100. See example of ConnectionTimeout type 2. JdbcConfig#url property, using regexp: jdbc:\\w+://\\w+:[0-9]{4,5}/\\w+ Example: type MyType = String Refined MatchesRegex[W.`""" REG EXP HERE """`.T] b) for properties: driver, user, password use non empty String type c) for connection timeout use the existing type, which is already defined as reference d) in Module.scala, fix the parameters of Transactor.fromDriverManager, by accessing values of every parameter using .value mehtod on each of them. For example: cfg.driver.value Check Task4 via test: sbt "testOnly *Task4Test”
52. Task 5 Data Layer
53. Data Layer trait Repository[F[_]] { def delete(id: Int) : F[Int] def update(id: Int, row: Trip): F[Int] def createSchema(): F[Unit] def schemaExists(): F[Unit] def insert(row: Trip): F[Int] def selectAll(page: Int, pageSize: Int, sort: String): F[Seq[Trip]] def select(id: Int): F[Option[Trip]] }
54. Meet Doobie override def select(id: Int): F[Option[Trip]] = sql"SELECT * FROM trips WHERE id = $id" .query[Trip] .option .transact(xa) select: doobie is a pure functional JDBC layer for Scala and Cats. It is not an ORM class TripRepository[F[_]: Sync](xa: Transactor[F]) extends Repository[F] import doobie._ import doobie.implicits._ Docs: https://tpolecat.github.io/doobie/docs/03-Connecting.html
55. Doobie val xa = Transactor.fromDriverManager[F]( cfg.driver.value, cfg.url.value, cfg.user.value, cfg.password.value) implicit val cs = IO.contextShift(ExecutionContext.global) resources: F[_]: Async: ContextShift A Transactor is a data type that knows how to work* with database example for cats-effect IO Docs: https://tpolecat.github.io/doobie/docs/14-Managing-Connections.html
56. Doobie insert: override def insert(row: Trip): F[Int] = { val values = fr"VALUES (${row.id}, ${row.city}, ${row.vehicle}, ${row.price}, " ++ fr"${row.completed}, ${row.distance}, ${row.endDate})" (insertFrag ++ values).update.run.transact(xa) } val insertFrag: Fragment = fr"INSERT INTO trips (id, city, vehicle, price, completed, distance, end_date)" Docs: https://tpolecat.github.io/doobie/docs/07-Updating.html
57. It’s the first year when Doobie overtook Slick! Scalar 2019:
58. Task #5 implement TripRepository & TripRepositoryQueries 1. in TripRepositoryQueries#selectAllQuery write a constant query and then use it inside the TripRepository#selectAll. After calling selectAllQuery call .stream then .drop, . take, compile, to[Seq] and finally transact(xa). In such way, we applied page and pageSize parameters to return a particular offset from the database using streaming approach. 2. in TripRepositoryQueries#updateQuery, write a query by concatenating TripRepositoryQueries#updateFrag with a query fragments for “values” and “predicate”. Use ++ method for concatenation and call .update on the result Check Task5 via test: sh start-dev-db.sh && sbt "testOnly *Task5Test”
59. Task 6 Testing
60. Testing - Scalatest: - path: HTTP Routes Application Mock Database - Scalatest + testcontainers-scala - to test integration - path: HTTP Routes Application Postgres Container - Note libraries like Mockito can be usefull as well, but it often it is OK to write mock manually. Thanks to composition of Scala programs and FP
61. val routes = Router(Module.apiPrefix -> (qr.routes <+> cr.routes)).orNotFound Testing Application val req = Request[IO](method = Method.GET, uri = getUri(s"/${lisbon.id}")) val res = routes.run(req) implicit ev: EntityDecoder[IO, Trip] res.as[Trip].unsafeRunSync should ===(lisbon) import cats.effect.IO import org.http4s.EntityDecoder val lisbon = Trip(3, "lisbon", Vehicle.Bike, 2000, completed = false, None, None) create request & run it via routes pass decoder to convert the response no test kit is needed, pure FP
62. testcontainers-scala - wraps Java library Test Containers - any Docker image can be used for tests - provides common containers for databases and Selenium - integrated with Scalatest using 2 traits: ForEachTestContainer, ForEachTestContainer Test lifecycle: Start container After Start hook Run test(s) Before Stop hook Stop container Docs: https://github.com/testcontainers/testcontainers-scala
63. class E2ETest extends FlatSpec with ForAllTestContainer { override val container = GenericContainer("nginx:latest", exposedPorts = Seq(80), waitStrategy = Wait.forHttp("/") ) Testing Application with DB waitStrategy can be customised to wait for a specific port, docker health check or a particular line in the log it should “call nginx” in { val port: Int = container.mappedPort(80) val ipAddress: String = container.containerIpAddress …. } override val container = MySQLContainer() // or use one of the few predefined containers Docs: https://www.testcontainers.org/features/startup_and_waits/
64. Postgres Docker container has been started by Scalatest
65. Task #6 implement Task6Test 1. in Task6Test.scala mix in ForAllTestContainer or ForEachTestContainer trait and override container property with out-of-the-box Postgres container class. Set image name to postgres:10.4 2. use container property to initialise dbProps map 3. Note: mod variable should be used either lazily or be used inside the test case code block 4. in Module.scala, remove overriden createSchema() method in val repo implementation. We need to use real method, not a stub Check Task6 via test: sbt "testOnly *Task6Test”
66. Task 7 Dependency Injection
67. https://blog.softwaremill.com/scalar-2019-whiteboard-voting-40b31e4f7f7
68. DI with MacWire MacWire generates new instance creation code of given classes, using values in the enclosing type for constructor parameters, with the help of Scala Macros. trait UserModule { lazy val databaseAccess = new DatabaseAccess() lazy val securityFilter = new SecurityFilter() lazy val userFinder = new UserFinder(databaseAccess, securityFilter) lazy val userStatusReader = new UserStatusReader(userFinder) } trait UserModule { import com.softwaremill.macwire._ lazy val databaseAccess = wire[DatabaseAccess] lazy val securityFilter = wire[SecurityFilter] lazy val userFinder = wire[UserFinder] lazy val userStatusReader = wire[UserStatusReader]} compile-time checked macwire version
69. MacWire: factory - supports factories: class TaxCalculator(taxBase: Double, taxDeductionLibrary: TaxDeductionLibrary) def taxCalculator(taxBase: Double) = wire[TaxCalculator] // or: lazy val taxCalculator = (taxBase: Double) => wire[TaxCalculator] class TripService[F[_]](repo: Repository[F]) {} val service = wire[TripService[Future]] private def createStubRepo: Repository[Future] = ??? and factory methods:
70. MacWire: other features - Akka, Play integration - Works with Scala.js - Module composition - Scopes: singleton (val), dependant (def), request/session - Dynamic lookup (wired.lookup(classOf[DatabaseConnector])) - Interceptors - Qualifiers case class Basket(blueberry: Berry @@ Blue, blackberry: Berry @@ Black) lazy val blueberry = wire[Berry].taggedWith[Blue] lazy val blackberry = wire[Berry].taggedWith[Black] lazy val basket = wire[Basket]
71. Task #7 implement Module.scala - in Module.scala use MacWire’s wire function to instantiate all values instead of using “new” operator. It should be around 5 values wired with MacWire Check Task7 via test: sbt "testOnly *Task7Test” Also run previous test to check it is not broken: sbt "testOnly *Task6Test” Docs: https://github.com/softwaremill/macwire
72. Task 8 SBT Plugins
73. sbt-native-packager - builds application packages in native formats SbtNativePackager + | | +-------+ Universal +--------+-------------+----------------+ | + | | | | | | | | | | | | | + + + + + Docker +-+ Linux +-+ Windows JDKPackager GraalVM native-image | | | | + + Debian RPM
74. Docker Format + Java App Archetype plugins.sbt: addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % “x.y.z”) build.sbt: enablePlugins(JavaAppPackaging) FROM openjdk:8 WORKDIR /opt/docker ADD --chown=daemon:daemon opt /opt USER daemon ENTRYPOINT ["/opt/docker/bin/akka-crud-service"] CMD [] $ sbt stage $ sbt docker:publishLocal
75. Task #8 1. configure SBT native-packager plugin - in build.sbt of root module, use another docker image such as “openjdk:8-jre- alpine” and “AshScriptPlugin” to generate “busybox" compatible image shell script. AshScriptPlugin is provided by native-packager-plugin, so just use it Check by building and running Docker image: sbt docker:publishLocal docker stop postgres & docker rm postgres ./start.sh 1 postgres and 1 scala service containers should be started https://www.scala-sbt.org/sbt-native-packager/archetypes/misc_archetypes.html
76. sbt-release plugin plugins.sbt: addSbtPlugin(“com.github.gseitz" % "sbt-release" % “x.y.z”) Easy to bump version, publish artefacts, pusg to Git and more 1.2.1-SNAPSHOT -> 1.2.1 -> 1.2.2-SNAPSHOT - provides a customisable release process - interactive and non-interactive mode - versioning customisation
77. release steps releaseProcess := Seq[ReleaseStep]( checkSnapshotDependencies, // : ReleaseStep inquireVersions, // : ReleaseStep runClean, // : ReleaseStep runTest, // : ReleaseStep setReleaseVersion, // : ReleaseStep commitReleaseVersion, // : ReleaseStep, performs the initial git checks tagRelease, // : ReleaseStep publishArtifacts, // : ReleaseStep, checks whether `publishTo` is properly set up setNextVersion, // : ReleaseStep commitNextVersion, // : ReleaseStep pushChanges // : ReleaseStep, also checks that an upstream branch is properly configured ) Docs: https://github.com/sbt/sbt- release#release-process
78. Task #8 2. configure SBT release plugin 2. 1. First, commit all the changes: git add -A && git commit -m "ready for release” 2.2. release current project version and bump version to the next snapshot. sbt- release configuration is already added to build.sbt. Just run below commands to see how the version is increased and what will be in the GIT log as commits, tags Check by running in shell: RELEASE_VERSION_BUMP=true sbt “release with-defaults” git log --oneline cat version.sbt // look at new version RELEASE_PUBLISH=true sbt “release with-defaults” git log --oneline cat version.sbt // look at new version
80. More Info 1. Scalar 2018, 2019 whiteboard voting results! https://blog.softwaremill.com/scalar-2018-whiteboard-voting-results-c6f50f8fb16d https://blog.softwaremill.com/scalar-2019-whiteboard-voting-40b31e4f7f7 2. Scala Developer Survey Results 2018 (link) 3. Scaladex: https://index.scala-lang.org 4. Source Code: https://github.com/novakov-alexey/akka-crud-service 5. Some Scala posts: https://medium.com/se-notes-by-alexey-novakov Twitter: @alexey_novakov
81. https://unsplash.com/photos/n8wuzsypQ1M https://unsplash.com/photos/WvDYdXDzkhs https://unsplash.com/photos/wy_L8W0zcpI https://unsplash.com/photos/buWcS7G1_28 https://unsplash.com/photos/CQwNdMxwjfk https://unsplash.com/photos/P4a43pThV3c https://unsplash.com/photos/Lx_GDv7VA9M https://unsplash.com/photos/gzeTjGu3b_k Images https://unsplash.com/photos/oqStl2L5oxI