About me - 4yrs in Scala, 10yrs in Java - a fan of Data Processing, - Distributed Systems, - Functional Programming Olá, I am Alexey! I am working at Ultra Tendency
Nice to know what is: 1. Functional Patterns: Functor, Applicative, Monad, MonadError (Cats), IO Monad 2. Referencial transparency in FP 3. Pure vs. Impure function 4. Encoding of Type Classes in Scala 5. Lazy vs. Eager evaluation
G o a l o f t h e Workshop - inspire you to write functional code in Scala - be familiar with some of the Typelevel libraries as your building blocks Not a goal: - learn how to write micro services - complete all exercises
Add: POST /api/v1/trips, body = JSON Update: PUT /api/v1/trips/, body = JSON Delete: DELETE /api/v1/trips/ Select all: GET /api/v1/trips?sort=id&page=1&pageSize=100 Select one: GET /api/v1/trips/ Ride booking API
Models (Protocol) final case class Trip( id: Int, city: String, vehicle: Vehicle, price: Int, completed: Boolean, distance: Option[Int], endDate: Option[LocalDate] ) final case class Trips(trips: Seq[Trip]) object Vehicle extends Enumeration { type Vehicle = Value val Bike, Taxi, Car = Value } final case class CommandResult(count: Int)
HTTP Server - http4s - Components for the workshop: - http4s is built with cats, cats-effect, fs2-io "org.http4s" %% "http4s-blaze-server" % Version "org.http4s" %% "http4s-blaze-client" % Version % Test “org.http4s" %% "http4s-circe" % Version "org.http4s" %% "http4s-dsl" % Version blaze project is NIO micro framework
routes: pattern matching val routes: HttpRoutes[F] = HttpRoutes.of[F] { case GET -> Root / IntVar(id) => // http4s-dsl provides a shortcut to create an F[Response] for Status Code Ok(…) case req @ POST -> Root / “some_string” / someVal => // do something with request object req.as[MyCaseClass].flatMap( _ => Ok(…)) case PUT -> Root / “some_string” / someVal => Ok(…) } … sequence of functions Docs: https://http4s.org/v0.20/dsl/
Cats: IOApp import cats.effect._, cats.implicits._ object Main extends IOApp { override def run(args: List[String]): IO[ExitCode] = ??? } Docs: https://typelevel.org/cats-effect/datatypes/ioapp.html - IOApp - is an entry point to a pure FP program - brings ContextShift[IO] and Timer[IO] - provides interruption handler
Routes - An HttpRoutes[F] is a type alias for: Kleisli[OptionT[F, ?], Request[F], Response[F]] Kleisli: A => F[B] def of[F[_]](pf: PartialFunction[Request[F], F[Response[F]]])( implicit F: Sync[F]): HttpRoutes[F] - HttpRoutes.of definition: - Eventually, we will use cats.effect.IO as effect type for F
Cats Effect: Sync trait Sync[F[_]] extends Bracket[F, Throwable] with Defer[F] { def suspend[A](thunk: => F[A]): F[A] def delay[A](thunk: => A): F[A] = suspend(pure(thunk)) } A Monad that can suspend the execution of side effects in the F[_] context.
Scala Logging - It wraps slf4j and requires logging backend library like logback - Usage: class MyClass extends LazyLogging { logger.debug("This is very convenient ;-)”) … } “com.typesafe.scala-logging” %% scala-logging” if (logger.isDebugEnabled) logger.debug(s"Some $expensive message!") - check-enabled-idiom is applied automatically by Scala macros
Task #1 1.1. implement HTTP routes • in org.scalamsi.http.QueryRoutes add a route for GET /ping to Ok(“pong”) and leave it assigned to val routes: HttpRoutes[F]. See examples in the code. • instantiate QueryRoutes in org.scalamsi.Module and assign it to val routes: HttpRoutes[F] using such code to combine Uri prefix with a route: Router(apiPrefix -> (qr.routes))
Task #1 cont. 1.2. In Main.scala, implement run function 1.3 start HTTP server via SBT 1.4. use scala-logging in your Main or Module classes This can be done by extending or mixing-in StrictLogging or LazyLogging traits into your class Once mixed in, a logger instance is in your scope, so just use it to write some useful info message First, extend Main object from IOApp. Override its run method. Implement it by calling stream function with IO as effect type and then call .compile.drain.as(ExitCode.Success) Check Task1 via test: sbt "testOnly *Task1Test” - run shell command: sbt run. Console output should report that web-server is started - (optional check) use curl to call the ping route to get pong reply back: curl localhost:8080/api/v1/trips/ping - stop sbt via Ctrl+C
Add: POST /api/v1/trips, body = JSON Update: PUT /api/v1/trips/, body = JSON Delete: DELETE /api/v1/trips/ Select all: GET /api/v1/trips?sort=id&page=1&pageSize=100 Select one: GET /api/v1/trips/ Ride booking API
Circe - JSON library for Scala - built with Cats - supports JVM and Scala.js - allows to parse JSON text into JSON AST, Traverse JSON - Codecs can be derived automatically, semi-automatically or be written manually
Task #2 2.1. Add Circe codecs for protocol case classes • in org.scalamsi.json.CirceJsonCodecs, add implicit Encoder & Decoder instances - Trip, Trips and CommandResult In order to add these instances just leverage circe functions import io.circe.generic.semiauto.{deriveDecoder, deriveEncoder} • mixin org.scalamsi.json.CirceJsonCodecs into QueryRoutes & CommandRoutes Docs: https://circe.github.io/circe/codecs/semiauto-derivation.html
Task #2 cont. 2.2. implement 5 routes of the Ride API • in - org.scalamsi.http.QueryRoutes (2 routes) - org.scalamsi.http.CommandRoutes (3 routes) assign them to val routes::HttpRoutes[F]. See examples in the code • in org.scalamsi.Module, instantiate QueryRoutes and CommandRoutes • assign both to val routes: HttpRoutes[F] using below code to combine them: Router(apiPrefix -> (qr.routes <+> cr.routes)) Check Task2 via test: sbt "testOnly *Task2Test" Docs: https://typelevel.org/cats/typeclasses/semigroupk.html
Algebra definition or DSL as parameterised trait Interpreter as class/object implementation This technique is called - Tagless Final See more in this short and nice blogpost: https:// www.basementcrowd.com/2019/01/17/an-introduction-to-tagless-final-in- scala/
Task #3 implement TripService Service logic is mainly delegating everything to the Repository level and doing a few input parameter validation. 1. Complete selectAll method implementation by flatMapping the sorting field and calling the selectAll on the repository. Try to use for-comprehension here 2. Implement insert and update method: use existing validate method and then call Repository method to insert or update, if the input data is valid 3. in Module.scala, instantiate real TripService and assign it to val service. Check Task3 via test: sbt "testOnly *Task3Test”
Task #4 implement refined types for JdbcConfig a) in configs.scala, define two more types inside the object refined: 1. A type for JdbcConfig#maximumPoolSize as Int range from 1 to 100. See example of ConnectionTimeout type 2. JdbcConfig#url property, using regexp: jdbc:\\w+://\\w+:[0-9]{4,5}/\\w+ Example: type MyType = String Refined MatchesRegex[W.`""" REG EXP HERE """`.T] b) for properties: driver, user, password use non empty String type c) for connection timeout use the existing type, which is already defined as reference d) in Module.scala, fix the parameters of Transactor.fromDriverManager, by accessing values of every parameter using .value mehtod on each of them. For example: cfg.driver.value Check Task4 via test: sbt "testOnly *Task4Test”
Meet Doobie override def select(id: Int): F[Option[Trip]] = sql"SELECT * FROM trips WHERE id = $id" .query[Trip] .option .transact(xa) select: doobie is a pure functional JDBC layer for Scala and Cats. It is not an ORM class TripRepository[F[_]: Sync](xa: Transactor[F]) extends Repository[F] import doobie._ import doobie.implicits._ Docs: https://tpolecat.github.io/doobie/docs/03-Connecting.html
Doobie val xa = Transactor.fromDriverManager[F]( cfg.driver.value, cfg.url.value, cfg.user.value, cfg.password.value) implicit val cs = IO.contextShift(ExecutionContext.global) resources: F[_]: Async: ContextShift A Transactor is a data type that knows how to work* with database example for cats-effect IO Docs: https://tpolecat.github.io/doobie/docs/14-Managing-Connections.html
Task #5 implement TripRepository & TripRepositoryQueries 1. in TripRepositoryQueries#selectAllQuery write a constant query and then use it inside the TripRepository#selectAll. After calling selectAllQuery call .stream then .drop, . take, compile, to[Seq] and finally transact(xa). In such way, we applied page and pageSize parameters to return a particular offset from the database using streaming approach. 2. in TripRepositoryQueries#updateQuery, write a query by concatenating TripRepositoryQueries#updateFrag with a query fragments for “values” and “predicate”. Use ++ method for concatenation and call .update on the result Check Task5 via test: sh start-dev-db.sh && sbt "testOnly *Task5Test”
Testing - Scalatest: - path: HTTP Routes Application Mock Database - Scalatest + testcontainers-scala - to test integration - path: HTTP Routes Application Postgres Container - Note libraries like Mockito can be usefull as well, but it often it is OK to write mock manually. Thanks to composition of Scala programs and FP
val routes = Router(Module.apiPrefix -> (qr.routes <+> cr.routes)).orNotFound Testing Application val req = Request[IO](method = Method.GET, uri = getUri(s"/${lisbon.id}")) val res = routes.run(req) implicit ev: EntityDecoder[IO, Trip] res.as[Trip].unsafeRunSync should ===(lisbon) import cats.effect.IO import org.http4s.EntityDecoder val lisbon = Trip(3, "lisbon", Vehicle.Bike, 2000, completed = false, None, None) create request & run it via routes pass decoder to convert the response no test kit is needed, pure FP
testcontainers-scala - wraps Java library Test Containers - any Docker image can be used for tests - provides common containers for databases and Selenium - integrated with Scalatest using 2 traits: ForEachTestContainer, ForEachTestContainer Test lifecycle: Start container After Start hook Run test(s) Before Stop hook Stop container Docs: https://github.com/testcontainers/testcontainers-scala
class E2ETest extends FlatSpec with ForAllTestContainer { override val container = GenericContainer("nginx:latest", exposedPorts = Seq(80), waitStrategy = Wait.forHttp("/") ) Testing Application with DB waitStrategy can be customised to wait for a specific port, docker health check or a particular line in the log it should “call nginx” in { val port: Int = container.mappedPort(80) val ipAddress: String = container.containerIpAddress …. } override val container = MySQLContainer() // or use one of the few predefined containers Docs: https://www.testcontainers.org/features/startup_and_waits/
Task #6 implement Task6Test 1. in Task6Test.scala mix in ForAllTestContainer or ForEachTestContainer trait and override container property with out-of-the-box Postgres container class. Set image name to postgres:10.4 2. use container property to initialise dbProps map 3. Note: mod variable should be used either lazily or be used inside the test case code block 4. in Module.scala, remove overriden createSchema() method in val repo implementation. We need to use real method, not a stub Check Task6 via test: sbt "testOnly *Task6Test”
DI with MacWire MacWire generates new instance creation code of given classes, using values in the enclosing type for constructor parameters, with the help of Scala Macros. trait UserModule { lazy val databaseAccess = new DatabaseAccess() lazy val securityFilter = new SecurityFilter() lazy val userFinder = new UserFinder(databaseAccess, securityFilter) lazy val userStatusReader = new UserStatusReader(userFinder) } trait UserModule { import com.softwaremill.macwire._ lazy val databaseAccess = wire[DatabaseAccess] lazy val securityFilter = wire[SecurityFilter] lazy val userFinder = wire[UserFinder] lazy val userStatusReader = wire[UserStatusReader]} compile-time checked macwire version
Task #7 implement Module.scala - in Module.scala use MacWire’s wire function to instantiate all values instead of using “new” operator. It should be around 5 values wired with MacWire Check Task7 via test: sbt "testOnly *Task7Test” Also run previous test to check it is not broken: sbt "testOnly *Task6Test” Docs: https://github.com/softwaremill/macwire
Task #8 1. configure SBT native-packager plugin - in build.sbt of root module, use another docker image such as “openjdk:8-jre- alpine” and “AshScriptPlugin” to generate “busybox" compatible image shell script. AshScriptPlugin is provided by native-packager-plugin, so just use it Check by building and running Docker image: sbt docker:publishLocal docker stop postgres & docker rm postgres ./start.sh 1 postgres and 1 scala service containers should be started https://www.scala-sbt.org/sbt-native-packager/archetypes/misc_archetypes.html
Task #8 2. configure SBT release plugin 2. 1. First, commit all the changes: git add -A && git commit -m "ready for release” 2.2. release current project version and bump version to the next snapshot. sbt- release configuration is already added to build.sbt. Just run below commands to see how the version is increased and what will be in the GIT log as commits, tags Check by running in shell: RELEASE_VERSION_BUMP=true sbt “release with-defaults” git log --oneline cat version.sbt // look at new version RELEASE_PUBLISH=true sbt “release with-defaults” git log --oneline cat version.sbt // look at new version
alexeyn
honma12345
koba04
ufoo68
jmatsu
umizaru
satoshi7190
munetoshi
ivargrimstad
little_rubyist
77web
headius
reverentgeek
thoeni
rocio
eitanlees
brad_frost
tenderlove
brianwarren
myddelton
frogandcode
mongodb
chrisshort
carmenintech