John De Goes at Scala in the City

ZIO schedule

Transcript

1. ZIO Schedule John A. De Goes — @jdegoes - http://degoes.net Scala in the City ZIO Schedule Scala in the City, October 2018 - London, UK John A. De Goes @jdegoes - http://degoes.net
2. What is ZIO? Retries Repeats Schedule Composition
3. What is ZIO? Retries Repeats Schedule Composition ZIO ZIO lets you build high-performance, type-safe, concurrent, asynchronous applications that don’t leak resources, and are easy to reason about compositionally, test, and refactor.
4. github.com/scalaz/scalaz-zio ZIO What is ZIO? Retries Repeats Schedule Composition
5. def main: Unit = { println("Hello, what is your name?") val name = readLine() println(s"Good morning, $name!") } Second-Class Effects ✗ Pass to functions ✗ Return from functions ✗ Store in data structures ✗ Async/sync/concurrency ✗ Async/sync resource-safety What is ZIO? Retries Repeats Schedule Composition
6. def main: IO[IOException, Unit] = for { _ <- putStrLn("Hello, what is your name?") name <- getStrLn _ <- putStrLn(s"Good morning, $name!") } yield () First-Class Effects ✓ Pass to functions ✓ Return from functions ✓ Store in data structures ✓ Async/sync/concurrency ✓ Async/sync resource-safety What is ZIO? Retries Repeats Schedule Composition
7. IO[E, A] IO[E, A] is an immutable value that describes an effectful program, which may fail with a value of type E, or return a value of type A. What is ZIO? Retries Repeats Schedule Composition ZIO IO
8. What is ZIO? Retries Repeats Schedule Composition IO[Nothing, A] Never fails ZIO IO
9. What is ZIO? Retries Repeats Schedule Composition IO[E, Nothing] Never returns ZIO IO
10. What is ZIO? Retries Repeats Schedule Composition IO[Nothing, Nothing] Never fails or returns ZIO IO
11. for { queue <- Queue.unbounded[String] worker = queue.take.flatMap(putStrLn).forever workers10k = List.fill(10000)(worker) _ <- IO.forkAll(workers10k) _ <- queue.offer("Chocolate!").forever.fork } yield () Fibers ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition What is ZIO? Retries Repeats Schedule Composition
12. for { resp <- api.getUser(userId) _ <- db.updateUser(userId, resp.profile) _ <- promise.complete(()) } yield () What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Async
13. requests.parTraverse { request => for { product <- findProduct(request.id) update <- applyDiscount(product.id, discount) _ <- db.updateProduct(product.id, update) } yield product.id } What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Async
14. def fib(n: Int): IO[Nothing, Int] = if (n <= 1) IO.now(n) else fib(n-1).parWith(fib(n-2))(_ + _) fib(Int.MaxValue).fork(_.interrupt) What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Interrupt
15. openFile.bracket(closeFile(_)) { handle => def loop(ref: Ref[Chunk[Byte]]) = for { chunk <- handle.readChunk <- ref.update(_ ++ chunk) all <- if (handle.hasMore) loop(ref) else ref.get } yield all Ref(Chunk.empty).flatMap(loop) } What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Bracket
16. def webCrawler[E: Monoid, A: Monoid](seeds: Set[URL], router: URL => Set[URL], processor: (URL, String) => IO[E, A]): IO[Nothing, Crawl[E, A]] = { def loop(seeds: Set[URL], ref: Ref[CrawlState[E, A]]): IO[Nothing, Unit] = ref.update(_ |+| CrawlState.visited(seeds)) *> IO.parTraverse(seeds)( seed => getURL(seed).redeem(_ => IO.unit, html => for { visited <- ref.get.map(_.visited) seeds <- IO.now(extractURLs(seed, html).toSet .flatMap(router) -- visited) crawl <- processor(seed, html).redeemPure(Crawl(_, mzero[A]), Crawl(mzero[E], _)) _ <- ref.update(_ |+| CrawlState.crawled(crawl)) _ <- loop(seeds, ref) } yield ())).void Ref(mzero[CrawlState[E, A]]).flatMap(ref => loop(seeds, ref).map(_ => ref.get.map(_.crawl))) } What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Composable
17. Retries What is ZIO? Retries Repeats Schedule Composition
18. What is ZIO? Retries Repeats Schedule Composition App Web API
19. What is ZIO? Retries Repeats Schedule Composition App Web API 408
20. What is ZIO? Retries Repeats Schedule Composition App Web API 408 429
21. What is ZIO? Retries Repeats Schedule Composition App Web API 408 429 423
22. What is ZIO? Retries Repeats Schedule Composition App Web API 408 429 423 500
23. What is ZIO? Retries Repeats Schedule Composition App Web API 408 429 423 500 503
24. What is ZIO? Retries Repeats Schedule Composition App Web API 408 429 423 500 503 504
25. What is ZIO? Retries Repeats Schedule Composition App Web API 408 429 423 500 503 504
26. What is ZIO? Retries Repeats Schedule Composition App Database Cache Local Storage Cloud Storage Streaming Logs Analytics Web APIs
27. def networkRequest(): A = ??? def networkRequestWithRetries(max: Int, millis: Long): A = { var i = 0 while (i < max) { try { return networkRequest() } catch { case _ : Exception => i = i + 1 Thread.sleep(millis) } } } What is ZIO? Retries Repeats Schedule Composition Retrying Synchronously
28. Repeats What is ZIO? Retries Repeats Schedule Composition
29. What is ZIO? Retries Repeats Schedule Composition App Generate Report
30. What is ZIO? Retries Repeats Schedule Composition App Generate Report Every user
31. What is ZIO? Retries Repeats Schedule Composition App Generate Report Every user Every day
32. What is ZIO? Retries Repeats Schedule Composition App Generate Report Every user Every day Every account
33. What is ZIO? Retries Repeats Schedule Composition App Generate Report Every user Every day Every account Every week
34. What is ZIO? Retries Repeats Schedule Composition App Service Heartbeat Data Retrieval Remote Uploading Report Generation Log Rotation File Cleanup User Reminders Garbage Collection
35. What is ZIO? Retries Repeats Schedule Composition def reportGen(): A = ??? def repeatedReportGen(max: Int, millis: Long): List[A] = { var list = List.empty[A] var i = 0 while (i < max) { list = reportGen() :: list Thread.sleep(millis) i = i + 1 } list } Repeating Synchronously
36. Schedule What is ZIO? Retries Repeats Schedule Composition
37. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Schedule[A, B] Schedule[A, B] is an immutable value that describes an effectful schedule, which after consuming an A, produces a B and decides to halt or continue after some delay d.
38. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Schedule[A, B] A B B Continue? Continue after delay d Halt immediately
39. What is ZIO? Retries Repeats Schedule Composition Should I repeat a program that failed with an E? Should I repeat a program that returned an A? Retries Repeats
40. What is ZIO? Retries Repeats Schedule Composition Retries Repeats Schedule[E, B] Schedule[A, B] Retry policy Consumes errors of type E Emits values of type B Repeat policy Consumes return values of type A Emits values of type B
41. What is ZIO? Retries Repeats Schedule Composition Schedule.never : Schedule[Any, Nothing] Zero Recurrences Accepts any input Never recurs, never emitting a value
42. What is ZIO? Retries Repeats Schedule Composition Schedule.once : Schedule[Any, Unit] One Recurrence Accepts any input Recurs once, emittingunit
43. What is ZIO? Retries Repeats Schedule Composition Schedule.forever : Schedule[Any, Int] Infinite Recurrences Consumes any input Recurs forever without delay, emitting the number of recurrences so far
44. What is ZIO? Retries Repeats Schedule Composition Schedule.recurs(n: Int) : Schedule[Any, Int] Fixed Recurrences Accepts any input Recurs the specified number of times, emitting number of recurrences so far
45. duration task task What is ZIO? Retries Repeats Schedule Composition Schedule.spaced(d: Duration) : Schedule[Any, Int] Spaced Recurrences Accepts any input Recurs forever with delay, emitting the number of recurrences so far
46. duration task task What is ZIO? Retries Repeats Schedule Composition Schedule.fixed(d: Duration) : Schedule[Any, Int] Fixed Recurrences Accepts any input Recurs forever with a delay, emitting number of recurrences so far duration
47. What is ZIO? Retries Repeats Schedule Composition Schedule.exponential(d: Duration, f: Double = 2.0) : Schedule[Any, Duration] Exponential Recurrences Accepts any input Recurs forever with delay, emitting the current delay between steps Scaling factor Starting duration
48. What is ZIO? Retries Repeats Schedule Composition Schedule.doWhile[A](f: A => Boolean): Schedule[A, A] Conditional Recurrences Accepts an A Recurs while the predicate is true without delay, emitting the same A Recurrence condition
49. What is ZIO? Retries Repeats Schedule Composition Schedule.doUntil[A](f: A => Boolean): Schedule[A, A] Conditional Recurrences Accepts an A Recurs until the predicate is true without delay, emitting the same A Recurrence condition
50. What is ZIO? Retries Repeats Schedule Composition Schedule.collect: Schedule[A, List[B]] Collecting Recurrences Recurs forever without delay, emitting a list of all consumed A’s Consumes an A
51. What is ZIO? Retries Repeats Schedule Composition Schedule.identity[A]: Schedule[A, A] Identity Recurrences Recurs forever without delay, emitting the same A Consumes an A
52. What is ZIO? Retries Repeats Schedule Composition Schedule.unfold[A](a: => A)(f: A => A): Schedule[Any, A] Unfold Recurrences Recurs forever without delay, emitting an A by unfolding the seed through repeated application Consumes any value
53. What is ZIO? Retries Repeats Schedule Composition Schedule.point[A](a: => A): Schedule[Any, A] Constant Output Recurs forever without delay, emitting a constant Consumes any value
54. What is ZIO? Retries Repeats Schedule Composition Schedule.lift[A, B](f: A => B): Schedule[A, B] Function Output Recurs forever without delay, emitting the function applied to the input Consumes an A
55. What is ZIO? Retries Repeats Schedule Composition Retries Repeats val action: IO[E, A] = ??? val policy: Schedule[E, B] = ??? val retried: IO[E, A] = action retry policy val action: IO[E, A] = ??? val policy: Schedule[A, B] = ??? val repeated: IO[E, B] = action repeat policy
56. What is ZIO? Retries Repeats Schedule Composition Retries Repeats val action: IO[E, A] = ??? val policy: Schedule[E, B] = ??? val orElse: (E, B) => IO[E2, A] = ??? val retried: IO[E2, A] = action retryOrElse (policy, orElse) val action: IO[E, A] = ??? val policy: Schedule[A, B] = ??? val orElse: (E, Option[B]) => IO[E2, B] = ??? val repeated: IO[E2, B] = action repeatOrElse (policy, onError)
57. What is ZIO? Retries Repeats Schedule Composition Retries Repeats val action: IO[E, A] = ??? val policy: Schedule[E, B] = ??? val orElse: (E, B) => IO[E2, B] = ??? val retried: IO[E2, Either[B, A]] = action retryOrElse0 (policy, orElse) val action: IO[E, A] = ??? val policy: Schedule[A, B] = ??? val orElse: (E, Option[B]) => IO[E2, C] = ??? val repeated: IO[E2, Either[C, B]] = action repeatOrElse0 (policy, onError)
58. Composition What is ZIO? Retries Repeats Schedule Composition
59. What is ZIO? Retries Repeats Schedule Composition def networkRequestWithRetries(factor: Float = 1.5f, init: Int = 1, cur: Int = 0) (implicit as: ActorSystem): Future[String] = { networkRequest().recoverWith { case NetworkException => val next: Int = if (cur == 0) init else Math.ceil(cur * factor).toInt after(next.milliseconds, as.scheduler, global, Future.successful(1)).flatMap { _ => networkRequestWithRetries(factor, init, next) } case t: Throwable => throw t } } Future Retries* * https://hackernoon.com/exponential-back-off-with-scala-futures-7426340d0069
60. What is ZIO? Retries Repeats Schedule Composition def retry[F[_]: Timer: RaiseThrowable, O]( fo : F[O], delay : FiniteDuration, nextDelay : FiniteDuration => FiniteDuration, maxAttempts : Int, retriable : Throwable => Boolean): Stream[F, O] = ??? FS2 Retry
61. What is ZIO? Retries Repeats Schedule Composition Monix Retry & Repeat Retries Repeats val task : Task[A] = ??? val retries: Int = ??? val retried1: Task[A] = task onErrorRestart (retries) val pred: Throwable => Boolean = ??? val retried2: Task[A] = task onErrorRestartIf (pred) // onErrorRestartLoop val task: Task[A] = ??? val pred: A => Boolean = ??? val repeated: Task[A] = task restartUntil (pred)
62. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero
63. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap
64. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap Category id compose
65. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap Category id compose Profunctor lmap rmap dimap
66. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap Category id compose Profunctor lmap rmap dimap Strong first second
67. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap Category id compose Profunctor lmap rmap dimap Strong first second Choice left right
68. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) && (s2 : Schedule[A, C]) : Schedule[A, (B, C)] Intersection s1 s2 s1 && s2 Continue : Boolean b1 b2 b1 && b2 Delay : Duration d1 d2 d1.max(d2) Emit : (A, B) a b (a, b)
69. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) || (s2 : Schedule[A, C]) : Schedule[A, (B, C)] Union s1 s2 s1 || s2 Continue : Boolean b1 b2 b1 || b2 Delay : Duration d1 d2 d1.min(d2) Emit : (A, B) a b (a, b)
70. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) andThen (s2 : Schedule[A, C]) : Schedule[A, (B, C)] Sequence A A B C s1 s2 s1 andThen s2 A Either[B, C]
71. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) >>> (s2 : Schedule[A, C]) : Schedule[A, (B, C)] Compose A B B C s1 s2 s1 >>> s2 A C
72. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).jittered : Schedule[A, B] Jittering A A B B s1 s1’ Delays randomly jittered
73. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).collect : Schedule[A, List[B]] Collecting A A B List[C] s1 s1’ Emissions collected
74. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).whileInput(f: A => Boolean) : Schedule[A, B] Filtering By Input A A B B s1 s1’ Continues while/until f returns true (s1 : Schedule[A, B]).untilInput(f: A => Boolean) : Schedule[A, B]
75. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).whileOutput(f: B => Boolean) : Schedule[A, B] Filtering By Output A A B B s1 s1’ Continues while/until f returns true (s1 : Schedule[A, B]).untilOutput(f: B => Boolean) : Schedule[A, B]
76. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).map(f: B => C) : Schedule[A, C] Mapping A A B C s1 s1’ B mapped to C
77. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) *> (s2 : Schedule[A, C]) : Schedule[A, C] (s1 && s2).map(_._2) Left / Right Ap (s1 : Schedule[A, B]) <* (s2 : Schedule[A, C]) : Schedule[A, B] (s1 && s2).map(_._1)
78. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).logInput(f: A => IO[Nothing, Unit]) : Schedule[A, B] Logging Inputs A A B B s1 s1’ Inputs logged to f
79. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).logOutput(f: B => IO[Nothing, Unit]) : Schedule[A, B] Logging Outputs A A B B s1 s1’ Outputs logged to f
80. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule
81. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule
82. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Schedule.exponential(10.milliseconds)
83. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule (Schedule.exponential(10.milliseconds) andThen ???)
84. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen ???)
85. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen Schedule.fixed(60.seconds))
86. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen (Schedule.fixed(60.seconds) && Schedule.recurs(100))
87. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen (Schedule.fixed(60.seconds) && Schedule.recurs(100)).jittered
88. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen (Schedule.fixed(60.seconds) && Schedule.recurs(100)).jittered *> Schedule.identity[A].collect
89. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule def customSchedule[A]: Schedule[A, List[A]] = { import Schedule._ (exponential(10.millis).whileOutput(_ < 60.secs) andThen (fixed(60.secs) && recurs(100)).jittered *> identity[A].collect }
90. Github https://github.com/scalaz/scalaz-zio Microsite https://scalaz.github.io/scalaz-zio/ ZIO Chat https://gitter.im/scalaz/scalaz-zio/ Scalaz Chat https://gitter.im/scalaz/scalaz/ That’s A Wrap!
91. Functional Scala by John A. De Goes Local Remote Date London London Time Oct 21 - 23 Paris Paris Time Oct 25 - 27 SF / SV Pacific Coast Time Nov 18 - 21 That’s A Wrap!
92. THANK YOU! @jdegoes - http://degoes.net That’s A Wrap!