What happens in ZIO, stays in ZIO

Talk on ZIO presented at functionalscala.com 2020

Transcript

1. Willem Vermeer - 3 December 2020 What happens in ZIO, stays in ZIO functionalscala.com @willemvermeer
2. Agenda • introduction • use case: ZMail • ZIO-NIO • ZIO-gRPC • ZIO-config • demo, conclusion and next steps
3. Agenda • introduction • use case: ZMail • ZIO-NIO • ZIO-gRPC • ZIO-config • demo, conclusion and next steps Yo Wilhelm, who are you anyway?
4. How it all started
5. Somewhere in 2016
6. Amsterdam.scala - 2 March 2019
7. Amsterdam.scala - 2 March 2019 OK Boomer, can we get started already?
8. ZMail use case SMTP Service Message Store GRPC Service GRPC Client ZMail Frontend ZMail Server
9. Staying in ZIO - why? • only one library to learn • error management is the same from top to bottom • one thread/fiber management model
10. ZMail use case SMTP Service Message Store GRPC Service GRPC Client ZMail Frontend ZMail Server
11. Simple Mail Transfer Protocol 101 Client: telnet smtp.example.com 25 Server: 220 smtp.example.com Example SMTP Service Client: HELO smtp.sender.org Server: 250 Hello Client: MAIL FROM: <sender@sender.org> Server: 250 sender@sender.org... Sender OK Client: RCPT TO: <friend@example.com> Server: 250 friend@example.com... Recipient OK Client: DATA Server: 354 Please start mail input Client: Subject: Datatest Client: Data...data...data... Client: . Server: 250 Mail queued for delivery. Client: QUIT Server: 221 Goodbye.
12. Using ZIO-NIO to listen on port 25 val smtpServer = AsynchronousServerSocketChannel().mapM { socket => for { address <- SocketAddress.inetSocketAddress(“0.0.0.0", 25) _ <- socket.bind(address) _ <- socket.accept.preallocate .flatMap( _.ensuring(putStrLn("Connection closed")) .use(channel => receive(channel)) .fork ) .forever.fork } yield () }
13. Using ZIO-NIO to read a line of text def receive(channel: AsynchronousSocketChannel) = ZStream .fromEffectOption( channel .readChunk(1024) .tap(c => putStrLn(s"Read chunk of length ${c.size}")) .orElseFail(None) ) .flattenChunks .transduce(ZTransducer.utf8Decode) .run(Sink.foldLeft("")(_ + _))
14. Using ZIO-NIO to write a line of text def writeToChannel( txt: String, channel: AsynchronousSocketChannel ) = ZStream .fromChunk(Chunk.fromArray(txt.toCharArray)) .transduce(Charset.Standard.utf8.newEncoder.transducer()) .foreachChunk(chunk => channel.writeChunk(chunk))
15. Process line by line…follow protocol Client: MAIL FROM: <sender@sender.org> def nextCommand = for { line <- readLineFromChannel _ <- logInbound(line) command <- SmtpParser.parse(line) } yield command
16. Cheat: use fastparse for SMTP protocol parsing object SmtpParser { def EOL[_: P]: P[Unit] = P("\r" ~ "\n") // see https://tools.ietf.org/html/rfc5321#section-4.1.2 def digit[_: P]: P[Unit] = P(CharIn("0-9")) def alpha[_: P]: P[Unit] = P(CharIn("a-z") | CharIn("A-Z")) def letters[_: P]: P[Unit] = P(alpha.rep(1)) def letdig[_: P]: P[String] = P((alpha | digit).!) def prascii[_: P]: P[Unit] = P( "!" | "#" | "$" | "%" | "&" | "'" | "*" | "+" | "-" | "/" | "=" | "?" | "^" | "_" | "`" | "{" | "|" | "}" | "~" ) def atext[_: P]: P[Unit] = P(alpha | digit | prascii) def atom[_: P]: P[Unit] = P((alpha | digit | prascii).rep(1)) def dotstring[_: P]: P[Unit] = P(atom ~ ("." ~ atom).rep(0)) def ldhstr[_: P]: P[Unit] = P((letdig | "-").rep(0) ~ letdig) def subdomain[_: P]: P[Unit] = P(letdig ~ ldhstr.?) def domain[_: P]: P[Unit] = P(subdomain ~ ("." ~ subdomain).rep(0)) def qtextsmtp[_: P]: P[Unit] = P(" " | "!" | CharIn("#-[") | CharIn("]- ~")) // bnf: %d32-33 / %d35-91 / %d93-126 def quotedpairsmtp[_: P]: P[Unit] = P("\\" ~ CharIn(" -~")) // bnf: %d92 %d32-126 def qcontentsmtp[_: P]: P[Unit] = P(qtextsmtp | quotedpairsmtp) def quotedstring[_: P]: P[Unit] = P("\"" ~ qcontentsmtp.rep(0) ~ "\"") def localpart[_: P]: P[Unit] = P(dotstring | quotedstring) def snum[_: P]: P[Unit] = P(digit.rep(min = 1, max = 3)) def ipv4addressliteral[_: P]: P[Unit] = P(snum ~ ("." ~ snum).rep(exactly = 3)) // TODO: ipv6 support def dcontent[_: P]: P[Unit] = P(CharIn("!-Z") | CharIn("^-~")) // bnf: %d33-90 / %d94-126 def standardizedtag[_: P]: P[Unit] = ldhstr def generaladdressliteral[_: P]: P[Unit] = P(standardizedtag ~ ":" ~ dcontent.rep(1)) def addressliteral[_: P]: P[Unit] = P("[" ~ (ipv4addressliteral | generaladdressliteral) ~ "]") def domainC[_: P]: P[Domain] = domain.!.map(Domain) def mailbox[_: P]: P[Unit] = P(localpart ~ "@" ~ (domain | addressliteral)) def atdomain[_: P]: P[Unit] = P("@" ~ domain) def adl[_: P]: P[Unit] = P(atdomain ~ ("," ~ atdomain).rep(0)) // only the first @domain will be picked up def path[_: P]: P[String] = P("<" ~ (adl ~ ":").? ~ mailbox.! ~ ">") def forwardpath[_: P]: P[String] = P(path) def reversepath[_: P]: P[String] = P(path | "<>").map(_.toString) def esmtpvalue[_: P]: P[Unit] = P((CharIn("!-<") | CharIn(">- ~")).rep(1)) // bnf 1*(%d33-60 / %d62-126) def esmtpkeyword[_: P]: P[Unit] = P(letdig ~ (letdig | "-").rep(0)) def esmtpparam[_: P]: P[Unit] = P(esmtpkeyword ~ ("=" ~ esmtpvalue).?) def mailparameters[_: P]: P[Unit] = P(esmtpparam ~ (" " ~ esmtpparam).rep(0)) def helo[_: P]: P[Helo] = P("HELO" ~ " " ~ domainC).map(Helo) def ehlo[_: P]: P[Ehlo] = P("EHLO" ~ " " ~ domainC).map(Ehlo) def mailfrom[_: P]: P[MailFrom] = P("MAIL FROM:" ~ reversepath ~ (" " ~ mailparameters).?).map(p => MailFrom(ReversePath(p))) def rcptto[_: P]: P[RcptTo] = P("RCPT TO:" ~ forwardpath ~ (" " ~ mailparameters).?).map(p => RcptTo(ReversePath(p))) def data[_: P]: P[Data.type] = P("DATA").map(_ => Data) def quit[_: P]: P[Quit.type] = P("QUIT").map(_ => Quit) def command[_: P]: P[Command] = P((helo | ehlo | mailfrom | rcptto | data | quit) ~ EOL) def parse(line: String): IO[ParseError, Command] = fastparse.parse(line, SmtpParser.command(_)) match { case Parsed.Success(cmd, _) => ZIO.succeed(cmd) case Parsed.Failure(label, index, _) => ZIO.fail(ParseError(s"ParserError $label at $index for $line")) } case class ParseError(msg: String) extends Exception(msg) } https://www.lihaoyi.com/fastparse/
17. object SmtpParser { def EOL[_: P]: P[Unit] = P("\r" ~ "\n") // see https://tools.ietf.org/html/rfc5321#section-4.1.2 def digit[_: P]: P[Unit] = P(CharIn("0-9")) def alpha[_: P]: P[Unit] = P(CharIn("a-z") | CharIn("A-Z")) def letters[_: P]: P[Unit] = P(alpha.rep(1)) def letdig[_: P]: P[String] = P((alpha | digit).!) def prascii[_: P]: P[Unit] = P( "!" | "#" | "$" | "%" | "&" | "'" | "*" | "+" | "-" | "/" | "=" | "?" | "^" | "_" | "`" | "{" | "|" | "}" | "~" ) def atext[_: P]: P[Unit] = P(alpha | digit | prascii) def atom[_: P]: P[Unit] = P((alpha | digit | prascii).rep(1)) def dotstring[_: P]: P[Unit] = P(atom ~ ("." ~ atom).rep(0)) def ldhstr[_: P]: P[Unit] = P((letdig | "-").rep(0) ~ letdig) def subdomain[_: P]: P[Unit] = P(letdig ~ ldhstr.?) def domain[_: P]: P[Unit] = P(subdomain ~ ("." ~ subdomain).rep(0)) def qtextsmtp[_: P]: P[Unit] = P(" " | "!" | CharIn("#-[") | CharIn("]- ~")) // bnf: %d32-33 / %d35-91 / %d93-126 def quotedpairsmtp[_: P]: P[Unit] = P("\\" ~ CharIn(" -~")) // bnf: %d92 %d32-126 def qcontentsmtp[_: P]: P[Unit] = P(qtextsmtp | quotedpairsmtp) def quotedstring[_: P]: P[Unit] = P("\"" ~ qcontentsmtp.rep(0) ~ "\"") def localpart[_: P]: P[Unit] = P(dotstring | quotedstring) def snum[_: P]: P[Unit] = P(digit.rep(min = 1, max = 3)) def ipv4addressliteral[_: P]: P[Unit] = P(snum ~ ("." ~ snum).rep(exactly = 3)) // TODO: ipv6 support def dcontent[_: P]: P[Unit] = P(CharIn("!-Z") | CharIn("^-~")) // bnf: %d33-90 / %d94-126 def standardizedtag[_: P]: P[Unit] = ldhstr def generaladdressliteral[_: P]: P[Unit] = P(standardizedtag ~ ":" ~ dcontent.rep(1)) def addressliteral[_: P]: P[Unit] = P("[" ~ (ipv4addressliteral | generaladdressliteral) ~ "]") def domainC[_: P]: P[Domain] = domain.!.map(Domain) def mailbox[_: P]: P[Unit] = P(localpart ~ "@" ~ (domain | addressliteral)) def atdomain[_: P]: P[Unit] = P("@" ~ domain) def adl[_: P]: P[Unit] = P(atdomain ~ ("," ~ atdomain).rep(0)) // only the first @domain will be picked up def path[_: P]: P[String] = P("<" ~ (adl ~ ":").? ~ mailbox.! ~ ">") def forwardpath[_: P]: P[String] = P(path) def reversepath[_: P]: P[String] = P(path | "<>").map(_.toString) def esmtpvalue[_: P]: P[Unit] = P((CharIn("!-<") | CharIn(">- ~")).rep(1)) // bnf 1*(%d33-60 / %d62-126) def esmtpkeyword[_: P]: P[Unit] = P(letdig ~ (letdig | "-").rep(0)) def esmtpparam[_: P]: P[Unit] = P(esmtpkeyword ~ ("=" ~ esmtpvalue).?) def mailparameters[_: P]: P[Unit] = P(esmtpparam ~ (" " ~ esmtpparam).rep(0)) def helo[_: P]: P[Helo] = P("HELO" ~ " " ~ domainC).map(Helo) def ehlo[_: P]: P[Ehlo] = P("EHLO" ~ " " ~ domainC).map(Ehlo) def mailfrom[_: P]: P[MailFrom] = P("MAIL FROM:" ~ reversepath ~ (" " ~ mailparameters).?).map(p => MailFrom(ReversePath(p))) def rcptto[_: P]: P[RcptTo] = P("RCPT TO:" ~ forwardpath ~ (" " ~ mailparameters).?).map(p => RcptTo(ReversePath(p))) def data[_: P]: P[Data.type] = P("DATA").map(_ => Data) def quit[_: P]: P[Quit.type] = P("QUIT").map(_ => Quit) def command[_: P]: P[Command] = P((helo | ehlo | mailfrom | rcptto | data | quit) ~ EOL) def parse(line: String): IO[ParseError, Command] = fastparse.parse(line, SmtpParser.command(_)) match { case Parsed.Success(cmd, _) => ZIO.succeed(cmd) case Parsed.Failure(label, index, _) => ZIO.fail(ParseError(s"ParserError $label at $index for $line")) } case class ParseError(msg: String) extends Exception(msg) } Cheat: use fastparse for SMTP protocol parsing https://www.lihaoyi.com/fastparse/ But you said we would stay in ZIO!!
18. Lift fastparse result into ZIO https://www.lihaoyi.com/fastparse/ object SmtpParser { def parse(line: String): IO[ParseError, Command] = fastparse.parse(line, SmtpParser.command(_)) match { case Parsed.Success(cmd, _) => ZIO.succeed(cmd) case Parsed.Failure(label, index, _) => ZIO.fail(ParseError( s"ParserError $label at $index for $line” )) } }
19. Lift fastparse result into ZIO https://www.lihaoyi.com/fastparse/ object SmtpParser { def parse(line: String): IO[ParseError, Command] = fastparse.parse(line, SmtpParser.command(_)) match { case Parsed.Success(cmd, _) => ZIO.succeed(cmd) case Parsed.Failure(label, index, _) => ZIO.fail(ParseError( s"ParserError $label at $index for $line” )) } } // input: “MAIL FROM: <sender@sender.org>” case class MailFrom(from: String) extends Command
20. How to store the message? SMTP Service Message Store GRPC Service GRPC Client ZMail Frontend ZMail Server
21. How to store the messages? Use a zio.Ref object MessageStore { trait Service { def storeMessage(msg: Message): UIO[Unit] } case class MessageStoreImpl(store: Ref[Map[EmailAddress, MessageList]]) extends Service { override def storeMessage(msg: Message): UIO[Unit] = for { ref <- store.get _ <- ref.get(msg.to) match { case None => store.update(_ + (msg.to -> MessageList(Seq(msg)))) case Some(messageList) => store.update(_ + (msg.to -> messageList.add(msg))) } } yield () } }
22. ZIO web interface - what are the options?
23. ZIO web layer https://github.com/polynote/uzhttp
24. Using Http4s in ZIO - it’s possible, but… https://timpigden.github.io/_pages/zio-http4s/zio-http4s-part3.html
25. Akka/http and ZIO interoperability
26. Web interface - gRPC SMTP Service Message Store GRPC Service GRPC Client ZMail Frontend ZMail Server
27. zio-gRPC • same (main) author wrote ScalaPB • built with ZIO and ZIO-Streams • generates gRPC server code and gRPC client code • gRPC Client can be compiled to Scala.JS => runs in the browser!! • supports one-off calls as well as streaming (both directions)
28. gRPC 101 - service definitions in protobuf service MailBoxService { rpc GetMailBox (GetMailBoxRequest) returns (MailBox) {} rpc GetMessage (GetMessageRequest) returns (MailBoxEntry) {} rpc GetMessageStream (GetMailBoxRequest) returns (stream MailBoxEntry) {} } message GetMailBoxRequest { string username = 1; } message MailBox { repeated MailBoxEntry entries = 1; int32 total = 2; } .. more definitions ..
29. Web interface - gRPC server SMTP Service Message Store GRPC Service GRPC Client ZMail Frontend ZMail Server
30. gRPC 101 - code is generated from service defs object ZioService { trait ZMailBoxService[-R, -Context] extends ZGeneratedService[R, Context, ZMailBoxService] { self => def getMailBox(request: GetMailBoxRequest): ZIO[R with Context, Status, MailBox] } type MailBoxService = ZMailBoxService[Any, Any] // .. much more code .. } final case class GetMailBoxRequest( username: String = "", unknownFields: UnknownFieldSet = UnknownFieldSet.empty ) extends scalapb.GeneratedMessage with Updatable[GetMailBoxRequest] { .. }
31. gRPC 101 - work with generated code class LiveMailBoxService(messageStore: MessageStore.Service) extends ZIOService.MailBoxService { override def getMailBox(request: GetMailBoxRequest): UIO[MailBox] = { val username = request.username for { msgs <- messageStore.getMessages(EmailAddress(username)) result <- ZIO.succeed( MailBox( entries = msgs.map(m => MailBoxEntry.of(m)), total = msgs.size ) ) } yield result }
32. Web interface - gRPC client SMTP Service Message Store GRPC Service GRPC Client ZMail Frontend ZMail Server
33. gRPC 101 - generated client code object ZioService { object MailBoxServiceClient { trait ZService[R] { def getMailBox(request: GetMailBoxRequest): ZIO[R, Status, MailBox] } // .. much more code .. } }
34. gRPC 101 - work with client code def getMessages: ZIO[MailBoxServiceClient, Status, MailBox] = for { response <- MailBoxServiceClient .getMailBox(GetMailBoxRequest(“email@example.org")) } yield response
35. gRPC 101 - process the results (1) def getMessages: ZIO[MailBoxServiceClient, Status, Unit] = for { response <- MailBoxServiceClient .getMailBox(GetMailBoxRequest(“email@example.org")) q <- Queue.unbounded[Event] _ <- fillTable(response, q) } yield () def fillTable(mailbox: MailBox, q: Queue[Event]) = ZIO.foreach_(mailbox.entries)(addRow(_, q))
36. gRPC 101 - process the results (2) def findTable(): UIO[Option[HTMLTableElement]] = UIO.effectTotal { dom.document.getElementById("inbox") match { case table: HTMLTableElement => Some(table) case _ => None } } def addRow(entry: MailBoxEntry, q: Queue[Event]) = for { table <- findTable().some rt <- ZIO.runtime[Any] _ <- ZIO.effectTotal { table.insertRow() match { case tr: HTMLTableRowElement => val td = tr.insertCell() td.innerText = entry.subject // .. more td’s .. tr.onclick = { (_: MouseEvent) => rt.unsafeRunSync(q.offer(RowClickEvent(entry.id))) } case _ => } } } yield ()
37. gRPC 101 - process the results (3) def getMessages: RIO[MailBoxServiceClient, Unit] = for { response <- MailBoxServiceClient.getMailBox(GetMailBoxRequest(“email@example.org")) q <- Queue.unbounded[Event] _ <- fillTable(response, q) _ <- Stream .fromQueue(q) .foldM(Idle: State) { case (Idle, RowClickEvent(msgId)) => for { msg <- MailBoxServiceClient .getMessage(GetMessageRequest(“email@example.org", msgId)) _ <- q.offer(MessageReceivedEvent(msg.body)) } yield Idle case (Idle, MessageReceivedEvent(body)) => for { _ <- ZIO.effectTotal { dom.document.getElementById("mailbody").innerHTML = body } } yield Idle } } yield ()
38. gRPC 101 - streaming def getMessages: RIO[MailBoxServiceClient, Unit] = for { // .. code omitted .. _ <- MailBoxServiceClient.getMessageStream( GetMailBoxRequest(“email@example.org“) ).foreach(addRow(_, q)) .fork } yield ()
39. Stream all the things SMTP Service Message Store GRPC Service GRPC Client ZMail Frontend ZMail Server
40. How to deploy the client • compile client side scala code to javascript:
 
 sbt fullOptJS 
 
 • create html + css files • package into a docker + envoy (check https://www.envoyproxy.io/)
41. ZIO-config for the server libraryDependencies += "dev.zio" %% "zio-config" % <version> libraryDependencies += "dev.zio" %% "zio-config-typesafe" % <version>
42. ZIO-config for the server { smtp { port = 8125 port = ${?SMTP_PORT} } grpc { port = 9000 } domain = "fizzle.email" tmpDir = "/tmp/zmail" } case class ZmailConfig(smtp: SmtpConfig, grpc: GrpcConfig, domain: String, tmpDir: String) case class SmtpConfig(port: Int) case class GrpcConfig(port: Int) object Config { val live = loadConfig.orDie.toLayer private def loadConfig = for { source <- ZIO.fromEither( TypesafeConfigSource.fromDefaultLoader) config <- ZIO.fromEither( read(descriptor[ZmailConfig] from source)) } yield config } application.conf
43. Wiring the server val zioServices = Clock.live ++ Blocking.live ++ Console.live val config: Layer[Nothing, ZConfig[ZmailConfig]] = Config.live val messageStore = MessageStore.live val grpcService = (Console.live ++ messageStore) >>> GrpcServer.service val appEnv = grpcService ++ messageStore ++ zioServices ++ config val mainLogic = for { config <- getConfig[ZmailConfig].toManaged_ _ <- SmtpServer.smtpServer _ <- GrpcServer.server(config.grpc.port) _ <- ZManaged.fromEffect(putStrLn("Zmail has started")) } yield () override def run(args: List[String]) = mainLogic.useForever.provideLayer(appEnv).exitCode
44. Live recorded demo
45. Live Recorded demo
46. • Staying inside ZIO: gets you a long way • When learning ZIO[R, E, A]: work from right to left • participate in active community • many more cool things on their way Conclusions/learnings
47. • open source zmail source code for learning purposes • offer as public service on fizzle.email • dockerize for ‘internal’ deployments • use zio-logging • persistency: zio-sql, zio-query, ??? • web layer: zio-web • SSL on SMTP and on gRPC Next steps
48. • https://github.com/willemvermeer/zmail - ZMail source code • http://fizzle.email/ - ZMail live in action • https://zio.github.io/zio-nio/ - ZIO-NIO • https://scalapb.github.io/zio-grpc/ - ZIO-gRPC • https://github.com/thesamet/AnyHike - ZIO-gRPC example project • https://www.lihaoyi.com/fastparse/ - Li Haoyi’s FastParse project • https://www.zionomicon.com/ to keep learning • https://github.com/o4oren/Ad-Hoc-Email-Server - ZMail use case inspiration Further reading
49. Thank you! @willemvermeer