Share
Embed
What happens in ZIO, stays in ZIO
Talk on ZIO presented at functionalscala.com 2020
Willem Vermeer
December 02, 2020
Transcript
49 slides
-
Willem Vermeer - 3 December 2020 What happens in ZIO,
stays in ZIO functionalscala.com @willemvermeer -
Agenda • introduction • use case: ZMail • ZIO-NIO •
ZIO-gRPC • ZIO-config • demo, conclusion and next steps -
Agenda • introduction • use case: ZMail • ZIO-NIO •
ZIO-gRPC • ZIO-config • demo, conclusion and next steps Yo Wilhelm, who are you anyway? -
Amsterdam.scala - 2 March 2019 OK Boomer, can we get
started already? -
ZMail use case SMTP Service Message Store GRPC Service GRPC
Client ZMail Frontend ZMail Server -
Staying in ZIO - why? • only one library to
learn • error management is the same from top to bottom • one thread/fiber management model -
ZMail use case SMTP Service Message Store GRPC Service GRPC
Client ZMail Frontend ZMail Server -
Simple Mail Transfer Protocol 101 Client: telnet smtp.example.com 25 Server:
220 smtp.example.com Example SMTP Service Client: HELO smtp.sender.org Server: 250 Hello Client: MAIL FROM: <[email protected]> Server: 250 [email protected]... Sender OK Client: RCPT TO: <[email protected]> Server: 250 [email protected]... Recipient OK Client: DATA Server: 354 Please start mail input Client: Subject: Datatest Client: Data...data...data... Client: . Server: 250 Mail queued for delivery. Client: QUIT Server: 221 Goodbye. -
Using ZIO-NIO to listen on port 25 val smtpServer =
AsynchronousServerSocketChannel().mapM { socket => for { address <- SocketAddress.inetSocketAddress(“0.0.0.0", 25) _ <- socket.bind(address) _ <- socket.accept.preallocate .flatMap( _.ensuring(putStrLn("Connection closed")) .use(channel => receive(channel)) .fork ) .forever.fork } yield () } -
Using ZIO-NIO to read a line of text def receive(channel:
AsynchronousSocketChannel) = ZStream .fromEffectOption( channel .readChunk(1024) .tap(c => putStrLn(s"Read chunk of length ${c.size}")) .orElseFail(None) ) .flattenChunks .transduce(ZTransducer.utf8Decode) .run(Sink.foldLeft("")(_ + _)) -
Using ZIO-NIO to write a line of text def writeToChannel(
txt: String, channel: AsynchronousSocketChannel ) = ZStream .fromChunk(Chunk.fromArray(txt.toCharArray)) .transduce(Charset.Standard.utf8.newEncoder.transducer()) .foreachChunk(chunk => channel.writeChunk(chunk)) -
Process line by line…follow protocol Client: MAIL FROM: <[email protected]> def
nextCommand = for { line <- readLineFromChannel _ <- logInbound(line) command <- SmtpParser.parse(line) } yield command -
Cheat: use fastparse for SMTP protocol parsing object SmtpParser {
def EOL[_: P]: P[Unit] = P("\r" ~ "\n") // see https://tools.ietf.org/html/rfc5321#section-4.1.2 def digit[_: P]: P[Unit] = P(CharIn("0-9")) def alpha[_: P]: P[Unit] = P(CharIn("a-z") | CharIn("A-Z")) def letters[_: P]: P[Unit] = P(alpha.rep(1)) def letdig[_: P]: P[String] = P((alpha | digit).!) def prascii[_: P]: P[Unit] = P( "!" | "#" | "$" | "%" | "&" | "'" | "*" | "+" | "-" | "/" | "=" | "?" | "^" | "_" | "`" | "{" | "|" | "}" | "~" ) def atext[_: P]: P[Unit] = P(alpha | digit | prascii) def atom[_: P]: P[Unit] = P((alpha | digit | prascii).rep(1)) def dotstring[_: P]: P[Unit] = P(atom ~ ("." ~ atom).rep(0)) def ldhstr[_: P]: P[Unit] = P((letdig | "-").rep(0) ~ letdig) def subdomain[_: P]: P[Unit] = P(letdig ~ ldhstr.?) def domain[_: P]: P[Unit] = P(subdomain ~ ("." ~ subdomain).rep(0)) def qtextsmtp[_: P]: P[Unit] = P(" " | "!" | CharIn("#-[") | CharIn("]- ~")) // bnf: %d32-33 / %d35-91 / %d93-126 def quotedpairsmtp[_: P]: P[Unit] = P("\\" ~ CharIn(" -~")) // bnf: %d92 %d32-126 def qcontentsmtp[_: P]: P[Unit] = P(qtextsmtp | quotedpairsmtp) def quotedstring[_: P]: P[Unit] = P("\"" ~ qcontentsmtp.rep(0) ~ "\"") def localpart[_: P]: P[Unit] = P(dotstring | quotedstring) def snum[_: P]: P[Unit] = P(digit.rep(min = 1, max = 3)) def ipv4addressliteral[_: P]: P[Unit] = P(snum ~ ("." ~ snum).rep(exactly = 3)) // TODO: ipv6 support def dcontent[_: P]: P[Unit] = P(CharIn("!-Z") | CharIn("^-~")) // bnf: %d33-90 / %d94-126 def standardizedtag[_: P]: P[Unit] = ldhstr def generaladdressliteral[_: P]: P[Unit] = P(standardizedtag ~ ":" ~ dcontent.rep(1)) def addressliteral[_: P]: P[Unit] = P("[" ~ (ipv4addressliteral | generaladdressliteral) ~ "]") def domainC[_: P]: P[Domain] = domain.!.map(Domain) def mailbox[_: P]: P[Unit] = P(localpart ~ "@" ~ (domain | addressliteral)) def atdomain[_: P]: P[Unit] = P("@" ~ domain) def adl[_: P]: P[Unit] = P(atdomain ~ ("," ~ atdomain).rep(0)) // only the first @domain will be picked up def path[_: P]: P[String] = P("<" ~ (adl ~ ":").? ~ mailbox.! ~ ">") def forwardpath[_: P]: P[String] = P(path) def reversepath[_: P]: P[String] = P(path | "<>").map(_.toString) def esmtpvalue[_: P]: P[Unit] = P((CharIn("!-<") | CharIn(">- ~")).rep(1)) // bnf 1*(%d33-60 / %d62-126) def esmtpkeyword[_: P]: P[Unit] = P(letdig ~ (letdig | "-").rep(0)) def esmtpparam[_: P]: P[Unit] = P(esmtpkeyword ~ ("=" ~ esmtpvalue).?) def mailparameters[_: P]: P[Unit] = P(esmtpparam ~ (" " ~ esmtpparam).rep(0)) def helo[_: P]: P[Helo] = P("HELO" ~ " " ~ domainC).map(Helo) def ehlo[_: P]: P[Ehlo] = P("EHLO" ~ " " ~ domainC).map(Ehlo) def mailfrom[_: P]: P[MailFrom] = P("MAIL FROM:" ~ reversepath ~ (" " ~ mailparameters).?).map(p => MailFrom(ReversePath(p))) def rcptto[_: P]: P[RcptTo] = P("RCPT TO:" ~ forwardpath ~ (" " ~ mailparameters).?).map(p => RcptTo(ReversePath(p))) def data[_: P]: P[Data.type] = P("DATA").map(_ => Data) def quit[_: P]: P[Quit.type] = P("QUIT").map(_ => Quit) def command[_: P]: P[Command] = P((helo | ehlo | mailfrom | rcptto | data | quit) ~ EOL) def parse(line: String): IO[ParseError, Command] = fastparse.parse(line, SmtpParser.command(_)) match { case Parsed.Success(cmd, _) => ZIO.succeed(cmd) case Parsed.Failure(label, index, _) => ZIO.fail(ParseError(s"ParserError $label at $index for $line")) } case class ParseError(msg: String) extends Exception(msg) } https://www.lihaoyi.com/fastparse/ -
object SmtpParser { def EOL[_: P]: P[Unit] = P("\r" ~
"\n") // see https://tools.ietf.org/html/rfc5321#section-4.1.2 def digit[_: P]: P[Unit] = P(CharIn("0-9")) def alpha[_: P]: P[Unit] = P(CharIn("a-z") | CharIn("A-Z")) def letters[_: P]: P[Unit] = P(alpha.rep(1)) def letdig[_: P]: P[String] = P((alpha | digit).!) def prascii[_: P]: P[Unit] = P( "!" | "#" | "$" | "%" | "&" | "'" | "*" | "+" | "-" | "/" | "=" | "?" | "^" | "_" | "`" | "{" | "|" | "}" | "~" ) def atext[_: P]: P[Unit] = P(alpha | digit | prascii) def atom[_: P]: P[Unit] = P((alpha | digit | prascii).rep(1)) def dotstring[_: P]: P[Unit] = P(atom ~ ("." ~ atom).rep(0)) def ldhstr[_: P]: P[Unit] = P((letdig | "-").rep(0) ~ letdig) def subdomain[_: P]: P[Unit] = P(letdig ~ ldhstr.?) def domain[_: P]: P[Unit] = P(subdomain ~ ("." ~ subdomain).rep(0)) def qtextsmtp[_: P]: P[Unit] = P(" " | "!" | CharIn("#-[") | CharIn("]- ~")) // bnf: %d32-33 / %d35-91 / %d93-126 def quotedpairsmtp[_: P]: P[Unit] = P("\\" ~ CharIn(" -~")) // bnf: %d92 %d32-126 def qcontentsmtp[_: P]: P[Unit] = P(qtextsmtp | quotedpairsmtp) def quotedstring[_: P]: P[Unit] = P("\"" ~ qcontentsmtp.rep(0) ~ "\"") def localpart[_: P]: P[Unit] = P(dotstring | quotedstring) def snum[_: P]: P[Unit] = P(digit.rep(min = 1, max = 3)) def ipv4addressliteral[_: P]: P[Unit] = P(snum ~ ("." ~ snum).rep(exactly = 3)) // TODO: ipv6 support def dcontent[_: P]: P[Unit] = P(CharIn("!-Z") | CharIn("^-~")) // bnf: %d33-90 / %d94-126 def standardizedtag[_: P]: P[Unit] = ldhstr def generaladdressliteral[_: P]: P[Unit] = P(standardizedtag ~ ":" ~ dcontent.rep(1)) def addressliteral[_: P]: P[Unit] = P("[" ~ (ipv4addressliteral | generaladdressliteral) ~ "]") def domainC[_: P]: P[Domain] = domain.!.map(Domain) def mailbox[_: P]: P[Unit] = P(localpart ~ "@" ~ (domain | addressliteral)) def atdomain[_: P]: P[Unit] = P("@" ~ domain) def adl[_: P]: P[Unit] = P(atdomain ~ ("," ~ atdomain).rep(0)) // only the first @domain will be picked up def path[_: P]: P[String] = P("<" ~ (adl ~ ":").? ~ mailbox.! ~ ">") def forwardpath[_: P]: P[String] = P(path) def reversepath[_: P]: P[String] = P(path | "<>").map(_.toString) def esmtpvalue[_: P]: P[Unit] = P((CharIn("!-<") | CharIn(">- ~")).rep(1)) // bnf 1*(%d33-60 / %d62-126) def esmtpkeyword[_: P]: P[Unit] = P(letdig ~ (letdig | "-").rep(0)) def esmtpparam[_: P]: P[Unit] = P(esmtpkeyword ~ ("=" ~ esmtpvalue).?) def mailparameters[_: P]: P[Unit] = P(esmtpparam ~ (" " ~ esmtpparam).rep(0)) def helo[_: P]: P[Helo] = P("HELO" ~ " " ~ domainC).map(Helo) def ehlo[_: P]: P[Ehlo] = P("EHLO" ~ " " ~ domainC).map(Ehlo) def mailfrom[_: P]: P[MailFrom] = P("MAIL FROM:" ~ reversepath ~ (" " ~ mailparameters).?).map(p => MailFrom(ReversePath(p))) def rcptto[_: P]: P[RcptTo] = P("RCPT TO:" ~ forwardpath ~ (" " ~ mailparameters).?).map(p => RcptTo(ReversePath(p))) def data[_: P]: P[Data.type] = P("DATA").map(_ => Data) def quit[_: P]: P[Quit.type] = P("QUIT").map(_ => Quit) def command[_: P]: P[Command] = P((helo | ehlo | mailfrom | rcptto | data | quit) ~ EOL) def parse(line: String): IO[ParseError, Command] = fastparse.parse(line, SmtpParser.command(_)) match { case Parsed.Success(cmd, _) => ZIO.succeed(cmd) case Parsed.Failure(label, index, _) => ZIO.fail(ParseError(s"ParserError $label at $index for $line")) } case class ParseError(msg: String) extends Exception(msg) } Cheat: use fastparse for SMTP protocol parsing https://www.lihaoyi.com/fastparse/ But you said we would stay in ZIO!! -
Lift fastparse result into ZIO https://www.lihaoyi.com/fastparse/ object SmtpParser { def
parse(line: String): IO[ParseError, Command] = fastparse.parse(line, SmtpParser.command(_)) match { case Parsed.Success(cmd, _) => ZIO.succeed(cmd) case Parsed.Failure(label, index, _) => ZIO.fail(ParseError( s"ParserError $label at $index for $line” )) } } -
Lift fastparse result into ZIO https://www.lihaoyi.com/fastparse/ object SmtpParser { def
parse(line: String): IO[ParseError, Command] = fastparse.parse(line, SmtpParser.command(_)) match { case Parsed.Success(cmd, _) => ZIO.succeed(cmd) case Parsed.Failure(label, index, _) => ZIO.fail(ParseError( s"ParserError $label at $index for $line” )) } } // input: “MAIL FROM: <[email protected]>” case class MailFrom(from: String) extends Command -
How to store the message? SMTP Service Message Store GRPC
Service GRPC Client ZMail Frontend ZMail Server -
How to store the messages? Use a zio.Ref object MessageStore
{ trait Service { def storeMessage(msg: Message): UIO[Unit] } case class MessageStoreImpl(store: Ref[Map[EmailAddress, MessageList]]) extends Service { override def storeMessage(msg: Message): UIO[Unit] = for { ref <- store.get _ <- ref.get(msg.to) match { case None => store.update(_ + (msg.to -> MessageList(Seq(msg)))) case Some(messageList) => store.update(_ + (msg.to -> messageList.add(msg))) } } yield () } } -
Web interface - gRPC SMTP Service Message Store GRPC Service
GRPC Client ZMail Frontend ZMail Server -
zio-gRPC • same (main) author wrote ScalaPB • built with
ZIO and ZIO-Streams • generates gRPC server code and gRPC client code • gRPC Client can be compiled to Scala.JS => runs in the browser!! • supports one-off calls as well as streaming (both directions) -
gRPC 101 - service definitions in protobuf service MailBoxService {
rpc GetMailBox (GetMailBoxRequest) returns (MailBox) {} rpc GetMessage (GetMessageRequest) returns (MailBoxEntry) {} rpc GetMessageStream (GetMailBoxRequest) returns (stream MailBoxEntry) {} } message GetMailBoxRequest { string username = 1; } message MailBox { repeated MailBoxEntry entries = 1; int32 total = 2; } .. more definitions .. -
Web interface - gRPC server SMTP Service Message Store GRPC
Service GRPC Client ZMail Frontend ZMail Server -
gRPC 101 - code is generated from service defs object
ZioService { trait ZMailBoxService[-R, -Context] extends ZGeneratedService[R, Context, ZMailBoxService] { self => def getMailBox(request: GetMailBoxRequest): ZIO[R with Context, Status, MailBox] } type MailBoxService = ZMailBoxService[Any, Any] // .. much more code .. } final case class GetMailBoxRequest( username: String = "", unknownFields: UnknownFieldSet = UnknownFieldSet.empty ) extends scalapb.GeneratedMessage with Updatable[GetMailBoxRequest] { .. } -
gRPC 101 - work with generated code class LiveMailBoxService(messageStore: MessageStore.Service)
extends ZIOService.MailBoxService { override def getMailBox(request: GetMailBoxRequest): UIO[MailBox] = { val username = request.username for { msgs <- messageStore.getMessages(EmailAddress(username)) result <- ZIO.succeed( MailBox( entries = msgs.map(m => MailBoxEntry.of(m)), total = msgs.size ) ) } yield result } -
Web interface - gRPC client SMTP Service Message Store GRPC
Service GRPC Client ZMail Frontend ZMail Server -
gRPC 101 - generated client code object ZioService { object
MailBoxServiceClient { trait ZService[R] { def getMailBox(request: GetMailBoxRequest): ZIO[R, Status, MailBox] } // .. much more code .. } } -
gRPC 101 - work with client code def getMessages: ZIO[MailBoxServiceClient,
Status, MailBox] = for { response <- MailBoxServiceClient .getMailBox(GetMailBoxRequest(“[email protected]")) } yield response -
gRPC 101 - process the results (1) def getMessages: ZIO[MailBoxServiceClient,
Status, Unit] = for { response <- MailBoxServiceClient .getMailBox(GetMailBoxRequest(“[email protected]")) q <- Queue.unbounded[Event] _ <- fillTable(response, q) } yield () def fillTable(mailbox: MailBox, q: Queue[Event]) = ZIO.foreach_(mailbox.entries)(addRow(_, q)) -
gRPC 101 - process the results (2) def findTable(): UIO[Option[HTMLTableElement]]
= UIO.effectTotal { dom.document.getElementById("inbox") match { case table: HTMLTableElement => Some(table) case _ => None } } def addRow(entry: MailBoxEntry, q: Queue[Event]) = for { table <- findTable().some rt <- ZIO.runtime[Any] _ <- ZIO.effectTotal { table.insertRow() match { case tr: HTMLTableRowElement => val td = tr.insertCell() td.innerText = entry.subject // .. more td’s .. tr.onclick = { (_: MouseEvent) => rt.unsafeRunSync(q.offer(RowClickEvent(entry.id))) } case _ => } } } yield () -
gRPC 101 - process the results (3) def getMessages: RIO[MailBoxServiceClient,
Unit] = for { response <- MailBoxServiceClient.getMailBox(GetMailBoxRequest(“[email protected]")) q <- Queue.unbounded[Event] _ <- fillTable(response, q) _ <- Stream .fromQueue(q) .foldM(Idle: State) { case (Idle, RowClickEvent(msgId)) => for { msg <- MailBoxServiceClient .getMessage(GetMessageRequest(“[email protected]", msgId)) _ <- q.offer(MessageReceivedEvent(msg.body)) } yield Idle case (Idle, MessageReceivedEvent(body)) => for { _ <- ZIO.effectTotal { dom.document.getElementById("mailbody").innerHTML = body } } yield Idle } } yield () -
gRPC 101 - streaming def getMessages: RIO[MailBoxServiceClient, Unit] = for
{ // .. code omitted .. _ <- MailBoxServiceClient.getMessageStream( GetMailBoxRequest(“[email protected]“) ).foreach(addRow(_, q)) .fork } yield () -
Stream all the things SMTP Service Message Store GRPC Service
GRPC Client ZMail Frontend ZMail Server -
How to deploy the client • compile client side scala
code to javascript: sbt fullOptJS • create html + css files • package into a docker + envoy (check https://www.envoyproxy.io/) -
ZIO-config for the server libraryDependencies += "dev.zio" %% "zio-config" %
<version> libraryDependencies += "dev.zio" %% "zio-config-typesafe" % <version> -
ZIO-config for the server { smtp { port = 8125
port = ${?SMTP_PORT} } grpc { port = 9000 } domain = "fizzle.email" tmpDir = "/tmp/zmail" } case class ZmailConfig(smtp: SmtpConfig, grpc: GrpcConfig, domain: String, tmpDir: String) case class SmtpConfig(port: Int) case class GrpcConfig(port: Int) object Config { val live = loadConfig.orDie.toLayer private def loadConfig = for { source <- ZIO.fromEither( TypesafeConfigSource.fromDefaultLoader) config <- ZIO.fromEither( read(descriptor[ZmailConfig] from source)) } yield config } application.conf -
Wiring the server val zioServices = Clock.live ++ Blocking.live ++
Console.live val config: Layer[Nothing, ZConfig[ZmailConfig]] = Config.live val messageStore = MessageStore.live val grpcService = (Console.live ++ messageStore) >>> GrpcServer.service val appEnv = grpcService ++ messageStore ++ zioServices ++ config val mainLogic = for { config <- getConfig[ZmailConfig].toManaged_ _ <- SmtpServer.smtpServer _ <- GrpcServer.server(config.grpc.port) _ <- ZManaged.fromEffect(putStrLn("Zmail has started")) } yield () override def run(args: List[String]) = mainLogic.useForever.provideLayer(appEnv).exitCode -
• Staying inside ZIO: gets you a long way •
When learning ZIO[R, E, A]: work from right to left • participate in active community • many more cool things on their way Conclusions/learnings -
• open source zmail source code for learning purposes •
offer as public service on fizzle.email • dockerize for ‘internal’ deployments • use zio-logging • persistency: zio-sql, zio-query, ??? • web layer: zio-web • SSL on SMTP and on gRPC Next steps -
• https://github.com/willemvermeer/zmail - ZMail source code • http://fizzle.email/ - ZMail
live in action • https://zio.github.io/zio-nio/ - ZIO-NIO • https://scalapb.github.io/zio-grpc/ - ZIO-gRPC • https://github.com/thesamet/AnyHike - ZIO-gRPC example project • https://www.lihaoyi.com/fastparse/ - Li Haoyi’s FastParse project • https://www.zionomicon.com/ to keep learning • https://github.com/o4oren/Ad-Hoc-Email-Server - ZMail use case inspiration Further reading