Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to code a SMTP service in Go

How to code a SMTP service in Go

Sometimes it is useful to receive data in the form of e-mails instead of having traditional HTTP(S) APIs.

A traditional case for it is handling customer replies to incident tickets, for which it is much easier for them to reply to an e-mail notification from their mailbox than it would be for them to access the ticketing system to reply.

Fortunately, Go makes it easy to write such SMTP servers, in a way which even resembles HTTP server. See how it is done.

Frédéric G. MARAND

April 13, 2023
Tweet

More Decks by Frédéric G. MARAND

Other Decks in Programming

Transcript

  1. Frédéric G. MARAND - 23 avril 2023
    A Go SMTP service
    Devoxx FR 2023
    1

    View full-size slide

  2. © 2023 OSInet
    The problem to solve
    2

    View full-size slide

  3. © 2023 OSInet
    How to automate inbound emails handling
    Use case for the idea: a business
    partner that does only exposes issues
    using emails instead of APIs.


    Other use case: handling ticket
    responses by email instead of forcing
    customers to go to the ticketing tool.


    Also, intellectual curiosity 👻


    The problem to solve
    3

    View full-size slide

  4. © 2023 OSInet 4
    Solutions
    • Pull


    • POP3


    • IMAP4


    • Custom APIs (e.g. CDO, MAPI)
    • Push


    • (E)SMTP


    • Homegrown (this presentation)


    • AWS SES


    • Lambda


    • SNS topic


    • S3


    • Autres : header, bounce, stop


    • Google AppEngine SMTP ➢ HTTP


    • inbound_services.mail in app.yaml


    View full-size slide

  5. © 2023 OSInet 5
    Pros
    • Pull


    • Fully capable and secure SMTP
    server upstream
    • Push


    • SMTPD server code is incredibly
    simple


    • SES for push is also fully capable and
    secure


    • SES ➢ Lambda is simple and cheap at
    that scale


    • SES output is simpler than MIME


    • GAE Inbound mail is HTTP

    View full-size slide

  6. © 2023 OSInet 6
    Cons
    • Pull


    • POP3/IMAP APIs are complex
    and brittle


    • CDO is limited to MS Exchange


    • MAPI is…. Better forgotten


    • Maintenance on mail servers
    actually requires work
    • Push


    • Needs dedicated subdomain with MX
    record. Even for SES


    • Homegrown code needs and FW/
    WAF rules to allow inbound SMTP


    • Homegrown microservice will have to
    be strictly delimited to remain sage


    • SES API is proprietary


    • GAE does not parse like SES

    View full-size slide

  7. © 2023 OSInet
    • At demo/PoC scale, or if vendor lock-in is a real issue, a homegrown very limited server is better


    • Otherwise SES → Lambda is probably


    • Not much more complex


    • Very reliable


    • Observable


    • Still very cheap


    • GAE SMTP ➢ HTTP is a legacy service


    • Pull is more for personal, one-off or management actions (e.g. export, backup of existing service)
    7
    Choosing

    View full-size slide

  8. © 2023 OSInet
    Let’s do it
    8

    View full-size slide

  9. © 2023 OSInet
    SMTP push means :


    • Messages routed by an MX, so…


    • Need one or more MX records…


    • In a specific subdomain


    Inbound SMTP on platform (not SES/GAE) means:


    • Open flow to port 25, 587, 465, or 2525


    • Configure firewall / WAF
    9
    Infra setup

    View full-size slide

  10. © 2023 OSInet 10
    SMTP (very) simplified steps
    •HELO / EHLO


    •MAIL FROM:


    •RCPT TO:


    •DATA


    •Headers: From,To,Subject,others


    •Body


    •“.”


    •QUIT


    Note: no whitespace after “.” In MAIL FROM / RCPT TO.

    View full-size slide

  11. © 2023 OSInet 11
    Bradfitz/go-smtpd
    •smtpd.Connection : actual TCP connection


    •smtpd.Server


    •Addr: listen


    •Hostname: announce


    •(Read|Write)Timeout: DoS defense


    •PlainAuth: for SSL connections


    •.OnNewConnection


    •.OnNewMail


    •smtpd.Envelope

    View full-size slide

  12. © 2023 OSInet 12
    Smtp.Envelope contents
    • Unreliable network info


    • SmtpRemote


    • SmtpNetwork


    • Unreliable protocol addresses


    • MailFrom


    • RcptTo


    • Lines (data)


    • Headers


    • Body

    View full-size slide

  13. © 2023 OSInet 13
    Hooking Envelope events


    Library emits events while parsing the inbound message


    • AddRecipient: check recipients existence/interest


    • BeginData: set up write destination


    • Write: write parsed body


    • Close: finish writes

    View full-size slide

  14. © 2023 OSInet 14
    Handling headers


    All processing is provided by the Go stdlib:


    • Headers :

    net/mail.Message.Header


    • Email address parsing :

    net/mail.AddressParser


    • Reading body : net/mail.Message.Body

    is an io.Reader

    View full-size slide

  15. © 2023 OSInet 15
    In practice


    • Define an Envelope implementation


    • Suggestion: parse body on close


    • doesn't scale but enough for these needs


    • at that point, header and body are already parsed and ready


    • write them to a net/mail.Message


    • Add custom logic as needed

    View full-size slide

  16. © 2023 OSInet
    Sample code
    16

    View full-size slide

  17. © 2023 OSInet 17
    Sample inbound data

    View full-size slide

  18. © 2023 OSInet 18
    Sample Server events listening

    View full-size slide

  19. © 2023 OSInet 19
    Sample Envelope events handling

    View full-size slide

  20. © 2023 OSInet 20
    Sample main parsing

    View full-size slide

  21. © 2023 OSInet
    The complete code
    21
    https://code.osinet.fr/fgm/smtpd

    View full-size slide

  22. © 2023 OSInet
    Demo time
    22

    View full-size slide

  23. © 2023 OSInet 23
    How to run the demo


    • git clone https://code.osinet.fr/fgml/smtpd


    • cd smtp


    • In one terminal tab:


    • go run ./cmd


    • In another terminal tab:


    • nc localhost 25 < fixture/simple.eml

    View full-size slide

  24. © 2023 OSInet 24
    Results


    View full-size slide

  25. © OSInet
    Thanks for
    watching
    Questions ?
    [email protected]

    View full-size slide