How to code a SMTP service in Go

Sometimes it is useful to receive data in the form of e-mails instead of having traditional HTTP(S) APIs.

A traditional case for it is handling customer replies to incident tickets, for which it is much easier for them to reply to an e-mail notification from their mailbox than it would be for them to access the ticketing system to reply.

Fortunately, Go makes it easy to write such SMTP servers, in a way which even resembles HTTP server. See how it is done.

Transcript

1. Frédéric G. MARAND - 23 avril 2023
   A Go SMTP service
   Devoxx FR 2023
   1
   

   View Slide

2. © 2023 OSInet
   The problem to solve
   2
   

   View Slide

3. © 2023 OSInet
   How to automate inbound emails handling
   Use case for the idea: a business
   partner that does only exposes issues
   using emails instead of APIs.
   
   
   Other use case: handling ticket
   responses by email instead of forcing
   customers to go to the ticketing tool.
   
   
   Also, intellectual curiosity 👻
   
   
   The problem to solve
   3
   

   View Slide

4. © 2023 OSInet 4
   Solutions
   • Pull
   
   
   • POP3
   
   
   • IMAP4
   
   
   • Custom APIs (e.g. CDO, MAPI)
   • Push
   
   
   • (E)SMTP
   
   
   • Homegrown (this presentation)
   
   
   • AWS SES
   
   
   • Lambda
   
   
   • SNS topic
   
   
   • S3
   
   
   • Autres : header, bounce, stop
   
   
   • Google AppEngine SMTP ➢ HTTP
   
   
   • inbound_services.mail in app.yaml
   
   
   

   View Slide

5. © 2023 OSInet 5
   Pros
   • Pull
   
   
   • Fully capable and secure SMTP
   server upstream
   • Push
   
   
   • SMTPD server code is incredibly
   simple
   
   
   • SES for push is also fully capable and
   secure
   
   
   • SES ➢ Lambda is simple and cheap at
   that scale
   
   
   • SES output is simpler than MIME
   
   
   • GAE Inbound mail is HTTP
   

   View Slide

6. © 2023 OSInet 6
   Cons
   • Pull
   
   
   • POP3/IMAP APIs are complex
   and brittle
   
   
   • CDO is limited to MS Exchange
   
   
   • MAPI is…. Better forgotten
   
   
   • Maintenance on mail servers
   actually requires work
   • Push
   
   
   • Needs dedicated subdomain with MX
   record. Even for SES
   
   
   • Homegrown code needs and FW/
   WAF rules to allow inbound SMTP
   
   
   • Homegrown microservice will have to
   be strictly delimited to remain sage
   
   
   • SES API is proprietary
   
   
   • GAE does not parse like SES
   

   View Slide

7. © 2023 OSInet
   • At demo/PoC scale, or if vendor lock-in is a real issue, a homegrown very limited server is better
   
   
   • Otherwise SES → Lambda is probably
   
   
   • Not much more complex
   
   
   • Very reliable
   
   
   • Observable
   
   
   • Still very cheap
   
   
   • GAE SMTP ➢ HTTP is a legacy service
   
   
   • Pull is more for personal, one-off or management actions (e.g. export, backup of existing service)
   7
   Choosing
   

   View Slide

8. © 2023 OSInet
   Let’s do it
   8
   

   View Slide

9. © 2023 OSInet
   SMTP push means :
   
   
   • Messages routed by an MX, so…
   
   
   • Need one or more MX records…
   
   
   • In a specific subdomain
   
   
   Inbound SMTP on platform (not SES/GAE) means:
   
   
   • Open flow to port 25, 587, 465, or 2525
   
   
   • Configure firewall / WAF
   9
   Infra setup
   

   View Slide

10. © 2023 OSInet 10
    SMTP (very) simplified steps
    •HELO / EHLO
    
    
    •MAIL FROM:
    
    
    •RCPT TO:
    
    
    •DATA
    
    
    •Headers: From,To,Subject,others
    
    
    •Body
    
    
    •“.”
    
    
    •QUIT
    
    
    Note: no whitespace after “.” In MAIL FROM / RCPT TO.
    

    View Slide

11. © 2023 OSInet 11
    Bradfitz/go-smtpd
    •smtpd.Connection : actual TCP connection
    
    
    •smtpd.Server
    
    
    •Addr: listen
    
    
    •Hostname: announce
    
    
    •(Read|Write)Timeout: DoS defense
    
    
    •PlainAuth: for SSL connections
    
    
    •.OnNewConnection
    
    
    •.OnNewMail
    
    
    •smtpd.Envelope
    

    View Slide

12. © 2023 OSInet 12
    Smtp.Envelope contents
    • Unreliable network info
    
    
    • SmtpRemote
    
    
    • SmtpNetwork
    
    
    • Unreliable protocol addresses
    
    
    • MailFrom
    
    
    • RcptTo
    
    
    • Lines (data)
    
    
    • Headers
    
    
    • Body
    

    View Slide

13. © 2023 OSInet 13
    Hooking Envelope events
    
    
    Library emits events while parsing the inbound message
    
    
    • AddRecipient: check recipients existence/interest
    
    
    • BeginData: set up write destination
    
    
    • Write: write parsed body
    
    
    • Close: finish writes
    

    View Slide

14. © 2023 OSInet 14
    Handling headers
    
    
    All processing is provided by the Go stdlib:
    
    
    • Headers :
    

    net/mail.Message.Header
    
    
    • Email address parsing :
    

    net/mail.AddressParser
    
    
    • Reading body : net/mail.Message.Body
    

    is an io.Reader
    

    View Slide

15. © 2023 OSInet 15
    In practice
    
    
    • Define an Envelope implementation
    
    
    • Suggestion: parse body on close
    
    
    • doesn't scale but enough for these needs
    
    
    • at that point, header and body are already parsed and ready
    
    
    • write them to a net/mail.Message
    
    
    • Add custom logic as needed
    

    View Slide

16. © 2023 OSInet
    Sample code
    16
    

    View Slide

17. © 2023 OSInet 17
    Sample inbound data
    

    View Slide

18. © 2023 OSInet 18
    Sample Server events listening
    

    View Slide

19. © 2023 OSInet 19
    Sample Envelope events handling
    

    View Slide

20. © 2023 OSInet 20
    Sample main parsing
    

    View Slide

21. © 2023 OSInet
    The complete code
    21
    https://code.osinet.fr/fgm/smtpd
    

    View Slide

22. © 2023 OSInet
    Demo time
    22
    

    View Slide

23. © 2023 OSInet 23
    How to run the demo
    
    
    • git clone https://code.osinet.fr/fgml/smtpd
    
    
    • cd smtp
    
    
    • In one terminal tab:
    
    
    • go run ./cmd
    
    
    • In another terminal tab:
    
    
    • nc localhost 25 < fixture/simple.eml
    

    View Slide

24. © 2023 OSInet 24
    Results
    
    
    

    View Slide

25. © OSInet
    Thanks for
    watching
    Questions ?
    fgm@osinet.fr
    

    View Slide