How to code a SMTP service in Go

Sometimes it is useful to receive data in the form of e-mails instead of having traditional HTTP(S) APIs.

A traditional case for it is handling customer replies to incident tickets, for which it is much easier for them to reply to an e-mail notification from their mailbox than it would be for them to access the ticketing system to reply.

Fortunately, Go makes it easy to write such SMTP servers, in a way which even resembles HTTP server. See how it is done.

Transcript

1. Frédéric G. MARAND - 23 avril 2023 A Go SMTP service Devoxx FR 2023 1
2. © 2023 OSInet The problem to solve 2
3. © 2023 OSInet How to automate inbound emails handling Use case for the idea: a business partner that does only exposes issues using emails instead of APIs. Other use case: handling ticket responses by email instead of forcing customers to go to the ticketing tool. Also, intellectual curiosity 👻 The problem to solve 3
4. © 2023 OSInet 4 Solutions • Pull • POP3 • IMAP4 • Custom APIs (e.g. CDO, MAPI) • Push • (E)SMTP • Homegrown (this presentation) • AWS SES • Lambda • SNS topic • S3 • Autres : header, bounce, stop • Google AppEngine SMTP ➢ HTTP • inbound_services.mail in app.yaml
5. © 2023 OSInet 5 Pros • Pull • Fully capable and secure SMTP server upstream • Push • SMTPD server code is incredibly simple • SES for push is also fully capable and secure • SES ➢ Lambda is simple and cheap at that scale • SES output is simpler than MIME • GAE Inbound mail is HTTP
6. © 2023 OSInet 6 Cons • Pull • POP3/IMAP APIs are complex and brittle • CDO is limited to MS Exchange • MAPI is…. Better forgotten • Maintenance on mail servers actually requires work • Push • Needs dedicated subdomain with MX record. Even for SES • Homegrown code needs and FW/ WAF rules to allow inbound SMTP • Homegrown microservice will have to be strictly delimited to remain sage • SES API is proprietary • GAE does not parse like SES
7. © 2023 OSInet • At demo/PoC scale, or if vendor lock-in is a real issue, a homegrown very limited server is better • Otherwise SES → Lambda is probably • Not much more complex • Very reliable • Observable • Still very cheap • GAE SMTP ➢ HTTP is a legacy service • Pull is more for personal, one-off or management actions (e.g. export, backup of existing service) 7 Choosing
8. © 2023 OSInet Let’s do it 8
9. © 2023 OSInet SMTP push means : • Messages routed by an MX, so… • Need one or more MX records… • In a specific subdomain Inbound SMTP on platform (not SES/GAE) means: • Open flow to port 25, 587, 465, or 2525 • Configure firewall / WAF 9 Infra setup
10. © 2023 OSInet 10 SMTP (very) simplified steps •HELO / EHLO <domain> •MAIL FROM:<mail> •RCPT TO:<mail> •DATA •Headers: From,To,Subject,others •Body •“.” •QUIT Note: no whitespace after “.” In MAIL FROM / RCPT TO.
11. © 2023 OSInet 11 Bradfitz/go-smtpd •smtpd.Connection : actual TCP connection •smtpd.Server •Addr: listen •Hostname: announce •(Read|Write)Timeout: DoS defense •PlainAuth: for SSL connections •.OnNewConnection •.OnNewMail •smtpd.Envelope
12. © 2023 OSInet 12 Smtp.Envelope contents • Unreliable network info • SmtpRemote • SmtpNetwork • Unreliable protocol addresses • MailFrom • RcptTo • Lines (data) • Headers • Body
13. © 2023 OSInet 13 Hooking Envelope events Library emits events while parsing the inbound message • AddRecipient: check recipients existence/interest • BeginData: set up write destination • Write: write parsed body • Close: finish writes
14. © 2023 OSInet 14 Handling headers All processing is provided by the Go stdlib: • Headers : 
 net/mail.Message.Header • Email address parsing : 
 net/mail.AddressParser • Reading body : net/mail.Message.Body 
 is an io.Reader
15. © 2023 OSInet 15 In practice • Define an Envelope implementation • Suggestion: parse body on close • doesn't scale but enough for these needs • at that point, header and body are already parsed and ready • write them to a net/mail.Message • Add custom logic as needed
16. © 2023 OSInet Sample code 16
17. © 2023 OSInet 17 Sample inbound data
18. © 2023 OSInet 18 Sample Server events listening
19. © 2023 OSInet 19 Sample Envelope events handling
20. © 2023 OSInet 20 Sample main parsing
21. © 2023 OSInet The complete code 21 https://code.osinet.fr/fgm/smtpd
22. © 2023 OSInet Demo time 22
23. © 2023 OSInet 23 How to run the demo • git clone https://code.osinet.fr/fgml/smtpd • cd smtp • In one terminal tab: • go run ./cmd • In another terminal tab: • nc localhost 25 < fixture/simple.eml
24. © 2023 OSInet 24 Results
25. © OSInet Thanks for watching Questions ? fgm@osinet.fr