Tweet
Tweet

Slide 1

Slide 1 text

Frédéric G. MARAND - 23 avril 2023 A Go SMTP service Devoxx FR 2023 1

Slide 2

Slide 2 text

© 2023 OSInet The problem to solve 2

Slide 3

Slide 3 text

© 2023 OSInet How to automate inbound emails handling Use case for the idea: a business partner that does only exposes issues using emails instead of APIs. Other use case: handling ticket responses by email instead of forcing customers to go to the ticketing tool. Also, intellectual curiosity 👻 The problem to solve 3

Slide 4

Slide 4 text

© 2023 OSInet 4 Solutions • Pull • POP3 • IMAP4 • Custom APIs (e.g. CDO, MAPI) • Push • (E)SMTP • Homegrown (this presentation) • AWS SES • Lambda • SNS topic • S3 • Autres : header, bounce, stop • Google AppEngine SMTP ➢ HTTP • inbound_services.mail in app.yaml

Slide 5

Slide 5 text

© 2023 OSInet 5 Pros • Pull • Fully capable and secure SMTP server upstream • Push • SMTPD server code is incredibly simple • SES for push is also fully capable and secure • SES ➢ Lambda is simple and cheap at that scale • SES output is simpler than MIME • GAE Inbound mail is HTTP

Slide 6

Slide 6 text

© 2023 OSInet 6 Cons • Pull • POP3/IMAP APIs are complex and brittle • CDO is limited to MS Exchange • MAPI is…. Better forgotten • Maintenance on mail servers actually requires work • Push • Needs dedicated subdomain with MX record. Even for SES • Homegrown code needs and FW/ WAF rules to allow inbound SMTP • Homegrown microservice will have to be strictly delimited to remain sage • SES API is proprietary • GAE does not parse like SES

Slide 7

Slide 7 text

© 2023 OSInet • At demo/PoC scale, or if vendor lock-in is a real issue, a homegrown very limited server is better • Otherwise SES → Lambda is probably • Not much more complex • Very reliable • Observable • Still very cheap • GAE SMTP ➢ HTTP is a legacy service • Pull is more for personal, one-off or management actions (e.g. export, backup of existing service) 7 Choosing

Slide 8

Slide 8 text

© 2023 OSInet Let’s do it 8

Slide 9

Slide 9 text

© 2023 OSInet SMTP push means : • Messages routed by an MX, so… • Need one or more MX records… • In a specific subdomain Inbound SMTP on platform (not SES/GAE) means: • Open flow to port 25, 587, 465, or 2525 • Configure firewall / WAF 9 Infra setup

Slide 10

Slide 10 text

© 2023 OSInet 10 SMTP (very) simplified steps •HELO / EHLO •MAIL FROM: •RCPT TO: •DATA •Headers: From,To,Subject,others •Body •“.” •QUIT Note: no whitespace after “.” In MAIL FROM / RCPT TO.

Slide 11

Slide 11 text

© 2023 OSInet 11 Bradfitz/go-smtpd •smtpd.Connection : actual TCP connection •smtpd.Server •Addr: listen •Hostname: announce •(Read|Write)Timeout: DoS defense •PlainAuth: for SSL connections •.OnNewConnection •.OnNewMail •smtpd.Envelope

Slide 12

Slide 12 text

© 2023 OSInet 12 Smtp.Envelope contents • Unreliable network info • SmtpRemote • SmtpNetwork • Unreliable protocol addresses • MailFrom • RcptTo • Lines (data) • Headers • Body

Slide 13

Slide 13 text

© 2023 OSInet 13 Hooking Envelope events Library emits events while parsing the inbound message • AddRecipient: check recipients existence/interest • BeginData: set up write destination • Write: write parsed body • Close: finish writes

Slide 14

Slide 14 text

© 2023 OSInet 14 Handling headers All processing is provided by the Go stdlib: • Headers : 
 net/mail.Message.Header • Email address parsing : 
 net/mail.AddressParser • Reading body : net/mail.Message.Body 
 is an io.Reader

Slide 15

Slide 15 text

© 2023 OSInet 15 In practice • Define an Envelope implementation • Suggestion: parse body on close • doesn't scale but enough for these needs • at that point, header and body are already parsed and ready • write them to a net/mail.Message • Add custom logic as needed

Slide 16

Slide 16 text

© 2023 OSInet Sample code 16

Slide 17

Slide 17 text

© 2023 OSInet 17 Sample inbound data

Slide 18

Slide 18 text

© 2023 OSInet 18 Sample Server events listening

Slide 19

Slide 19 text

© 2023 OSInet 19 Sample Envelope events handling

Slide 20

Slide 20 text

© 2023 OSInet 20 Sample main parsing

Slide 21

Slide 21 text

© 2023 OSInet The complete code 21 https://code.osinet.fr/fgm/smtpd

Slide 22

Slide 22 text

© 2023 OSInet Demo time 22

Slide 23

Slide 23 text

© 2023 OSInet 23 How to run the demo • git clone https://code.osinet.fr/fgml/smtpd • cd smtp • In one terminal tab: • go run ./cmd • In another terminal tab: • nc localhost 25 < fixture/simple.eml

Slide 24

Slide 24 text

© 2023 OSInet 24 Results

Slide 25

Slide 25 text

© OSInet Thanks for watching Questions ? [email protected]