It's new, it's shiny, it's... email?

Email has been around forever and is taken for granted – but it has become more complicated in recent years. Newer features like SPF, DKIM, and DMARC are misunderstood and, judging by the questions I answer on Stack Overflow, many are getting even simple things wrong. So we'll go back to basics, review how email works, how to avoid common mistakes, take advantage of its new powers, and stay out of the spam folder of history.

This talk was given at ConFoo 2020 in Montreal, Canada

Transcript

1. It’s new, it’s shiny, it’s... email? Marcus Bointon Confoo 2020 Montréal Canada Marcus Bointon @SynchroM Email / Confoo 2020
2. Marcus Bointon @SynchroM Email / Confoo 2020 History • First email sent in 1971 by Ray Tomlinson • Email first defined formally by RFC733 in 1977 • "Modern" email defined in RFC821 and RFC822 in 1982 • First spam sent by DEC in 1978 • Now about 250 billion email messages sent every day • It's simple, universal, and much cooler than you think!
3. Marcus Bointon @SynchroM Email / Confoo 2020 Email anatomy • Message formats • RFC822 • MIME • Transport protocols • SMTP • POP3 • IMAP
4. Marcus Bointon @SynchroM Email / Confoo 2020 Email transport: SMTP • Simple Mail Transfer Protocol • RFC821, 2821, 5321 • 7-bit • Lots of extension RFCs • 1869, 1870, 2821, 2920, 3030, 3207, 4954, 5321, 6152, 6531, ... • Chatty protocol • Store & forward
5. Marcus Bointon @SynchroM Email / Confoo 2020 Connected protocols: HTTP Client Proxy Server
6. Marcus Bointon @SynchroM Email / Confoo 2020 Relay Relay Store & Forward: SMTP Sender Relay Receiver
7. Marcus Bointon @SynchroM Email / Confoo 2020 SMTP Transport Encryption • SMTP with TLS • "SMTPS": 1997 – implicit TLS on port 465 • "SMTP+STARTTLS": 1998 – explicit TLS on port 587 or 25
8. Marcus Bointon @SynchroM Email / Confoo 2020 An SMTP conversation HELO/EHLO STARTTLS EHLO AUTH MAIL FROM RCPT TO DATA QUIT/RSET < 220 mail.example.com ESMTP Hello, please be nice > EHLO sender.example.com < 250-mail.example.com < 250-PIPELINING < 250-SIZE 134217728 < 250-VRFY < 250-ETRN < 250-STARTTLS < 250-ENHANCEDSTATUSCODES < 250-8BITMIME < 250-DSN < 250 SMTPUTF8
9. Marcus Bointon @SynchroM Email / Confoo 2020 An SMTP conversation HELO/EHLO STARTTLS EHLO AUTH MAIL FROM RCPT TO DATA QUIT/RSET > STARTTLS < 220 2.0.0 Ready to start TLS > EHLO sender.example.com < 250-mail.example.com < 250-PIPELINING < 250-SIZE 134217728 < 250-VRFY < 250-ETRN < 250-AUTH PLAIN LOGIN XOAUTH2 < 250-ENHANCEDSTATUSCODES < 250-8BITMIME < 250-DSN < 250 SMTPUTF8
10. Marcus Bointon @SynchroM Email / Confoo 2020 An SMTP conversation HELO/EHLO STARTTLS EHLO AUTH MAIL FROM RCPT TO DATA QUIT/RSET > AUTH LOGIN < 334 VXNlcm5hbWU6 ("Username:") > dXNlckBleGFtcGxlLmNvbQ== < 334 UGFzc3dvcmQ6 ("Password:") > cGFzc3dvcmQ= < 235 2.7.0 Authentication successful
11. Marcus Bointon @SynchroM Email / Confoo 2020 An SMTP conversation HELO/EHLO STARTTLS EHLO AUTH MAIL FROM RCPT TO DATA QUIT/RSET > MAIL FROM:<user@example.com> "Envelope Sender" < 250 2.1.0 Ok > RCPT TO:<alice@example.com> "To" < 250 2.1.5 Ok > RCPT TO:<carol@example.com> "CC" < 250 2.1.5 Ok > RCPT TO:<eve@example.com> "BCC" < 250 2.1.5 Ok > MAIL FROM:<> "Bounce"
12. Marcus Bointon @SynchroM Email / Confoo 2020 An SMTP conversation HELO/EHLO STARTTLS EHLO AUTH MAIL FROM RCPT TO DATA QUIT/RSET > DATA < 354 End data with <CR><LF>.<CR><LF> > Date: Sun, 16 Feb 2020 22:29:13 +0000 > To: alice@example.com > CC: carol@example.com > From: Bob User <user@example.com> > Subject: Test > Message-ID: <lWUlxsntiT2Y0@sender.example.com> > > Hello > > . < 250 2.0.0 Ok: queued as 3993B6007A
13. Marcus Bointon @SynchroM Email / Confoo 2020 An SMTP conversation HELO/EHLO STARTTLS EHLO AUTH MAIL FROM RCPT TO DATA QUIT/RSET QUIT 221 2.0.0 Bye
14. Marcus Bointon @SynchroM Email / Confoo 2020 I know what a valid email address is
15. Marcus Bointon @SynchroM Email / Confoo 2020 I know what a valid email address is a@b user+!#$%&'*-/=?_{}|~test@example.com first.last @example.org user.@gmail.com (foo)us(bar)er@exam(ba(boo)z)ple.com(quux) user@café.com (IDN punycoded: xn--caf-dma.com) 语嫣@例例.cn (\r\n RCPT TO:user@example.com\r\n DATA \\\nSubject: spam10\\ \n\r\n Hello,\r\nthis is spam.\\\n.\r\n QUIT\r\n ) a@example.net https://sphinx.mythic-beasts.com/~pdw/cgi-bin/emailvalidate
16. Marcus Bointon @SynchroM Email / Confoo 2020 Date: Thu, 27 Feb 2020 11:02:40 +0000 From: Marcus <marcus@example.com> To: Yann <yann@example.com> Message-ID: <s0x95Aa2EFq@example.com> Subject: ConFoo rocks! Hey Yann, Thanks for making ConFoo awesome! Marcus Message format • RFC822... • Headers • Addresses, subject, ID... • A blank line • Body (ASCII!) • MIME RFC2045, 1996
17. Marcus Bointon @SynchroM Email / Confoo 2020 MIME structures • Simple MIME-Version: 1.0 Content-Type: text/html; charset=us-ascii <html> <head> <title>HTML email test</title> </head> <body> <h1>HTML email!</h1> </body> </html>
18. Marcus Bointon @SynchroM Email / Confoo 2020 MIME structures • Simple • Multipart MIME-Version: 1.0 Content-Type: multipart/...; boundary="b1_jZIgjXuUO3UB1PZDJxT2rlz97gHBehtnbtvXYlvGw" Content-Transfer-Encoding: 8bit This is a multi-part message in MIME format. ...
19. Marcus Bointon @SynchroM Email / Confoo 2020 MIME structures • Simple • Multipart • Mixed MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="b1_jZIgjXuUO3UB1PZDJxT2rlz97gHBehtnbtvXYlvGw" Content-Transfer-Encoding: 8bit This is a multi-part message in MIME format. --b1_jZIgjXuUO3UB1PZDJxT2rlz97gHBehtnbtvXYlvGw Content-Type: text/plain; charset=us-ascii Hello --b1_jZIgjXuUO3UB1PZDJxT2rlz97gHBehtnbtvXYlvGw Content-Type: image/png; name="image.png" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=image.png iVBORw0KGgoAAAANSUhEUgAAAEgAAABICAIAAADajyQQAAAAA3NCSVQICAjb4U/ gAAAACXBIWXMA...
20. Marcus Bointon @SynchroM Email / Confoo 2020 MIME structures • Simple • Multipart • Mixed • Related Content-Type: multipart/related; boundary="b1_WKl5pYvja2BPUdJxJWVvgc2Ypbp5KdivfKdE1r7Dbb8" Content-Transfer-Encoding: 8bit This is a multi-part message in MIME format. --b1_WKl5pYvja2BPUdJxJWVvgc2Ypbp5KdivfKdE1r7Dbb8 Content-Type: text/html; charset=us-ascii <img src="cid:myimage" alt="image"> --b1_WKl5pYvja2BPUdJxJWVvgc2Ypbp5KdivfKdE1r7Dbb8 Content-Type: image/png Content-Transfer-Encoding: base64 Content-ID: <myimage> Content-Disposition: inline iVBORw0KGgoAAAANSUhEUgAAAEgAAABICAIAAADajyQQAA...
21. Marcus Bointon @SynchroM Email / Confoo 2020 MIME structures • Simple • Multipart • Mixed • Related • Alternative Content-Type: multipart/alternative; boundary="b1_geLK2kYyrLH0T0PD4vel8nNr6onwF92LYy2LEaUJBbM" Content-Transfer-Encoding: 8bit This is a multi-part message in MIME format. --b1_geLK2kYyrLH0T0PD4vel8nNr6onwF92LYy2LEaUJBbM Content-Type: text/plain; charset=us-ascii Plain text body --b1_geLK2kYyrLH0T0PD4vel8nNr6onwF92LYy2LEaUJBbM Content-Type: text/html; charset=us-ascii <strong>HTML</strong> --b1_geLK2kYyrLH0T0PD4vel8nNr6onwF92LYy2LEaUJBbM Content-Type: text/calendar; method=CANCEL; charset=iso-8859-1 Content-Transfer-Encoding: 8bit BEGIN:VCALENDAR
22. Marcus Bointon @SynchroM Email / Confoo 2020 MIME structures • Simple • Multipart • Mixed • Related • Alternative • Nesting Content-Type: multipart/alternative; boundary="b1_JfLD5ltK7H7BNgzjlA45XEiNOgNPnWUiZLYGR0X7JY" --b1_JfLD5ltK7H7BNgzjlA45XEiNOgNPnWUiZLYGR0X7JY Content-Type: text/plain; charset=utf-8 Plain text --b1_JfLD5ltK7H7BNgzjlA45XEiNOgNPnWUiZLYGR0X7JY Content-Type: multipart/related; boundary="b2_JfLD5ltK7H7BNgzjlA45XEiNOgNPnWUiZLYGR0X7JY"; type="text/html" --b2_JfLD5ltK7H7BNgzjlA45XEiNOgNPnWUiZLYGR0X7JY Content-Type: text/html; charset=utf-8 <img src="cid:myimage" alt="image"> --b2_JfLD5ltK7H7BNgzjlA45XEiNOgNPnWUiZLYGR0X7JY Content-Type: image/png Content-Transfer-Encoding: base64 Content-ID: <myimage> iVBORw0KGgoAAAANSUhEUgAAAVQAAABaCAIAAABojTkmAAAWjklEQ...
23. Marcus Bointon @SynchroM Email / Confoo 2020 Email: A Vulnerability Buffet • Lots of attack surface in base specification • No authentication – forgeable • No source restrictions – anyone can send for any domain • Open relays – spoofing / phishing / reputation damage • Cleartext protocol – leak content • Cleartext messages – leak content • No integrity – message tampering
24. Marcus Bointon @SynchroM Email / Confoo 2020 SPF: Who can send? • Sender Policy Framework: where messages from your domain may originate • Somewhat blunt • If you allow gmail, all gmail IPs can fake mail from you • Doesn't look at message content, only SMTP envelope sender • DNS record: • v=spf1 ip4:192.168.1.0/24 ip6:fe80:e0a:4c6:2f80::/64 a mx include:myesp.net ~all
25. Marcus Bointon @SynchroM Email / Confoo 2020 DKIM: Did you really say that? • Cryptographic signatures of headers & body • Sign with private key; verify with public key in DNS • Proves origin, time, and sender (and provides non-repudiation) • Prevents tampering • Allows addition of extra headers without breaking signature • Canonicalisation to allow limited mangling!
26. Marcus Bointon @SynchroM Email / Confoo 2020 • phpmailer._domainkey.example.com. 10800 IN TXT "v=DKIM1; k=rsa; p=MIIBIjA...KaQIDAQAB" Date: Sun, 16 Feb 2020 21:42:57 +0000 From: Test <testtest@test.com> Message-ID: <Quwr4v48qgvoejKJswEcmZ8@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 DKIM-Signature: v=1; d=example.com; s=phpmailer; a=rsa-sha256; q=dns/txt; t=1581889377; c=relaxed/simple; h=Date:From:Message-ID:Content-Type:To:Subject; bh=zS7KNTV0HyeorkDDGwxB1AV6enuRKzO5rthkhdHIRnY=; b=IKPNrDlWNSHS2cwEtu+ilotkcMISl490yvkBkceYC5c... 20aa76/tWwx3tYepx9uU+EYQ/6d0RRm3sRkNgvuycDdFV4I2MIylUXcSyS 8faA12OpwQnqBcsNm7tYWd0/as0tXY0wDGQaCi3KjAA3zgY0kQ== To: testtest@test.com Subject: hello hello DKIM: Did you really say that?
27. Marcus Bointon @SynchroM Email / Confoo 2020 DMARC: What shall we do? • SPF & DKIM detect problems • DMARC says what to do with failures: • Nothing, quarantine, reject • Alignment – "From" header address, not SMTP envelope sender • Helps to diagnose network issues and identify attempted phishing attacks • DNS record published at _dmarc.example.com • v=DMARC1; p=reject; pct=100; rua=mailto:dmarcreports@example.com;
28. Marcus Bointon @SynchroM Email / Confoo 2020 MTA-STS • Like HTTP Strict Transport Security (HSTS) for email • "This domain always encrypts SMTP" • Servers that do support MTA-STS will NOT send you email without TLS • Servers that don't support MTA-STS WILL send you email • Serve a profile from https://mta-sts.example.com/.well-known/mta-sts.txt • DNS: _mta-sts.example.com. 300 IN TXT "v=STSv1; id=2020021701" version: STSv1 mode: testing mx: mail1.example.com mx: mail2.example.com max_age: 604800
29. Marcus Bointon @SynchroM Email / Confoo 2020 SMTP TLS-RPT • SMTP TLS Reporting; RFC8460 • If you have MTA-STS, how do you know if connections are failing? • Can send reports via email (like DMARC) or POST to a URL • Add DNS record _smtp._tls.example.com. 300 IN TXT "v=TLSRPTv1; rua=mailto:rpt@example.com" • Test it: https://aykevl.nl/apps/mta-sts/
30. Marcus Bointon @SynchroM Email / Confoo 2020 Message encryption • Sub-optimal – impossible to encrypt headers, no e2e encryption • S/MIME – RFC5751 • Universally supported • Rarely used • PGP/GPG • Poor support • Used often!
31. Marcus Bointon @SynchroM Email / Confoo 2020 Receiving email • Receiving SMTP server applies checks at various points: • Reverse DNS lookup – checks that EHLO name matches IP • SPF / Forgery checks on MAIL FROM • DKIM signature verification & spam filtering after DATA • Receiver adds headers to message before delivery: • Return-Path – contains the SMTP envelope sender • Received – may be multiple copies depending on relays • "X-*" informative headers, e.g. spam score, SPF, DKIM, and DMARC results
32. Marcus Bointon @SynchroM Email / Confoo 2020 Bounces and deliverability • Bounces sent to the SMTP envelope sender • Identified as bounces because the return-path header is empty: • Return-Path: <> • VERP addressing lets you know where the message was sent • Error codes let you know why message was bounced: • 5.1.1 unknown recipient • 5.7.1 spam filtered
33. Marcus Bointon @SynchroM Email / Confoo 2020 The future of email? • AMP for email (ick!) • hey.com (@dhh)
34. Marcus Bointon @SynchroM Email / Confoo 2020 Thank you! • Feedback please! • Marcus Bointon • marcus@smartmessages.net • @SynchroM & @PrivacySpider • Synchro on GitHub & Stack Exchange