Barry Warsaw - aiosmtpd - A better asyncio based SMTP server

smtpd.py has been in the standard library for many years. It's been a common tool for deploying SMTP and LMTP servers that handle email-based communication in Python, providing both basic protocol implementations and a fundamental module for higher level tools, such as lazr.smtptest for testing email clients. Based on asyncore and asynchat, smtpd.py is showing its age, and its API is unwieldy.

Fortunately, there's a new alternative available. aiosmtpd is a modern reinvention based on asyncio, with all the improvements that come along with such a new implementation. It provides servers for both the SMTP and LMTP protocols, as well as a higher level "controller" API for testing SMTP and LMTP clients. It exposes a much better API for customization, allowing the user to associate a simple "handler" to process incoming messages without having to worry about the details of the protocols, and it provides some useful hooks for subclassing.

This talk will describe the purpose and history of smtpd.py and aiosmtpd, show how users can extend the servers and implement specialized handlers, and show how applications can use the testing API for ensuring that their email sending applications do the right things. Examples will be taken from GNU Mailman 3, which uses aiosmtpd extensively.

https://us.pycon.org/2017/schedule/presentation/147/

Transcript

1. aiosmtpd An asyncio-based SMTP server Pycon 2017 Portland, Oregon May 21, 2017 Barry Warsaw
2. Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. - jwz (though maybe not) Zawinski's Law of Software Envelopment
3. Simple Mail Transport Protocol (and its little cousin LMTP Local Mail Transport Protocol)
4. Motivation
5. Why an SMTP server? Testing email clients Pythonically configurable server Experimental protocols Production quality server
6. SMTP and LMTP Some relevant RFCs RFC 821 (1982) - Original SMTP standard RFC 5321 (2008) - Defines ESMTP RFC 2033 - Defines LMTP RFC 1870 - message sizes RFC 6531 - internationalization (and so on...!)
7. SMTP in a nutshell 220 subdivisions Python SMTP 1.0 From: Geddy <geddy@example.com> To: Alex <alex@example.com> Bcc: Neil <neil@example.com> Subject: New Music Hey, we need to record a new album! HELO limelight 250 subdivisions MAIL FROM:<geddy@example.com> 250 OK RCPT TO:<alex@example.com> 250 OK RCPT TO:<neil@example.com> 250 OK DATA 354 End data with <CR><LF>.<CR><LF> . 250 OK ​ RFC 5321 RFC 5322 QUIT 221 Bye
8. Variations Extended SMTP (ESMTP) EHLO limelight Local Mail Transport Protocol (LMTP) LHLO limelight Original SMTP HELO limelight
9. (Very) brief history
10. asynchat/asyncore smtpd.py (2001, Python 2.1a2) lazr.smtptest (2009) asyncio (2014, Python 3.4) aiosmtp (Benjamin Bader) aiosmtpd (April 2015 - Washington DC)
11. Moving pictures
12. Components SMTP/LMTP classes - protocol Session & envelope - state Handlers - events Controller - optional, but helpful!
13. Session socket peer SSL details ESMTP flag event loop Newly created on every client connection
14. Envelope MAIL FROM address RCPT TO addresses Original content (always bytes) Content (bytes or str) Additional ESMTP options HELO/RSET state
15. SMTP class Verbs as coroutines STARTTLS UTF-8 handling Calls the event handler Exception hooks Extensible via subclassing
16. SMTP Verbs
17. HELO async def smtp_HELO(self, hostname): if not hostname: await self.push('501 Syntax: HELO hostname') return self._set_rset_state() status = await self._call_handler_hook('HELO', hostname) if status is MISSING: self.session.host_name = hostname status = '250' {}'.format(self.hostname) await self.push(status)
18. Get the server's time from aiosmtpd.smtp import SMTP from datetime import datetime class MySMTPish(SMTP): async def smtp_NOW(self, arg): if arg == 'UTC': now = datetime.utcnow() elif not arg: now = datetime.now() else: await self.push('501 Syntax: NOW [UTC]') now = now.replace(microsecond=0) await self.push('250 {}'.format(now))
19. Get the server's time % telnet localhost 8025 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 presto Python SMTP 1.0 NOW 250 2017-05-16 12:05:05 NOW 250 2017-05-16 12:05:08 NOW UTC 250 2017-05-16 19:05:10 NOW AND THEN 501 Syntax: NOW [UTC] QUIT 221 Bye
20. Event Handlers
21. Handlers Introspection for verbs Coroutines Return status code No default behavior async def handle_VERB(self, server, session, envelope)
22. HELO counter class Counter: helo_counter = 0 async def handle_HELO(self, server, session, envelope, hostname): self.session.hostname = hostname self.helo_counter += 1 return '250 OK' smtp = SMTP(Counter()) run_server_for_a_while(smtp) print('We saw {} HELOs'.format(smtp.event_handler.helo_counter))
23. Controller Runs server in subthread Start/stop semantics Pass in hostname, port, etc. Override factory() to use custom SMTP subclasses
24. Command line
25. $ python3 -m aiosmtpd -n -c MyHandler arg1 arg2 $ telnet localhost 8025 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 subdivisions Python SMTP 1.0 QUIT 221 Bye Connection closed by foreign host.
26. CLI arguments class MyHandler: @classmethod def from_cli(cls, parser, *args): kws = convert(args) return cls(**kws)
27. Acknowledgments Benjamin Bader - aiosmtp (no 'd') DC Hackathon Crew: Jason Coombs Andrew Kuchling Eric V. Smith Barry Warsaw R. David Murray (honorary) Konstantin Volkov Matthias Rav (and others... thanks!)
28. aiosmtpd https://github.com/aio-libs/aiosmtpd http://aiosmtpd.readthedocs.io/ Requires at least Python 3.4 (but see GH#16)
29. Barry Warsaw barry@{python,list,debian}.org barry@ubuntu.com @pumpichank github.com/warsaw gitlab.com/warsaw