and Python • SRE/Platform for Geomagical Labs, part of IKEA • We do CV/AR for the home 2 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Hi there, I'm Noah Kantrowitz. I'm an SRE at Geomagical Labs. We do computer vision and augmented reality stuff for IKEA.
@[email protected] And I'm not really here to talk about Python. Python is great and it's my primary tool for solving technical problems, but there's lots of other programming languages out there and some have features or workflows that Python lacks.
Bay Python 2024 – Noah Kantrowitz – @[email protected] The goal of this talk is not to advocate we immediately turn each of these features into a PEP, but to broaden our exposure to programming concepts and talk about the tradeoffs being made. If one of these ideas really grabs you, maybe try messing around with it in a library or see how it would work as a patch in CPython.
@[email protected] So I'm going to begin with PHP. I don't mean this ironically or to be a language hipster, PHP is pretty cool. I've spent the last few years on a side project almost entirely in PHP and it's given me a refreshed appreciation.
• Deep editor integration 6 North Bay Python 2024 – Noah Kantrowitz – @[email protected] The biggest thing I've grown to like about PHP is the simplicity of it. Each file maps unambiguously to a single URL, deployment is as simple as updating that file any way you prefer. Most editors these days support remote editing via SFTP so you can just click "new file" and start coding. This puts even "low boilerplate" web frameworks in Python to shame. And once you get going, changes in the code show up immediately, no need to restart a server or configure automatic reloads. You just write code and it runs, simple as that.
Foo; class Bar { } • No require required "autoload": { "psr-4": {"Foo\\": "src/"} } 7 North Bay Python 2024 – Noah Kantrowitz – @[email protected] In Python we like to say explicit is better than implicit, and overall I agree but man is juggling imports annoying sometimes. Even with editor help, the dance of "write a line of code, scroll to the top of the file to add an import, then down again" is jarring. Autoloaders bridge this gap while keeping it mostly explicit. If you enforce a one-to-one mapping between code symbols and filesystem paths then you know exactly where the symbol load is going to end up. This is also deeply integrated into the package manager, Composer, to pre-bake the mapping between class names and files, so the underlying import doesn't have to scan a packages directory. Honestly I don't know if I would actually prefer this kind of system in Python, but it's something everyone should at least know about.
• Database libraries • Libcurl • PECL as a secondary stdlib 8 North Bay Python 2024 – Noah Kantrowitz – @[email protected] The standard library for PHP is a little different from Python. It has evolved towards a single niche, web application development, so it biases towards things needed in that space. MySQL and Postgres clients are included by default, as is a high-performance HTTP client. And then beyond that is PHP Extension Community Library or PECL. An extension is specifically something using C code and so has to be loaded into the PHP config at startup. The difference between PECL and PyPI is that for the major extensions supported by the PHP team, the docs live right alongside the normal PHP ones. There's no need to go searching for a README, it's all just there.
?> <p>and some more HTML<"p> <?php } ?> <"div> 9 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Alongside the standard library being adapted to its major use case, PHP itself is too. As a web-first language, simple templating is directly baked into the language. As f-strings have matured in Python, we've definitely moved towards this, but like before the most basic PHP use cases run laps around Python's explicitly simplest web frameworks.
@[email protected] Ruby is probably my second most used language over my career. While the days of "Python and Ruby" being thought of as relatively neck and neck have waned as Python has taken over more niches, it's still a vibrant language and community.
sys.intern("foo") • Strings with bene!ts • myfn(foo=1), maybe MyStrEnum.FOO 11 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Symbols are a thing in a lot of languages but I'm going to highlight them here. At their simplest, symbols are just fancy strings. But internally they are de-duplicated so checking equality between two symbols is a constant time operation. We do have this in Python via intern-ed strings, but it's rarely used. At a deeper level, having a dedicated syntax makes it easy to differentiate between "text" strings and, for lack of a better term, "symbolic" strings. These often serve very differrent functions in code and despite being the same data type, they shouldn't be mixed. We already kind of have this with the keyword argument syntax, while they are effectively strings there is a clear boundary as to what kind of thing they are. String enums can also be an option, though usually that means something a bit different.
return x + 1 foo(_inner) 12 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Blocks are something much more specific to Ruby. Again deep down this is something Python has too, lambda expressions as anonymous functions. But blocks do a bit more than that. First off it's not limited to a single expression which is handy for anything complicated. And second, the calling syntax with do lets you easily hang a block off any function. Passing around lambda literals in Python is definitely doable and readable in cases like sort keys, but the utility falls off pretty quickly so its community practice to declare a local function and pass it as a normal argument. This results in "backwards code" in most cases, where the read order is the reverse of the execution flow.
|val| .." end def _one(val): ..# def _two(val): ..# y = x.one(_one).two(_two) 13 North Bay Python 2024 – Noah Kantrowitz – @[email protected] This kind of "read order is execution order" goes even further with method chaning, something Ruby excels at. While Python's built-in functions do this backwards, we can build APIs like Django's ORM. But blocks allow it to go a lot further with the same effort.
x.d() 14 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Another language that has a different take on this is Dart's double dot cascading operator. This allows interrupting a chain to go "one level up" so to speak, like in a file system path, performing a sequence of operations on the same object rather than propagating the chain through the return value of each link. The allows method chaining in Dart without any extra work by API authors.
other cfg.baz(2, 3) end do # What if? x = with Foo as cfg: cfg.bar = 1 if other: cfg.baz(2, 3) 15 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Another place Ruby makes excellent use of blocks is its version of the builder pattern. Many complex object constructors take a block which exposes a little configuration API. This can allow very compact initialization without sacrificing flexibility for unusual conditions and is easier for static analysis like type checking.
Diaper Bank Network • Chicago Tool Library • Habitat For Humanity 16 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Going a bit broader than Ruby as a language, one big thing Ruby as a community has that I wish Python was better at is the Ruby for Good organization. It's a dedicated support structure for Ruby projects which work with specific non-profit organizations. We have a lot of paths for advancing Python itself, but sometimes it's good to do real, tangible good in the world.
Kantrowitz – @[email protected] After Ruby, the next logical place to look for ideas is the JavaScript and TypeScript worlds. I'm not going to repeat every feature shared by each of these languages like how JavaScript supports a nice anonymous function syntax too because we would be here all day, just focusing on the interesting bits.
x is None else x x.foo.bar() if x is not None and x.foo is not None else None 18 North Bay Python 2024 – Noah Kantrowitz – @[email protected] This one has been discussed at great length already in the Python community but I personally think they should be added and it's my talk so one more time: null coalescing operators are like "or" except checking explicitly for null rather than a falsey value. Safe navigation operators build on that to allow accessing a nested chain of fields, interrupting things and returning null if needed. As you can see in even this very small example, safe navigation operators dramatically streamline code. This is something I miss deeply when working in Python now.
} # What if? def foo(x: TypedDict["bar": str]): baz(x["bar"]) 19 North Bay Python 2024 – Noah Kantrowitz – @[email protected] TypeScript has always has a fine line to walk. It applies typed semantics to JavaScript but can't fundamentally reshape the language or it would be too alienating. So they knew the majority of objects passed around would be the equivalent of a dict no matter what. And they wanted to allow typing information to be introduced gradually in existing codebases. Out of this comes a requirement to support inline type declarations again arbitrary dicts. In Python terms this is kind of similar to dataclasses but on an ad hoc basis and without requiring either upstream or downstream code to change. While this is certainly not as nice as a fully typed codebase, it can be a smoother onramp for doing one small bit of code at a time.
.then(resp =" resp.json()) .then(data =" console.log(data.name)); async def foo(): resp = await fetch("/search") data = await resp.json() resp = await fetch(f"/detail/{data['id']}") 20 North Bay Python 2024 – Noah Kantrowitz – @[email protected] I'm kind of cheating here, we do have Promises in Python as part of the concurrent futures library. And perhaps their moment is over anyway with the continued uptake in explicit async await APIs in Python and elsewhere. But they are still worth studying even if only as part of our collective history. Promise chaining is a compact way to handle multiple interacting concurrent operations without dedicated syntax. I'm pulling my punches a bit with such a simplistic example, and in practice many Promise chains were much harder to read, but perhaps the technique will find another application some day.
@[email protected] On the topic of concurrency, Go. It's no secret that Go's structured concurrency has made it a frequent choice in complex systems code. And honestly I think Python is already heading in many of the same directions.
messages #- "ping" }() msg :" #-messages # Soon? messages = Channel[str]() def func(): messages.append("ping") subinterpreter.run(func) msg = messages.pop() 22 North Bay Python 2024 – Noah Kantrowitz – @[email protected] If you've seen anything about Go, it's probably this. Goroutines are a lightweight concurrency mechanism, more or less the same as we have with async functions except truly concurrent, and channels are queues to move small bits of data between them. I think this model is about to get very interesting in Python as more work is done with GIL-isolated subinterpreters. There is a future where we get something similar to these, with full multi core concurrency.
ok = x.(Foo) y.Bar(1) @typing.runtime_checkable class Foo(typing.Protocol): def bar(self, x: int) -# int: ..% class Impl: def bar(self, x: str) -# str: ..% x = Impl() isinstance(x, Foo) =' True x.bar(1) # Oops wrong type 23 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Runtime type checks are something Python has always kept at arms length. We do have some very basic runtime interface checks using the runtime checkable decorator for protocols, but it very specifically only checks if the right method names exist, not their type signatures. Compare to Go's runtime casts which do a more complete check and can either fail with an error or crash if you prefer. This would require something in CPython that understands how to compare those type signatures but perhaps we've come far enough to be ready for that.
• ./myapp • xkcd://1987 24 North Bay Python 2024 – Noah Kantrowitz – @[email protected] In a totally different direction, Go is among the best in class, alongside Rust, for easily building standalone executables, as well as cross-platform compilation. In the server world this makes deployment inside minimalist containers very easy because all you need from the OS is some TLS certificates and maybe a timezone definition or two. And for command line tools you can distribute a single binary without concern for the system library versions or anything else. Briefcase has recently added relatively simple CLI packaging support and I'm hopeful it will improve things here.
If you were only here for things with an even tiny chance of happening in Python, I wish you well but now I want to look at some of the long tail. Weird language features that you've probably heard about once or twice but it seems unlikely Python would ever even discuss. Even against those long odds, I think there are things to learn
-m pie 're.sub("VAR", "Hello")' input.txt 26 North Bay Python 2024 – Noah Kantrowitz – @[email protected] On the complete opposite end from easy packaging, easy oneliners. Perl has a lot of interesting design elements but by far the most useful to me day to day is perl pie, a sequence of command line arguments that says "take the list of input files, run this expression on each line of each file and then write back the results". Taken together this allows for a superset of sed, awk, and bash loops in an incredibly compact package, and without the myriad of provider variances in sed. Making a Python version of this in a library would be easy and there are some, but a real value of perl pie is you can drop onto any server from about 1995 onward and you've got a good chance of having a perl executable somewhere. Maybe if we add this to the stdlib now, in 10 years it can be similarly omnipresent?
IEnumerable<int> scoreQuery = from score in scores where score > 80 select score; • Dataframes • DuckDB 27 North Bay Python 2024 – Noah Kantrowitz – @[email protected] LINQ is short for Language Integrated Query, and it's basically "what if SQL was our API?". This is most direct when used as type of ORM, but C# took this further and you can build arbitrary providers on top of anything. Python has taken some baby steps down this path with cross-library standards for dataframes, and DuckDB on top of that gives you very expressive syntax to comb through data. What if that was language syntax anything could plug into?
(call/cc (lambda (c) (+ 1 (c 42)))) =" 42 def call_cc(f): ex = type('Continuation', (Exception,), {}) def c(val): raise ex(val) try: return f(c) except ex as exc: return exc.args[0] def foo(c): c(42) call_cc(foo) =# 42 28 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Call with current continuation will take some explaining so just bear with me. At the basic level, it's a kind of non-local return, like an exception. And indeed we can implement this basic take in Python using exceptions. Run a function with something to call when you're ready to return and it will bubble up to the same point as where call cc happened. This alone would be pretty boring though, the real power of continuations is that they can be stored, moved around, in some languages even serialized, all allowing you jump right back to the point in the code where the continuation was generated. This makes it something like a structured GOTO. Imagine being able to save and restore the state of a generator function as if it were any other object.
^ Respond to client and then wait for another request" blog backgroundColor: color 29 North Bay Python 2024 – Noah Kantrowitz – @[email protected] While continuations are most associated with Scheme, they are used very effectively in in the Smalltalk web framework Seaside. In Seaside, the whole state of the application is manged via pickleable continuations so that invocation of ColorPicker above is going to show a modal dialog in the web app, wait for user input, and then pick up right where the server-side code left off. All the cleanliness of a client-side application but on a server.
return f() def if_false(self, f): pass class False: def if_true(self, f): pass def if_false(self, f): return f() def _inner(): print("Yes") x.if_true(_inner) value = 0 def _other(): x = value <# 10 def _inner(): value += 1 x.if_true(_inner) return x y = True() y.do_while(_other) 30 North Bay Python 2024 – Noah Kantrowitz – @[email protected] And because I really love how Smalltalk does flow control here's a very rough port of it to Python. In short, Smalltalk doesn't have special syntax for flow control. It's all implemented via pure object oriented methods. "If" is a method on booleans, and so is "while". In practical implementations these get optimized into special constructs by most Smalltalk compilers but it's a great example of doing more with less in language design.
preForth interpreter loop. It reads tokens \ and handles them until an error occurs or the input is exhausted. : quit ( -" ) token \ get next token \ run interpreters ?: \ :-definition ?code \ code definitions ?pre \ pre* ?\ \ comment dup ?exit drop \ unhandled or EOF tail quit ; \ cycle 31 North Bay Python 2024 – Noah Kantrowitz – @[email protected] And if you want to go even more minimal dive into tiny forths. You can bootstrap a whole language interpreter with a few hundred bytes of assembly and 20 to 30 keywords. From that you build a basic parser, then an interpreter loop, then basic control flow, and whole programs. Forth kernels have been written as small as 400 bytes of machine code. Minimalism is not a specific goal for Python but it's a good reminder of how little is needed for a productive system. What would a minimal Python look like, something just powerful enough to implement the rest?
– Oops! init(_Args) -" SupFlags = #%strategy =' one_for_one, intensity =' 1, period =' 5}, ChildSpecs = [#%id =' myapp, start =' {myapp, start_myapp, []}, restart =' permanent, shutdown =' brutal_kill, type =' worker, modules =' [myapp]}], {ok, {SupFlags, ChildSpecs}}. 32 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Another interesting blend of language features and frameworks is Erlang's Open Telephony Platform or OTP, specifically the Supervisor system. Erlang programs internally are, to use Python terms, a series of async tasks all running in parallel. We can certainly do this in Python but when one of those tasks crashes, it's often difficult to sort out how to respond exactly. OTP applies similar concepts from server and container management to individual tasks within a complex application. We can set the supervisor system to restart failed tasks automatically, customize exactly how that restart works, and even build dependency trees where one failed task should kick a whole pile of other tasks to ensure consistency. This is how Erlang builds incredibly high uptime systems like phone switches without requiring magical programmers who never write bugs.
= s1.create_task(cache_manager()) async with s1.sup(on_error=emit_log) as s2: s2.create_task(web_server(c)) async with loop.sup(watchdog=5) as s3: s3.create_task(cleanup()) 33 North Bay Python 2024 – Noah Kantrowitz – @[email protected] This could mesh well with the subinterpreter work we looked at before, and in fact Go's coroutines and channels were themselves heavily influenced by OTP's server and mailbox model. There could be a future where we have something like task groups not just as a synchronization mechanism, but also runtime supervision.
2024 – Noah Kantrowitz – @[email protected] I started off saying I wasn't aiming for you to go off and turn these ideas into PEPs, and while that is true, I wish it wasn't. A few of these features could be prototyped in a library and gain traction there, but most of these would require a core language change. The process for that has improved over the years, but the Steering Council has to walk a fine line as adding anything to CPython is a big commitment. This leaves us in kind of a bind, to get peer support for a new idea we need a core change and to make a core change we need peer support.
$typ + $b as $typ } } (defmacro setq2 (v1 v2 e) (list 'progn (list 'setq v1 e) (list 'setq v2 e))) 35 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Macros are a language construct to allow meta- programming inside the language itself. Add new syntax, change something that's already there, write code in ROT 13, the skys the limit. The two best places to look for ideas about macros are Rust and Lisp. C does also heavily feature macros but it takes a much less structured approach, treating source code as text to be templated and nothing more. This works, but Lisp and Rust show a deeper power, defining macros in terms of the symbols of the language but not the original syntax so we can create our own.
– @[email protected] Python does have a proposed macros system, heavily influenced by Rust's, allowing user code to register a macro provider at compile time which gets given an abstract syntax tree and is allowed to arbitrarily transform it. Unfortunately it seems to have fallen into the very same trap we were trying to escape and has had little discussion over the years.
NamedExpr(expr target, expr value) | BinOp(expr left, operator op, expr right) | UnaryOp(unaryop op, expr operand) | Lambda(arguments args, expr body) | IfExp(expr test, expr body, expr orelse) | Dict(expr* keys, expr* values) ..# 37 North Bay Python 2024 – Noah Kantrowitz – @[email protected] I want to start by talking about how Lisp does macros but before we can do that, we need to talk about how Lisp code is structured, and before that we need to talk about how Python does the same thing. Like most other languages, Python's parser produces an Abstract Syntax Tree or AST. This is not yet the bytecode which will actually be run, but it is a highly structured form ready for further compilation.
s-expressions • (a . b) (* 1 (+ 2 3)) _______ / ___\___ * / __\__ 1 / _\_ + / \ 2 3 38 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Lisp does have something like this, but it's far far simpler. Every piece of code turns into one of two things, either an atom or a pair of additional things. Compared to Python this is just the barest bit of structure, but it also means that macros can do almost anything you want as long is the input and output are both s-expressions.
Oh look, atom 'defmacro, record this into the list of macros • (* 1 (addone (- 2 1))) • Oh look, atom 'addone is a macro, do the macro thing! • Execute (list '+ '1 x) • Got back ('+ '1 ('- '2 '1)) • (* 1 (+ 1 (- 2 1))) 39 North Bay Python 2024 – Noah Kantrowitz – @[email protected] So the way defmacro works, very roughly speaking, is to fully parse the code into s-expressions, then start looking for special atoms. Defmacro is a special atom which takes a name for the macro, names for the arguments, and a code expression to evaulate when invoking the macro. Then if it later on sees that atom in the code it will expand the macro. That means it will take the arguments, which at this point are still just symbolic because the code hasn't been run yet, and evaluate the code of the macro, and put the resulting expression back in the original code.
Instead: Parse → Macro Compile → Macro Execute → Parse 40 North Bay Python 2024 – Noah Kantrowitz – @[email protected] The important takeaway here is that macros break the usual flow of things. Usually our code moves linearly from tokenizing to parsing to compiling to executing. Macros put their own little sub loop inside the parser step, compiling an executing the body of the macro and returning the output to the parser. This might seem like a minor thing but it's the pressure relief valve which allows us extend the parser in the standalone way, without whoever maintains that parser needing to put in any extra work.
• Function-like #[proc_macro] • pub fn foo(input: TokenStream) -> TokenStream { • Declarative macros – macro_rules! 41 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Rust has two main types of macros. Procedural macros are the equivalent of what we saw with Lisp. There's a few flavors depending on which kind of parsing you want to hook into, but at heart they are all function which takes a stream of tokens and return an updated stream of tokens with our output code. This allows effectively unlimited control, we can even operate on things which aren't valid Rust syntax in the first place, but often we just want a simpler shortcut within the usual syntax and for that we can use macro_rules!.
{ { let mut v = Vec:$new(); $( v.push($x); )* v } }; } let myvec: Vec<u32> = vec![1, 2, 3]; /" Becomes ..$ let myvec: Vec<u32> = { let mut v = Vec:&new(); v.push(1); v.push(2); v.push(3); v }; 42 North Bay Python 2024 – Noah Kantrowitz – @[email protected] Declaractive macros work on a similar basis to Python's match case statements, but the parameters are symbolic expressions or types and the code block is returned rather than executed. But we can still do some basic expansions and templating. This system goes much deeper than what I can show in 30 seconds but hopefully it gets the idea across. Almost any idea you can imagine could be prototyped inside a macro.
2024 – Noah Kantrowitz – @[email protected] So why look at all this? Personally I think it's just fun so see what else is out there. But if that's not enough then I certainly hope that learning about all these patterns and systems can help you better organize your code and APIs. Or at least shows you what other tools are available when you need them. And in the end, that makes us all better developers.
with Composer Expansive std lib Ruby Easy lambdas via blocks Flow programming Symbol literals Builder pattern Go Static compilation and FROM scratch Interfaces 45 North Bay Python 2024 – Noah Kantrowitz – @[email protected]
coderanger
lauravandoore
maggiecrowley
wjessup
stephaniewalter
aki_iinuma
jakevdp
yeseniaperezcruz
sugarenia
tammyeverts
bryan
mraible
garrettdimon
![25 North Bay Python 2024 – Noah Kantrowitz – @[email protected]](https://files.speakerdeck.com/presentations/fe9d72c4f1e44e4f9f9fafe08838c82a/slide_24.jpg){kind=link}
![C# AND LINQ int[] scores = [97, 92, 81, 60];](https://files.speakerdeck.com/presentations/fe9d72c4f1e44e4f9f9fafe08838c82a/slide_26.jpg){kind=link}
![RUST • Procedural macros • Custom #[derive] • Attribute-like #[proc_macro_attribute]](https://files.speakerdeck.com/presentations/fe9d72c4f1e44e4f9f9fafe08838c82a/slide_40.jpg){kind=link}
