Ruby: A Family History (Railsconf 2018)

A talk I gave at Railsconf 2018. Experimenting with including some lightly edited speaker notes to accompany the slides.

Rails is made possible by Ruby’s unique combination of deep dynamism and pragmatic elegance. In turn, Ruby inherited many of its core ideas from older languages including Lisp, Smalltalk, and Perl.

In this talk, we’ll take a tour of these languages and their myriad influences on Ruby, to better understand the foundations of our tools, improve our ability to use them to their full potential, and imagine how they might be further improved.

Transcript

1. I have a confession to make: I love Ruby. Something just clicks, it feels expressive, powerful, flexible and I suspect many of you feel the same way Ruby: A Family History Geoffrey Litt @geoffreylitt Panorama Education
2. * But recently I've been thinking about: why? This isn't just a matter of intellectual curiosity. * When we see successful systems like Ruby/Rails, and we want to make successful systems ourselves, it can be useful to deeply understand WHY other systems are successful. To some extent, this is a personal question, but there are still ways to systematically analyze this question. One technique is to compare Ruby with other programming languages. Why do I Ruby?
3. One approach is to compare ruby with modern programming languages: Python, or Rust, or Java. To me, this is like comparing a person to their friends and peers, which is OK... But if you really want to understand someone... By learning other languages, we can broaden our horizons and also more deeply understand Ruby. — matz
4. You gotta meet their parents. Today we're going to meet Ruby's parents! !
5. Today we're going to explore 3 prominent languages that contributed heavily to Ruby: Lisp, Smalltalk, Perl. We'll approach these languages through 3 lenses: 1) Understand the core ideas of these languages, to deepen our understanding of key principles in Ruby. 2) Rediscover lost and forgotten ideas from these languages 3) Most importantly: Not just explore these languages in isolation, but see how Matz combined them into a single coherent language that we love. We'll see a lot of subtle balance in his design decisions, that not only can help us answer "why do we love Ruby?", but can also teach us valuable lessons designing any system, whether a language, framework, or product.
6. * Let's start with LISP. LISP
7. According to Matz, Ruby actually originated as a Lisp! Ruby was a Lisp originally, in theory. Let's call it MatzLisp from now on. ;-) — matz
8. * Created in 1958-1960 by computer scientist John McCarthy. * His goal was NOT to create a programming language. He wanted to create a set of fundamental axioms for computing, a model of describing algorithms * Out of that work came Lisp: List Processing language. * Had a data structure of lists, and a small number of simple operators on lists: chopping them up, joining back together. Plus a keyword lambda for defining functions.
9. * Out of that small set of axioms, we can define arbitrary computation!!! * One example: given those primitives, we can define eval, a Lisp interpreter. Only about 30 lines! * to really appreciate this, let's place it in context An interpreter (defun eval (e a) (cond ((atom e) (assoc. e a)) ((atom (car e)) (cond ((eq (car e) 'quote) (cadr e)) ((eq (car e) 'atom) (atom (eval (cadr e) a))) ((eq (car e) 'eq) (eq (eval (cadr e) a) (eval (caddr e) a))) ((eq (car e) 'car) (car (eval (cadr e) a))) ((eq (car e) 'cdr) (cdr (eval (cadr e) a))) ((eq (car e) 'cons) (cons (eval (cadr e) a) (eval (caddr e) a))) ((eq (car e) 'cond) (evcon. (cdr e) a)) ;...
10. We can think of programming languages on a spectrum from low-level to high-level.
11. * Previously we were adding layers of abstraction on top of machine code.
12. * Lisp wasn't just another layer. It came from a totally different direction! * McCarthy was creating abstract description of computation, and how computers work is decoupled from that project. * When people say "Lisp was discovered, not invented", this is what they mean. * We still see this lineage in Ruby and Rails: human-centered tools that prioritize the way we think. * Optimizing for programmer productivity, not machine productivity. * To put in terms of David's keynote yesterday: these tools do conceptual compression Because Lisp was the first high-level language, Ruby got many many things from it: conditionals, dynamic typing, garbage collection, symbol types But one area that's particularly instructive: how Ruby incorporated Lisp's handling of functions
13. Lisp introduced idea of functions as values, created and manipulated at runtime. ^ Here's how you define an anonymous function in Lisp, using the lambda keyword. Functions (lambda (x) (* x 2))
14. * Lisp also introduced higher-order functions like map that take a function as an argument. * As Ruby developers, you probably understand how powerful things like map are. * In Ruby, these ideas of function values and higher-order functions are directly inherited from Lisp. Functions (map (lambda (x) (* x 2)) (1 2 3))
15. * We can translate this Lisp code directly into Ruby using Ruby's lambda keyword * But that's not how we normally do things in ruby, right? We have blocks. * We use blocks every day, but have you ever asked, why do they exist? And what's the point? Functions (map (lambda (x) (* x 2)) (1 2 3)) [1, 2, 3].map( &lambda { |x| x * 2 } )
16. We say "everything is an object" in ruby, but a block isn't a valid object Functions { |x| x * 2 } # => SyntaxError
17. You can only pass a block as an implicit argument. ^ This adds quite a lot of complexity. In Lisp, only need to know about lambdas. In Ruby, procs, lambdas, blocks... ^ What do we gain from this added complexity? What this gives us: more concise syntax for special cases where function takes just one anonymous function as an argument. Turns out this is useful in many different situations! Functions { |x| x * 2 } # => SyntaxError def takes_a_block yield 2 end takes_a_block { |x| x * 2 } # => 4
18. Lisp had the idea of pipelines of functional transformations... ^ but Ruby arguably offers a cleaner way to write them than Lisp itself, with blocks. Functions (remove-if (lambda (n) (< n 4)) (map (lambda (x) (* x 2)) (1 2 3))) [1, 2, 3]. map { |x| x * 2 }. reject{ |n| n < 4 }
19. Here I have some fake Rspec syntax in lisp. Internal DSLs are more reasonable to write in Ruby. We see this in Rails migrations, rake tasks, and more Now what if you want to pass in two anonymous functions? Can't do it with blocks, you have to resort to a totally different system. Blocks arguably add inconsistency and complexity to the language. Functions (describe "my machine" (lambda () ( (it "produces widgets" (lambda () ( ;... )))))) describe "my machine" do it "produces widgets" do #... end end
20. * Often people say "consistency is really important in design" * But, in this case: inconsistency has a payoff: helps make our code easier to read/write in a very common "special case" * You see this in Rails too! Idea of convention over configuration: introduce implementation complexity to make the common case easy * Different users will want different tradeoffs here, but the point is that consistency can be sacrificed in service of other goals. By the way, fun to see concise lambdas catching on... In last 10 years: Stabby lambda syntax in Ruby, fat arrow syntax in ES6. Language designers realizing this concise lambda definition is a valuable feature. Thoughtful inconsistency
21. * LISP's syntax is called "s-expressions" Fun tidbit of computing history: Originally an accident, s-expressions were intended as the internal representation. McCarthy had a plan for a whole other syntax called m-expressions, with fewer parens. But a researcher in his lab wrote an interpreter for s- expressions, that got popular, and the rest was history. McCarthy has a great quote reflecting on the m- expressions project: Syntax ( (let a 1) (let b 2) (+ a b))
22. The project...was neither finalized nor explicitly abandoned. It just receded into the indefinite future... — John McCarthy
23. * Lisp syntax mirrors the abstract syntax tree and has tons of beautiful properties: * very little parsing * macros for manipulating code as data * and Lisp hackers will say you can totally get used to the parentheses LISP Syntax ( (let a 1) (let b 2) (+ a b))
24. * This is syntax of a different language, FORTRAN * Matz moved from s-expression syntax to Fortran style. Kinda looks like Ruby. * Requires much more complex parsing, infix operators, etc...so if Ruby is a Lisp, why use Fortran syntax? * I can't read Matz's mind. Maybe he preferred this. But either way, it was the right thing to do. * This syntax had already won and was familiar to everybody. It was a necessary choice to drive adoption. FORTRAN Syntax a = 1 b = 2 result = a + b
25. * If you want widespread usage, sometimes you have to go along with the popular trend to remain relevant. * Ruby went with the prevailing trend in syntax. * Rails has a similar approach to the modern Javascript ecosystem with webpacker, going along with trends to remain relevant * Fun question: Lisp lets you manipulate your code as data. What if we want to do that in ruby? Pick your battles.
26. Using the parser gem, we can take a string of code and parse it... Ruby parsing/unparsing code = "2 + 3 * 4" ast = Parser::CurrentRuby.parse(code)
27. and we get access to that code as a tree of data! Ruby parsing/unparsing code = "2 + 3 * 4" ast = Parser::CurrentRuby.parse(code) # => [s(:send, # s(:int, 2), :+, # s(:send, # s(:int, 3), :*, # s(:int, 4))), []]
28. and we can then unparse it back into text. ^ You might be thinking, this is a terrible idea, and you're mostly right :) ^ But some interesting real world use cases of this. Gem called "mutant" that does mutation testing, where it randomly changes your code and makes sure your tests fail when your code changes. Uses this gem under the hood. But of course we don't need to resort to this often. ^ Reason: most of the time in Lisp, code manipulation is used for metaprogramming: code that creates code. In Ruby we don't need to use this approach because we have a variety of powerful tools, which come from another language: Ruby parsing/unparsing code = "2 + 3 * 4" ast = Parser::CurrentRuby.parse(code) # => [s(:send, # s(:int, 2), :+, # s(:send, # s(:int, 3), :*, # s(:int, 4))), []] 2.1.5 :005 > Unparser.unparse(ast) # => "2 + (3 * 4)"
29. * Smalltalk! * The first OO language * Huge influence on Ruby Smalltalk
30. * Early 70s, led by Alan Kay, lots of work from Adele Goldberg and Dan Ingalls at legendary Xerox PARC lab
31. Fundamental goal: expand computing to everyone, let anyone make their own programs. Architects making architecture software, artists making drawing software... Smalltalk also had a focus on kids This resonates with DHH's keynote from yesterday. Widening the group that has access. This is part of why it's named Smalltalk: focus on kids Also a joke: Kay was tired of systems with big names like Zeus/Thor doing nothing, so he made a system with a small, cute name but lots of power
32. Not just the first OO language, also an editing environment for that language. They invented the modern GUI and overlapping windows Model-View-Controller was also invented there for Smalltalk Basically everything we do today invented at this one lab! By the way, other labs down the hall were inventing things like Ethernet... Great book called Dealers of Lightning that recounts the history
33. We talk a lot about OO, and sometimes this question comes up, "what is OO?" What is "object-oriented programming"?
34. Maybe there's not a single answer to that question, but we CAN ask, what was the intention of the creator of OO when it was invented? This is a question that can yield some insight. To answer it, we look to Smalltalk, the first OO language Similarly to McCarthy, Alan Kay was looking for a consistent model of computation. What is "object-oriented programming"? What was the original intention of object- oriented programming?
35. * He was a fan of Lisp... but had problems with it * Amazingly, thought lisp wasn't consistent enough! * example: "lambda" isn't a function. Too much of lisp isn't implemented with functions. * He was looking for a more purely recursive approach to designing a model of computation. I could hardly believe how beautiful and wonderful the idea of LISP was...but there were deep flaws in its logical foundations. — Alan Kay
37. * People had been splitting up a computer into functions and data separately.
38. * What if we split into little computers instead?
39. * Each has state and process encapsulated. Like millions of little computers In computer terms, Smalltalk is a recursion on the notion of computer itself. — Alan Kay
40. Let's see what that looks like in practice. Message passing 3 + 4 "=> 7
43. * almost like making a remote API call, a post request to 3.com * Revolutionary idea: give RECEIVER so much control, extremely DYNAMIC behavior * Very very different from how functions work in C Much of Ruby's OO side comes very directly from Smalltalk:
44. Message passing similarities Smalltalk: 3 perform: '+' asSymbol with: 4 Ruby: 3.send(:+, 4)
45. Message passing similarities Smalltalk: 3 isKindOf: Integer Ruby: 3.is_a? Integer
46. classes are just objects Message passing similarities Smalltalk: Integer isKindOf: Class Ruby: Integer.is_a? Class
47. * have you ever thought about WHY ruby pushes us to do iteration over arrays this way? * having more things be message-based makes the language more CONSISTENT Control flow with message sending Smalltalk: array do: [ :element | Transcript show: element ] Ruby: array.each { |element| puts element }
48. In this example: changing what happens when an integer receives a method it doesn't understand. The Smalltalk and Ruby look nearly identical. Smalltalk has precedents for 2 key ideas in Ruby: * Opening classes, including built-ins * Handling missing methods * This all flows from the core mental model: extreme encapsulation. Rails uses these metaprogramming tools extensively ^ Opening up classes, dynamically defining/undefining methods, handling missing methods ^ Fair to debate how to use these powerful tools wisely, but at least we see their origins Metaprogramming Integer extend [ doesNotUnderstand: msg [ 'method not defined' printNl ] ] class Integer def method_missing(msg) puts 'method not defined' end end
49. * Obviously we got a lot * But how did Ruby DIVERGE from Smalltalk? There are no conditionals in Smalltalk. You saw Kay was committed to the purity of the recursion. Conditional logic in Smalltalk: Just a message send Message passing purity (2 + 2 == 5) ifTrue: [ Transcript show: 'true'. ] ifFalse: [ Transcript show: 'false'. ].
50. You can implement similar things in Ruby, but instead we got conditionals as a language feature. not sure why Matz chose this, but one theory: if statements just seem more straightforward to use. And he was willing to add complexity/ inconsistency to the language to accommodate that Thoughtful inconsistency
51. * Smalltalk had a different approach to code organization * This is a forgotten idea: a super highly integrated IDE * Smalltalk: no text files, always operating on a live image
52. * In this example, you can provide example input output and it'll find the corresponding method for you.
53. * Ruby didn't do image-based editing, too radical. Almost certainly a good idea for adoption. * Ruby/Rails is a text editor + terminal community, but we have rich editor heritage. Bring it back? Pick your battles.
54. * Perl: the "Swiss army chainsaw" of programming languages Perl
55. * You can tell this one is going to be a little different * Developed in late 80s by Larry Wall * linguist working as sysadmin for NASA * Larry Wall wasn't solving a deep theoretical problem. * He was trying to do sysadmin stuff. * Wanted to combine text processing (sed, awk) w/ Real Programming (C)
56. Perl provided many mantras that Ruby adopted A focus on pragmatism, productivity, just getting the job done somehow, getting out of your way. "A program is correct if it gets the job done before you get fired."
57. * One of Ruby's core principles! * in contrast w/ Python, in contrast with more academic communities. * Larry Wall thinks of programming as human language more than a mathematical system. "There's more than one way to do it"
58. * This brings us back to introducing thoughtful inconsistency, making special cases when it helps. "Easy things should be easy and hard things should be possible"
59. A good car has a solid chassis + transmission + airbags... but it also has to have a nice interior. The seats and interior details matter. Academic languages often focus on getting the chassis + transmission right, but often don't pay as much attention to this stuff Ruby is a polished language, feels nice to use. Like a car with a nice interior. And we got a lot of those things from Perl.
60. String interpolation is super super easy in Perl. ^ Ruby we don't have it quite this easy, but still have a good syntax for it ^ Javascript had nothing good until very recently, many languages have nothing ^ Wouldn't be surprised if we have Perl to thank Effortless string interpolation $x = 5; $msg = "The value is $x now.";
61. A lot of what makes Ruby feel delightful is this detailed polish work. ^ Rails has this too: 2.days.ago is the same type of thing More little details —native regex syntax —%w array syntax —heredocs —1_000_000 number syntax
62. Also got some things that might be more controversial. global variables: not common in many langs now, but came from Perl. Bad idea most of the time, but sometimes pragmatic and useful for hacking stuff together Globals $global_variable = 1
63. not only that, we got weird built-in global variables Globals $$
64. Cryptic and probably bad for production code... ^ but maybe useful if you're just hacking together a script ^ Very pragmatic! Globals $$ # => 19936 Process ID!
65. Some parts of Perl are too far in this direction so Ruby doesn't incorporate ^ In Perl, guess what this produces? Weak typing print "8" + "1";
66. Yuck. Contrast with Ruby, which has a more strongly typed core. Weak typing print "8" + "1"; # => 9
67. Weak typing @cities = qw( Berlin Tokyo London Boston ); # Assign the array to a scalar $cities_count = @cities;
68. * Perl tilts heavily towards implicit type coercion * Reminiscent of some of most frustrating parts of Javascript * Ruby has a more consistent core. * We got the elegance and ergonomics of Perl, without the core Weak typing @cities = qw( Berlin Tokyo London Boston ); # Assign the array to a scalar $cities_count = @cities; # => 4
69. Alright, brings us back to this question: why do we love Ruby? Why do I Ruby?
70. Ruby Achieves a delicate balance on many dimensions, by picking best parts of different solutions Consistency of Smalltalk/Lisp computation models. But willing to add exceptions to the rule for usability. ^ Academic flavor of Smalltalk/Lisp, layers on nice leather detailing from Perl ^ Takes some stances, like being very object-oriented. But, shies away from radical departures like Lisp's syntax or Smalltalk's editing paradigm. This is hard to pin down because Ruby isn't the most anything. Not most OO or most functional. Not the fastest, or the highest level. When we're designing systems, there's often pressure to stand out on some dimension. You see the cell networks competing for "most coverage" or "fastest network" Even Apple, a company that understands design, plays the game of making the "thinnest laptop ever" But sometimes, a beautiful system doesn't need to be the most anything. It can just be a perfect, well-balanced combination of many different ideas. Maybe next time you're designing a system, you can look to Ruby for inspiration for designing a well-balanced system.
71. Thanks! Twitter: @geoffreylitt