2016 - Alex Martelli - The Tower of Abstraction

​Description
Abstraction is a powerful servant, but a dangerous master. We code, design, think, debug ... on a tower of abstractions. Spolsky's Law tells us that "All abstractions leak". This talk explores why they leak, why that's often a problem, what to do about it; I also cover why sometimes abstractions SHOULD "leak", and how best to produce and consume abstraction layers.

Bio
Author of "Python in a Nutshell", co-author of "Python Cookbook", PSF Fellow, frequent speaker at Python conferences, prolific contributor to StackOverflow, and winner of the 2006 Frank Willison Memorial Award for contributions to Python, Alex currently leads "1:many tech support" for Google Cloud Platform. He's married to Anna Ravenscroft, his co-author in the "Cookbook" 2nd edition and "Nutshell" 3rd edition, also a PSF Fellow, and also a winner of the Frank Willison Memorial Award, in 2013.

https://youtu.be/zhpWhkW8kcc

Transcript

1. ©2016 Google -- aleax@google.com Zen and the Art of Abstraction Maintenance http:/ /www.aleax.it/pybay16_abst.pdf 1
2. This talk's "level" 2 Shu Ha Ri ("Learn") ("Detach") ("Transcend") 2
3. A Tower of Abstractions 3 3
4. Abstraction as Leverage 4 ...lets me do SO much more... ...but can come crashing down if things go wrong! 4
5. Can't Live Without it... 5 programming (& other "knowledge work") USES abstraction layers, often PRODUCES new layers 5
6. ...can live with it? 6 all abstractions "LEAK" ("Spolsky's Law") ...bugs, overloads, attacks... ...you MUST "get" a few layers below! +, they SHOULD "leak" (most of the time;-) - in designed, architected ways! and: abstraction *can slow you down*! 6
7. Abstract -> Procrastinate! McCrea, S. M., Liberman, N., Trope, Y., & Sherman, S. J. -- Construal level and procrastination. Psychological Science, Volume 19, Number 12, December 2008, pp. 1308-1314(7) remote events are mentally construed at higher abstraction levels than "near" ones reverse holds: higher-abstraction construal levels lead to > chance of procrastination (at least for psych students, typically the only experimental subjects available;-) 7 7
8. (is procrastination `BAD`...?) quoth the Zen of Python...: Now is better than never. Although never is often better than *right* now. Context: 1st part refers to `good enough is good enough`: release _now_, listen, iterate 2nd part refers to situations in which backwards compatibility must be kept `forever`: adding a badly, hastily designed feature will weigh you down for years 8 8
9. To Achieve, Think Concrete! Allen, "Getting Things Done": what's my SINGLE NEXT ACTION? interaction (& user-centered) design: NOT "the user", BUT "John, newbie trader, vast videogame experience" and "Mark, seasoned trader, started in Hammurabi's time, STILL prefers cuneiform on clay tablets" "prefer action to abstr-action" (J. Fried, founder of "37 signals") 9 9
10. Abstraction Penalty when a language allows low- and high- abstraction approaches, there can be a penalty for abstraction (Stepanov, http:/ / std.dkuug.dk/JTC1/SC22/WG21/docs/ PDTR18015.pdf & much later research) an issue of quality of implementation, not always true: in Python we're used to an abstraction *bonus*, not *penalty*! 10 10
11. Itertools FLIES! 11 $ python -mtimeit 'for x in range(42): pass' 100000 loops, best of 3: 1.11 usec per loop $ python -mtimeit 'for x in xrange(42): pass' 100000 loops, best of 3: 0.75 usec per loop $ python -mtimeit -s'import itertools' \ > 'for x in itertools.repeat(None, 42): pass' 100000 loops, best of 3: 0.70 usec per loop 11
12. The "Martian Smilie" rocks! $ python -mtimeit -s'x="abracadabra"' \ > 'y="".join(reversed(x))' 100000 loops, best of 3: 1.13 usec per loop $ python -mtimeit -s'x="abracadabra"' \ > 'y=x[::-1]' 1000000 loops, best of 3: 0.186 usec per loop 12 12
13. all abstractions leak, because... ...*all abstractions LIE*! before you can abstract, you must grok the details before you can step back, you must come close abstract only when you know ALL details →since you can't, be humble & flexible! All Abstractions Leak 13 the map is not the territory 13
14. A great abstraction: TCP/IP 14 14
15. TCP leaks: *TRUST*! 15 TCP/IP, superb stack of abstractions, BUT... ...designed in an ancient era of trust! The whole stack "leaks" all over the place in terms of security attacks from: "below" (ARP cache poisoning), "above" (DNS cache poisoning), "beside" (mendacious BGP), "within" (sniffing, pwd FTP/Telnet, ...) ...etc, etc... 15
16. TCP/IP today...:-( 16 16
17. One "leak": ARP poisoning 17 17
18. Other "leak": DNS poisoning 18 18
19. The worst: BGP Hijacking 19 19
20. ..."leaks" may be *good*! 20 e.g.: remote/distributed filesystems trying to "emulate" local ones "less local" → the costlier "abstraction" semantics, locking, reliability, ... "filesystem", splendid abstraction... "local filesystem", NOT! "don't subclass concrete classes" [Haahr] doesn't mean "abstraction is a bad thing" but, JUST the abstraction isn't enough needs systematic, usable LEAKS! 20
21. How to Abstract Wrong small scale: 1 class → 1 interface always "surfaces" implementation details mid-scale: "subclassing concrete classes" concrete class (== implementation) → NEVER the right base for subclassing mid-scale: encapsulation errors windows vs toolbars in MFC 4.* large scale: "floating framework" "framework" with just 1 application... 21 21
22. How to Abstract Well 22 master at least 1-2 layers BELOW to DESIGN an excellent abstraction: DEEP familiarity with SEVERAL possible implementations ("layers below") DEEP familiarity with SEVERAL intended uses ("layers above" which will USE it) no blinders, no shortcuts! YOU can be the next user or implementer! Golden Rule as a core principle of SW!-) http:/ /c2.com/cgi/wiki?TooMuchAbstraction 22
23. Donald Knuth: yes, you can! the psychological profiling [[of a good software developer]] is mostly the ability to shift levels of abstraction, from low level to high level. To see something in the small and to see something in the large. [[...]] [[great software developers]] see things simultaneously at the low level and the high level [[of abstraction]] 23 http:/ /www.ddj.com/184409858 23
24. Jason Fried: and you must! "Here's the problem with copying: Copying skips understanding. Understanding is how you grow. You have to understand why something works or why something is how it is. When you copy it, you miss that. You just repurpose the last layer instead of understanding the layers underneath." 24 http:/ /www.37signals.com/svn/posts/ 1561-why-you-shouldnt-copy-us-or-anyone-else Just '%s/copy/use existing high-level abstractions blindly/g' ...;-) 24
25. Quoting Jeff Attwood “don’t reinvent the wheel, unless you plan on learning more about wheels!” 25 http:/ /www.codinghorror.com/ blog/archives/001145.html 25
26. App Engine "Hacks" 26 RPC 26
27. Monkey-patch Hacking all operations go through an RPC layer, apiproxy_stub_map.MakeSyncCall not advisable: *monkey-patching*...: 27 from google.appengine.api import \ apiproxy_stub_map _org = apiproxy_stub_map.MakeSyncCall def fake(svc, cal, req, rsp): ... x = _org(svc, cal, req, rsp) ... apiproxy_stub_map.MakeSyncCall = fake 27
28. class Client(object): """Memcache client object... """ def __init__(self, servers=None, debug=0, pickleProtocol=pickle.HIGHEST_PROTOCOL, pickler=pickle.Pickler, unpickler=pickle.Unpickler, pload=None, pid=None, make_sync_call=apiproxy_stub_map.MakeSyncCall): """Create a new Client object.... """ ... self._make_sync_call = make_sync_call Why the Monkey is Sad 28 28
29. Better: use "Hooks"! http://blog.appenginefan.com/2009/01/ hacking-google-app-engine-part-1.html (with THANKS to Jens Scheffler!-) from google.appengine.api import apiproxy_stub_map def prehook(svc, cal, req, rsp): ... apiproxy_stub_map.apiproxy.GetPreCallHooks( ).Append('unique_name', prehook, 'opt_api_id') 29 29
30. How to Supply "Hooks"? ...without a "natural funnel" such as RPC? use key semantical "bottlenecks" if your system does SQL queries, pre-hooks w/SQL, post-hooks w/results "event/callback" approaches (Qt signal/slot) design patterns: pre/post hooks & events ~ Observer Template Method (e.g., Queue.Queue) Dependency Injection 30 30
31. Making Hooks: scheduler class ss(object): def __init__(self): self.i = itertools.count() self.q = somemodule.PriorityQueue() def add_event(self, when, c, *a, **k): self.q.push( (when, next(self.i), c, a, k)) def run(self): while self.q: when, n, c, a, k = self.q.pop() time.sleep(when - time.time()) c(*a, **k) 31 31
32. (PQ is "obvious"...): class PriorityQueue(object): def __init__(self): self.l = [] def __len__(self): return len(self.l) def push(self, obj): heapq.heappush(self.l, obj) def pop(self): return heapq.heappop(self.l) 32 32
33. Nice abstraction, but... ...how to test ss without long waits? ...how to integrate it with event-loops of other systems, simulations, etc...? Problem: ss "concretely depends" on specific objects (time.sleep and time.time). To "make the abstraction leak", you can...: 1.hack it with "Monkey Patching", OR… 2.design pattern: Dependency Injection 33 33
34. Monkey-patching... import ss class faker(object): pass fake = faker() ss.time = fake fake.sleep = ... fake.time = ... 34 useful in emergencies, but... ...too often an excuse for lazy design!-) subtle, hidden "communication" via dark byways (explicit is better than implicit!-) broken by optimizations &c... 34
35. Dependency Injection class ss(object): def __init__(self, tm=time.time, sl=time.sleep): self.tm = tm self.sl = sl ... self.sl(when - self.tm()) 35 i.e., just like sched in the standard library!-) 35
36. DI is a handy hook! class faketime(object): def __init__(self, t=0.0): self.t = t def time(self): return self.t def sleep(self, t): self.t += t f = faketime() s = ss(f.time, f.sleep) ... 36 36
37. DI example (app engine:-) class Client(object): """Memcache client object... """ def __init__(self, servers=None, debug=0, pickleProtocol=pickle.HIGHEST_PROTOCOL, pickler=pickle.Pickler, unpickler=pickle.Unpickler, pload=None, pid=None, make_sync_call=apiproxy_stub_map.MakeSyncCall): """Create a new Client object.... """ ... self._make_sync_call = make_sync_call 37 37
38. Q & A 38 ? ! www.aleax.it/pybay16_abst.pdf http:/ /shop.oreilly.com/product/0636920012610.do ebooks, all formats, DRM-free; at checkout: AUTHD 38