Discovering Python - Speaker Deck

Slide 1

Slide 1 text

Discovering Python David Beazley (@dabeaz) http:/ /www.dabeaz.com PyCon'2014 Montreal

Slide 2

Slide 2 text

In 2005... ... I was hired to go look at 1.5 TB (yes, that's Terabytes) of source code sitting in a secret vault.

Slide 3

Slide 3 text

Six Years Later... I testiﬁed in US district court about: - Concurrency - Threads - Event loops - Interrupts Good god!

Slide 4

Slide 4 text

Discovering with Python (or what happens when Python is brought into the ring of a legal battle)

Slide 5

Slide 5 text

Disclaimer Everything in this talk actually happened Names and details have been changed Non-disclosure (I'd have to kill you) All exhibits/photos are ﬁctional I know nothing, you'll learn nothing

Slide 6

Slide 6 text

Meet Alice

Slide 7

Slide 7 text

Alice Meet Bob

Slide 8

Slide 8 text

Alice Bob "No, I'll send YOU a message!"

Slide 9

Slide 9 text

Alice Bob Bob's Attorney

Slide 10

Slide 10 text

Alice Bob Bob's Attorney "Bwhahahaha!" Patent Infringement

Slide 11

Slide 11 text

Alice Bob Bob's Attorney "Prepare to die!" Alice's Attorney

Slide 12

Slide 12 text

Alice Bob Bob's Attorney "Prepare to die!" Alice's Attorney "Bring it!"

Slide 13

Slide 13 text

Let's Talk Patents A hot-button issue Myth: All patent lawsuits are trolls Myth: All patent lawsuits involving software are purely about software Fact: Patent litigation is hell

Slide 14

Slide 14 text

Patent Litigation Timeline You hear about patents a lot But what actually happens? This talk is about that! Initial Complaint Fact Discovery (9-12 months) Claim Construction Summary Judgement Trial

Slide 15

Slide 15 text

Fact Discovery Bob "Obvious Infringement"

Slide 16

Slide 16 text

Fact Discovery Alice "Obviously Different"

Slide 17

Slide 17 text

Fact Discovery Bob's Attorney Alice's Attorney Facts

Slide 18

Slide 18 text

"Just the facts, ma'am" Enter: Fact Expert Technical expert Unbiased party Privileged Works with legal

Slide 19

Slide 19 text

Reality Bob's Attorney Bob's Coworkers Fact Expert

Slide 20

Slide 20 text

The Team Bob's Attorney Bob's Coworkers Fact Expert Me

Slide 21

Slide 21 text

What Happens You are dropped into a ﬁrestorm No technical guidance Because no one knows anything... that's why they called you!

Slide 22

Slide 22 text

Quick Learning The Invention

Slide 23

Slide 23 text

Quick Learning The Invention 7. The system of claim 5 or 6, wherein the display and input means comprises displays means and input means, the input means being connected to the central processing unit, the display means being connected to the slithering means and the central processing unit, the display means being arranged to display the displays and the input means transferring the input responses to the central processing unit, and wherein the display and input task means further comprises display task means and input task means, the display task means being arranged to control the display means by transferring display commands to, and receiving the display responses from, the display means, the input task means being arranged to control the input means by transferring input commands to, and receiving input responses from, the input means. The Patent

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Invention has some code - 600 pages C - PDF - 1989

Slide 26

Slide 26 text

Patent Compilation Does the patent even work? Would the code compile? Can it be explained to others? You'd better ﬁnd out How?

Slide 27

Slide 27 text

Hand Compilation from PDF - Use highlighter

Slide 28

Slide 28 text

Enter Python definitions = { 450: [ 'spam', 'grok', ], 451: [ 'foo', ], 452: [ 'bar', ] } deﬁnitions calls = { 123: [ 'blah', 'read_input', 'send_msg', ], 124: [ 'spam', 'foo', 'bar' ] } calls Entered by hand (from paper copy) A long weekend

Slide 29

Slide 29 text

Just Link It symbols = { name: pageno for pageno, defns in definitions.items() for name in defns } unresolved = [ (name, pageno) for pageno, clist in calls.items() for name in clist if name not in symbols ] missing = defaultdict(list) for name, pageno in unresolved: missing[name].append(pageno) for item in missing.items() print("Missing: %s on pages %s" % item)

Slide 30

Slide 30 text

Secret Weapons List/dict/set comprehensions collections module

Slide 31

Slide 31 text

WHY?!?!?!?! Due diligence You'd better understand your side's invention Otherwise, you will die

Slide 32

Slide 32 text

Meet The Enemy Alice

Slide 33

Slide 33 text

Meet The Enemy Alice Alice's Ninja Rockstar Coders

Slide 34

Slide 34 text

Alice's Ninja Rockstar Coders Meet The Enemy Alice

Slide 35

Slide 35 text

Meet The Enemy Alice Alice's Adult Engineers SEI CMMI Level 4

Slide 36

Slide 36 text

Here Are Some Documents 500 pages

Slide 37

Slide 37 text

500 pages 5,000 pages Here Are Some Documents

Slide 38

Slide 38 text

500 pages 5,000 pages 500,000 pages Here Are Some Documents

Slide 39

Slide 39 text

(what's better than one? 300,000 that's what!) Sample Documents

Slide 40

Slide 40 text

Purported Source Code? ATTORNEY EYES ONLY ATTORNEY EYES ONLY 1677723 1677724

Slide 41

Slide 41 text

From: Guido van Rossum Date: Dec 9 23:21:42 CET 2011 Subject: [Python-Dev][PATCH] Adding braces to __future__ For me, if I had to design a new language today, I would probably use braces, not because they're better than whitespace, but because pretty much every other lanugage uses them, and there are more interesting concepts to distinguish a new language. That said, I don't regret that Python uses indentation, and the rest I have to say about the topic would violate the above request. -- --Guido van Rossum (python.org/~guido) Emails

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Alleged Prior Art

Slide 44

Slide 44 text

Deposition of Crazy Old Guy Prior art

Slide 45

Slide 45 text

We Have Their Software It's highly proprietary You're the only one approved to look at it It's actually sitting over in a vault AKA: Software escrow

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

The Vault

Slide 48

Slide 48 text

The Vault By the tracks

Slide 49

Slide 49 text

The Vault By the tracks Rock band rehearsal space

Slide 50

Slide 50 text

Vault Protocol No computers No phone No electronics No storage devices Pen, paper and books okay

Slide 51

Slide 51 text

The Vault PC in a locked cage (no network) Printer Special paper Log Book

Slide 52

Slide 52 text

What's There? A collection of large hard drives D:\ E:\ F:\ G:\ Each containing copies of CDs (>1.5 TB total) No documentation or organization

Slide 53

Slide 53 text

Perspective Software archive for the infringing invention

Slide 54

Slide 54 text

Perspective Software archive for the infringing invention Embedded Microcontroller System

Slide 55

Slide 55 text

Perspective Software archive for the infringing invention Embedded Microcontroller System Display Module Keypads 7-segment

Slide 56

Slide 56 text

Perspective Software archive for the infringing invention Embedded Microcontroller System Display Module Keypads 7-segment A PC

Slide 57

Slide 57 text

Perspective Software archive for the infringing invention Embedded Microcontroller System Display Module Keypads 7-segment A PC Custom PCI Board

Slide 58

Slide 58 text

Perspective Software archive for the infringing invention Embedded Microcontroller System Display Module Keypads 7-segment A PC Custom PCI Board Second PC

Slide 59

Slide 59 text

Perspective A PC Custom PCI Board Second PC Custom Router Actually, more of a distributed system

Slide 60

Slide 60 text

Perspective The software is "all stack" (a million lines of code) C++/ Win32 C/ASM DCOM/ CORBA C/ASM C/ASM VB Java RMI RTOS

Slide 61

Slide 61 text

Enter Time OS/2 90 92 94 95 96 97 98 00 01 WinNT V1 V2 V3 V4 V5 V6 V7 V8 V9 RevA RevB RevC • Weekly snapshots (52 x 15 years = 780 versions) • Multiple hardware revisions/conﬁgurations • Operating system changes/deployment changes

Slide 62

Slide 62 text

Enter Customers • Dozen major customers (corporations) • Customer-speciﬁc system modiﬁcations • Think "skins" on main system • Hundreds of interlocking versions Base System Version 2.51 ACME Vers 1.23 Buy N Large Vers 4.22 Tyrell Corp Vers 3.43

Slide 63

Slide 63 text

Provided Tools Windows-XP

Slide 64

Slide 64 text

Provided Tools Windows-XP Command Prompt

Slide 65

Slide 65 text

Provided Tools Windows-XP Command Prompt Search Mutt

Slide 66

Slide 66 text

Provided Tools Notepad

Slide 67

Slide 67 text

Ofﬁcial Tools Notepad Visual Studio

Slide 68

Slide 68 text

Printing You can print anything Must be logged Numbered, copied, given to opposing side

Slide 69

Slide 69 text

Constraints No working hardware setup (can't run code) No working build environ (can't compile) No tech support (can't call anyone) Fragmentary documentation (if any)

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

Secret Weapon

Slide 72

Slide 72 text

Python? What? How? Unknown: How did Python get placed on the machine in the vault? I have NO idea A new IBM PC with only "approved tools" Best Guess: Used by an IBM OEM tool (Yet, there it was, python... in the Windows path no less).

Slide 73

Slide 73 text

Desert Island Coding Admit it, you've probably thought Python might be a good choice Batteries included FOR THE WIN!

Slide 74

Slide 74 text

Strategy Create a fact discovery environment from scratch in the vault I was destined for this job... I wrote the book

Slide 75

Slide 75 text

Question: What are the objectives? (What does it mean to "look at" the code?)

Slide 76

Slide 76 text

Goals What was provided? Is it complete? How does the code work? Where is the patent in the code?

Slide 77

Slide 77 text

The Horror! The Horror! Reverse engineering the entire build environ Makefiles, config files, etc. Identifying all major software components Examples: .exe files, .DLLs, plugins, etc. Sorting out version histories

Slide 78

Slide 78 text

MKDEP= mkdep SHELL= /bin/sh # === Fixed definitions === OBJS= \ bltinmodule.o \ ceval.o cgensupport.o compile.o \ errors.o \ frozen.o \ getargs.o getcompiler.o getcopyright.o getm getplatform.o getversion.o graminit.o \ import.o importdl.o \ marshal.o modsupport.o mystrtoul.o \ pythonrun.o \ sigcheck.o structmember.o sysmodule.o \ traceback.o \ $(LIBOBJS) LIB= libPython.a Sources Library You try to ﬁgure it out

Slide 79

Slide 79 text

Tackle the Provided Code

Slide 80

Slide 80 text

Basic Tooling Reimplement Unix find grep wc diff tail head Because that Windows search mutt must die

Slide 81

Slide 81 text

Example: navigation import os def cd(dirname): os.chdir(dirname) def pwd(): print(os.getcwd()) def ls(dirname=''): os.system('dir %s' % dirname)

Slide 82

Slide 82 text

Example: diff # diff.py import sys, difflib def diff(fromfile, tofile): fromlines = open(fromfile).readlines() tolines = open(tofile).readlines() diff = difflib.context_diff(fromlines, tolines, fromfile, tofile) sys.stdout.writelines(diff)

Slide 83

Slide 83 text

Interactive Shell >>> cd('pycode') >>> pwd() D:\Files\pycode >>> diff('Python-2.6/Lib/collections.py', ... 'Python-2.6.2/Lib/collections.py') *** Python-2.6/Lib/collections.py --- Python-2.6.2/Lib/collections.py *************** *** 103,109 **** # where the named tuple is created. Bypass this step in where # sys._getframe is not defined (Jython for example). if hasattr(_sys, '_getframe'): ! result.__module__ = _sys._getframe(1).f_globals['__n return result --- 103,109 ---- # where the named tuple is created. Bypass this step in where

Slide 84

Slide 84 text

More Than Reinvention Actually implementing an entire workﬂow Building up layers of tools/analyses Not unlike what is done with IPython NB Can't understate Python awesomeness

Slide 85

Slide 85 text

Example def allfiles(topdir): return ((path, filename) for path, dirs, files in os.walk(topdir) for filename in files) >>> files = allfiles('AllPython') >>> next(files) ('AllPython/0/python-0.9.1', 'python.man') >>> next(files) ('AllPython/0/python-0.9.1', 'README') >>>

Slide 86

Slide 86 text

Example def filetypes(topdir): from collections import Counter from pprint import pprint c = Counter(os.path.splitext(name)[1] for _, name in allfiles(topdir)) pprint(c.most_common()) >>> filetypes('AllPython') [('.py', 125277), ('.c', 27200), ('', 17010), ('.rst', 15439), ('.h', 14782), ('.tex', 12257), ... allfiles()

Slide 87

Slide 87 text

Example def find(topdir, pattern): from fnmatch import fnmatch return ((path, name) for path, name in allfiles(topdir) if fnmatch(name, pattern)) >>> f = find('AllPython', '*.py') >>> next(f) ('AllPython/0/python-0.9.1/demo/scripts', 'findlinksto.py' >>> next(f) ('AllPython/0/python-0.9.1/demo/scripts', 'mkreal.py') >>> next(f) ('AllPython/0/python-0.9.1/demo/scripts', 'ptags.py') >>> allfiles() filetypes()

Slide 88

Slide 88 text

Example def create_versions(topdir): import re for path, _ in find(topdir, 'pgen.c'): pypath, _ = os.path.split(path) version = re.search(r'-(\w+\.\w+(\.\w+)?)$', pypath).group(1) yield version, pypath allfiles() filetypes() find()

Slide 89

Slide 89 text

Example >>> vers = find_versions('AllPython') >>> next(vers) ('0.9.1', 'AllPython/0/python-0.9.1') >>> next(vers) ('1.0.1', 'AllPython/1/python-1.0.1') >>> allfiles() filetypes() find() find_versions()

Slide 90

Slide 90 text

Example def write_manifest(topdir): import csv f = open('manifest.csv','w') csv.writer(f).writerows(find_versions(topdir)) f.close() allfiles() filetypes() find() find_versions()

Slide 91

Slide 91 text

Example allfiles() filetypes() find() find_versions() write_manifest()

Slide 92

Slide 92 text

Example allfiles() filetypes() find() find_versions() write_manifest() .csv Workﬂows!

Slide 93

Slide 93 text

Pile it Higher and Higher HDDs Snapshots "Virtual File System" View View View You keep building abstractions Reorganized ﬁle layer Different views (version, date, prod, debug, etc.) .csv

Slide 94

Slide 94 text

Example: Versioning def versions(filename): import hashlib from collections import defaultdict manifest = read_manifest() groups = defaultdict(list) for vers, path in manifest.items(): fullname = os.path.join(path, filename) if os.path.exists(fullname): digest = hashlib.new('md5') digest.update(open(fullname,'rb').read()) groups[digest.digest()].append(vers) return sorted([sorted(g) for g in groups.values()])

Slide 95

Slide 95 text

Example: Versioning >>> for x in versions('Python/thread.c'): ... print(x) ... ['1.0.1'] ['1.1'] ['1.2', '1.3'] ['1.4'] ['1.5', '1.5.1'] ['1.5.2', '1.5.2c1'] ['1.5.2b1', '1.5.2b2'] ['1.6', '1.6b1'] ['2.0', '2.0.1', '2.0c1', '2.1', '2.1.1', '2.1.2', '2.1.3'] ...

Slide 96

Slide 96 text

Navigational Tooling "Virtual File System" View View View Query tools for going to any version/ﬁle Navigational Tools >>> view('2.7.3', 'Python/ceval.c') >>> Typically launch windows tools (e.g., Vis Studio)

Slide 97

Slide 97 text

Timeline/Inventory Tools Link together every version of every component found Development timelines Ofﬁcial vs. Debug releases V1 V2 V3 V4 release release release release release release release

Slide 98

Slide 98 text

Commentary I don't know if the opposing side actually expected us to ﬁgure out their code We knew almost everything about everything Python FOR.THE.WIN.

Slide 99

Slide 99 text

How Does Code Work? Better make sure you understand everything about the code Software architecture Interaction between components Underlying algorithms

Slide 100

Slide 100 text

Problem: Code Sucks Nobody wants to read code Better: Design documents, specs Nobody wants to give you that "Go read the source."

Slide 101

Slide 101 text

Let's Go Fishing Interesting ﬁles Code comments TPS reports PDF, DOC, RTF, HTML, TXT / / See: Important Document Fixed bug. See important speciﬁcation. I ὑ re

Slide 102

Slide 102 text

Back and Forth An obscure ﬁnd /* See FS-6541-8v2.0 for details */ A request to attorneys "Tell opposing counsel we can't ﬁnd FS-6541-8v2.0" A few silent days pass....

Slide 103

Slide 103 text

No content

Slide 104

Slide 104 text

No content

Slide 105

Slide 105 text

No content

Slide 106

Slide 106 text

Casting a Wide Net Search for documents is far and wide Software change notices Unrelated software (peripheral devices) Emails The web (catalogs, manuals, job postings, etc.) Analogy: Pulling on a loose thread...

Slide 107

Slide 107 text

Commentary You're learning the invention from scratch Reading other people's code You're teaching attorneys about it The other side doesn't want you to succeed You will learn A LOT in this exercise

Slide 108

Slide 108 text

Some Lessons Learned SUCKS ROCKS

Slide 109

Slide 109 text

Some Lessons Learned SUCKS ROCKS C++ Assembly code

Slide 110

Slide 110 text

Some Lessons Learned SUCKS ROCKS C++ Assembly code Asynchronous Threads

Slide 111

Slide 111 text

Some Lessons Learned SUCKS ROCKS C++ Assembly code Asynchronous Threads Objects Functions

Slide 112

Slide 112 text

Some Lessons Learned SUCKS ROCKS C++ Assembly code Asynchronous Threads Objects Functions Makeﬁles IDEs

Slide 113

Slide 113 text

Some Lessons Learned SUCKS ROCKS C++ Assembly code Asynchronous Threads Objects Functions Makeﬁles IDEs CASE Tools Humans

Slide 114

Slide 114 text

Some Lessons Learned SUCKS ROCKS C++ Assembly code Asynchronous Threads Objects Functions Makeﬁles IDEs CASE Tools Humans UML Words

Slide 115

Slide 115 text

Some Lessons Learned SUCKS ROCKS C++ Assembly code Asynchronous Threads Objects Functions Makeﬁles IDEs CASE Tools Humans 1990s 1970s UML Words

Slide 116

Slide 116 text

Some Lessons Learned SUCKS ROCKS C++ Assembly code Asynchronous Threads Objects Functions Makeﬁles IDEs CASE Tools Humans 1990s 1970s (of course this is just my opinion, I could be wrong) UML Words

Slide 117

Slide 117 text

Speaking of Attorneys Do the "facts" support patent infringement? Does it look like it infringes? Can it be proven that it infringes? (Let the game begin)

Slide 118

Slide 118 text

No content

Slide 119

Slide 119 text

Remember this? 7. The system of claim 5 or 6, wherein the display and input means comprises displays means and input means, the input means being connected to the central processing unit, the display means being connected to the slithering means and the central processing unit, the display means being arranged to display the displays and the input means transferring the input responses to the central processing unit, and wherein the display and input task means further comprises display task means and input task means, the display task means being arranged to control the display means by transferring display commands to, and receiving the display responses from, the display means, the input task means being arranged to control the input means by transferring input commands to, and receiving input responses from, the input means. The Patent

Slide 120

Slide 120 text

Slide 121

Slide 121 text

No content

Slide 122

Slide 122 text

Deﬁning Claim Terms 7. The system of claim 5 or 6, wherein the display and input means comprises displays means and input means, the input means being connected to the central processing unit, the display means being connected to the slithering means and the central processing unit, the display means being arranged to display the displays and the input means transferring the input responses to the central processing unit, and wherein the display and input task means further comprises display task means and input task means, the display task means being arranged to control the display means by transferring display commands to, and receiving the display responses from, the display means, the input task means being arranged to control the input means by transferring input commands to, and receiving input responses from, the input means. The Patent Term Plaintiff Defendant central processing unit display means

Slide 123

Slide 123 text

Claim Construction Claim terms have to be supported by reality If not, it's game over A lot of attorney/expert consultation Problem: very speciﬁc facts and structure File: Widget/foo.c, lines 230-255. Requires a deep dive

Slide 124

Slide 124 text

Problem Matching claims to 800 versions of a million line program Pick one version? Which one? Match them all?

Slide 125

Slide 125 text

Fragment Versioning You're familiar with source code control Imagine applying it to code fragments/excerpts In reverse Hmmm.

Slide 126

Slide 126 text

/* source.c */ void grok() { if (spam) { foo(); bar(); } ... } void blah() { ... }

Slide 127

Slide 127 text

/* source.c */ void grok() { if (spam) { foo(); bar(); } ... } void blah() { ... } file: source.c start: 'void grok()' end: 'void blah()' Fragment

Slide 128

Slide 128 text

/* source.c */ void grok() { if (spam) { foo(); bar(); } ... } void blah() { ... } file: source.c start: 'void grok()' end: 'void blah()' Fragment Snapshots (>800) Global fragment search across all versions

Slide 129

Slide 129 text

/* source.c */ void grok() { if (spam) { foo(); bar(); } ... } void blah() { ... } file: source.c start: 'void grok()' end: 'void blah()' Fragment Snapshots (>800) Ver1 Ver2 Ver3 void grok() { if (spam) { foo(); bar(); } } void blah() { void grok() { if (spam) { foo(); bar(x); } } void blah() { void grok() { if (spam) { new_foo(); bar(x); } } void blah() {

Slide 130

Slide 130 text

Big Picture Reduce a massive data set to something sane "This claim matches this structure in the code. There have only been six versions of this code over 15 years. Here are the six versions." Keep in mind: All this happening in the vault Big collection of fragment histories

Slide 131

Slide 131 text

PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON

Slide 132

Slide 132 text

PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON Python makes the impossible possible

Slide 133

Slide 133 text

PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON PYTHON Python makes the impossible possible (even Python 3)

Slide 134

Slide 134 text

Final Thoughts If you get the chance to do this, do it! You will learn A LOT! Would I want to do it again? Not sure.

Slide 135

Slide 135 text

But How Did it End?

Slide 136

Slide 136 text

No content

Slide 137

Slide 137 text

My End Game I learned a lot about generator functions Ultimately a well-known PyCon tutorial...

Slide 138

Slide 138 text

Postscript: Expert Report You may be asked to write an expert report Outlines all factual ﬁndings Ties facts to patent claims A scientiﬁc document It's a document that WILL be read

Slide 139

Slide 139 text

Postscript: Deposition You A room of attorneys Opposing expert Court reporter Videographer 8 hours It will be one of the most intense, surreal, awesome/worst experiences of your whole life.

Slide 140

Slide 140 text

Postscript: Court Testimony Like deposition, but dialed up to 11 Twice as many attorneys, more experts Judge & clerks

Slide 141

Slide 141 text

Questions