Code Walkers for Lisp ʙ or How to implement an inside DSL in Common Lisp ʙ Kansai Lisp Users’ Meeting 2019-09-28 @nfunato 1 

Summary Abstract code walkerͱ͍͏ϢʔςΟϦςΟ଒ʹ͍ͭͯɺ΍͘͞͠঺հ Lispͷcode walkerͬͯɺ͍ͭ/ͳͥཁΔͷ? Ͳ͏໾ཱͭͷ? యܕతͳ༻్: ͪΐ͍ڽΓϚΫϩ, ղੳπʔϧ, ಺෦DSL౳ ࣮ྫͱͯ͠ɺࠓͦ͜ʹ͋Δ Common Lisp code walker Ͱ͋Δ
 PCL code walker (ported to SBCL) ͷ࢖͍ํΛ֓؍͢Δ code walkerͬͯɺ࢖ͬͯΔͱ͜Ζ͋Μ·Γݟͳ͘Ͷ? 
 ͱ͍͏ٙ࿭ʹ͍ͭͯগ͠ߟ͑ͯΈΔ 2 

Who’re you? Why Lisp?
 ʙ it started from PCL ʙ ֶੜ࣌୅͸ॴଐݚڀࣨͰίϯύΠϥपลΛֶͿ ͳ͓ɺ35೥Ҏ্લʹ͖ͭ࣌ޮͰ͢ :-) ৬ྺ͸֓Ͷ୺຤ػثͷ௨৴ܥ(IP૚ΑΓԼ͕ଟ͍) ࢓ࣄ͸௿ϨΠϠਓͩͬͨͷʹ Կނ͔Lispʹ͸ັͤΒΕΔ
 CLʹັ͔ΕͨΩοΧέ: ’87ࠒ comp.sourcesʹྲྀΕͯͨ PCL
 ɹ̋ Portable Common Loops ✕ Practical Common Lisp FORTH, Haskell(Gofer), Smalltalk, OCaml, Erlang౳΋טͬͨ ஶऀԕӨ 3 

BTW, Portable Common Loops is… ͘͢͝େࡶ೺ʹӠ͏ͱ CLOSͷϓϩτλΠϓ
 ɾen.wikipedia.org/wiki/CommonLoops (ྺ࢙)
 ɾwww.cs.cmu.edu/Groups/AI/lang/lisp/oop/clos/pcl/ 
 (ຊՈXerox൛ͷ ’92ͷarchive)
 ɾgithub.com/sbcl/sbcl/tree/master/src/pcl/ Authors: Gregor Kiczales, Jim des Rivieres
 (AMOPຊ΍AOPͷݯྲྀ) গͳ͔Β͵CLॲཧܥͷ“CLOS෦෼”ͷίʔυϕʔε
 including cmucl, sbcl, ecl, … 4 

Back on Track: a little opportunity twitter.com/nfunato/status/1113851067333103617 ˞ ਖ਼֬ʹ͸ɺεϨશମ΍ຊࢿྉ(ͱ࣮ࡍͷwalker)Λޚཡ͍ͩ͘͞ 5 

code walker : “ίʔυ”Λา͖ճΔͨΊͷϢʔςΟϦςΟͷ૯শ า͘(૸ࠪ͢Δ)աఔͰɺίʔυͷղੳ΍ม׵Λߦ͏͜ͱΛ
 ໨తͱͨ͠ϑϨʔϜϫʔΫ Lispͷ৔߹ɺ૸ࠪର৅͸ Sࣜίʔυͦͷ΋ͷͩͬͨΓɺ
 ରԠ͢ΔAST(CLOS object nodeͷtreeͱ͔)ͩͬͨΓ ݹయతLisp༻walker(ؚPCL’s)͸ɺ௚઀ SࣜίʔυΛ૸ࠪͯ͠Δ So, what is a code walker? ͜Ε͕ code walking ͷग़͔ࣗͱࢥ͍·͢ (ASTͷ૸ࠪࣗମ͸
 ίϯύΠϥͷsemantic analysis౳͕ී௨ʹߦ͍ͬͯΔ͜ͱͰ͢) 6 

Unfortunately, in practice, it’s not really the case that “it’s really easy to parse Lisp code.” …… (ঞ༁)
 SࣜΛread͢Δͷ͸؆୯΍ɻ͚ͲɺίʔυΛparse͢Δͷ͸ඞͣ ͠΋ͦ͏΍ͳ͍ɻS্ࣜʹఆٛ͞ΕͨCommon Lispͷ “ߏจ” Λ ཧղͤͳ͔͋Μ͔Β΍ɻͨ·ͨ·νϟνϟͬͱͰ͚Δ৔߹΋
 ͋Δ͔΋΍͚Ͳɺ׬શʹparse͢Δͷ͸େม΍Ͱ! ஌ΒΜ͚Ͳɻ SࣜҎ֎ʹLispͷ “ߏจ” ͳΜ͚ͯ͋ͬͨͬ ??? Somewhat infamous 
 “Lisp code walker” problem [5] 7 

Simple tree walkers [6] ;; શͯͷsubform(cons·ͨ͸atom)ʹରͯ͠ ؔ਺fnΛݺͿ (defun walk-tree (tree fn) (subst-if t (constantly nil) tree :key fn) ;; subform͕atomͷ৔߹ʹ ؔ਺fn ΛݺͿ (defun walk-tree-atoms (tree fn) (tree-equal tree tree 
 :test (lambda (atom1 atom2) (declare (ignore atom2)) (funcall fn atom1) t))) ͜ΕΒ͸SࣜΛίʔυͰͳ͘୯ͳΔσʔλͱͯ͠ݟ͍ͯΔ͚ͩ ྫ͑͹ɺࣗ༝ม਺Λநग़͍ͨ͠ɺͱ͍ͬͨ৔߹͸ Ͳ͏͢Δͷʁ
 ؤுͬͯ fn Λॻ͘ͷ? 8 

Another Problem posed by [3] ˞ ЕΛlambdaʹஔ׵ͯ͠[3]ͱશ͘ಉ͡ίʔυʹ͢ΔͱɺANSI CLͰ͸
 ɹ ແݶ࠶ؼݺग़ʹͳΔ (ޙड़ͷsb-cltl2:macroexpand-all ͳΒେৎ෉) (defmacro kond (&rest cls) ; (if cls `(if ,(caar cls) (progn ,@(cdar cls)) (kond ,@(cdr cls)))))
 
 (defun my-macroexpand-all (tree); (let ((x (macroexpand tree))) (if (atom x) x (mapcar #’my-macroexpand-all x)))) condͷnaiveͳϚΫϩఆٛ ໰୊ͷ͋ΔԶʑmacroexpand-all 9 ;;
 (my-macroexpand-all ’(mapcar #’(λ (kond) (car kond)) list)) ҎԼΛ࣮ߦ͢ΔͱͲ͏ͳΔͷ͔ʁ => (mapcar #’(λ nil (car kond)) list) 

So, WTF is a “Lisp” code walker!? Common Lispͷcode walker͸ “ίʔυ” Λద੾ʹ૸ࠪ͢ΔͨΊʹ
 CLͷsyntax formͷ஌ࣝ Λ͍࣋ͬͯΔʂ ্ه “஌ࣝ” ͸ ௨ৗͷݴޠͰ͍͏ "semantics" ʹ૬౰͢Δɺͱ͍͏ͷ͕
 ଥ౰Ͱ͠ΐ͏͕ɺCLͷ৔߹͸ reader macro, macrolet, symbol-macrolet ౳΋͋ΔͷͰɺparse΍syntaxͱ͔͍ͬͨݴ༿ͷ࢖͍ํʹ஫ҙ͕ཁΔ͔΋ ͠Ε·ͤΜ (͜͜Ͱ͸ਂೖΓ͠·ͤΜ)
 10 

Review: what is a syntax form? Common Lispͷߏจنଇʹ͓͚Δform (ධՁର৅ͱͳΔSࣜ) atom: γϯϘϧɺϦςϥϧ(਺஋ɺจࣈɺ…) compound form: Χοίͷ͋ΔSࣜ function form: ؔ਺ద༻ — શҾ਺͕ؔ਺ݺग़͠લʹධՁ͞ΕΔ syntax form: ධՁ͞ΕΔҾ਺΍ධՁॱ͸ݸผʹఆٛ͞ΕΔ! special form: led by known 25 operators (CLHS 3.1.2.1.2.1) built-in macro form: ୔ࢁ͋Γͦ͏͚ͩͲ Ͳ͏͢Δͷ? user-defined macro form: ޙ͔Β௥Ճ͞ΕͨΒ Ͳ͏ͳΔͷ? 11 

So, WTF is a “Lisp” code walker!? Common Lispͷcode walker͸ “ίʔυ” Λద੾ʹ૸ࠪ͢ΔͨΊʹ
 ANSI CLͷ25छྨͷsyntax operatorͷ஌ࣝ Λ͍࣋ͬͯΔʂ ϢʔβఆٛϚΫϩϑΥʔϜͷධՁنଇ͸Ͳ͏͢Δͷ?
 ૊ࠐΈϚΫϩϑΥʔϜͷධՁنଇ͸Ͳ͏͢Δͷ?
 PCL͸ɺCLtL1্ͷ಺෦DSL (ྲྀߦΓݴ༿ͰӠ͑͹ CLOS͔ΒCLtL1΁ͷ
 τϥϯεύΠϥ)Ͱɺ࠷Լ૚ʹ code walkerͷϨΠϠΛ͍࣋ͬͯͨʂ
 12 ˠ ࣮͸ ૊ࠐΈϚΫϩϑΥʔϜ΋expandͰ͖Δʂ 㱺 SBCLʹ΋ sb-walkerύοέʔδͱͯ͠࢒͓ͬͯΓɺ࢖͑·͢ʂ ˠ expand͢Ε͹ɺεϖγϟϧϑΥʔϜʴؔ਺ϑΥʔϜʹͳΔʂ 

You could even expand a defclass form! ࣮͸ ૊ࠐΈϚΫϩϑΥʔϜ΋expandͰ͖Δʂ (cf. CLHS 3.1.2.1.2.2) 13 

Functionality of the PCL code walker [1] subform૸ࠪͷࡍɺ໘౗ͳॲཧΛ୅ߦͯ͘͠ΕΔϑϨʔϜϫʔΫ special formͷ૸ࠪํ๏Λ஌͍ͬͯΔ ධՁنଇΛ஌͍ͬͯΔ (ධՁ͢΂͖subform ΍ ධՁͷॱং) ૸ࠪதʹ let౳ʹૺ۰͢Ε͹ɺࢀর͢Δ؀ڥΛద੾ʹߋ৽͢Δ macro formʹૺ۰͢Ε͹ɺͦͷ৔Ͱల։͔ͯ͠Β૸ࠪͯ͘͠ΕΔ macrolet ΍ symbol-macrolet ͱ͍ͬͨہॴϚΫϩʹ΋ରԠ Ϣʔβ͕ߦ͍͍ͨॲཧ͸ɺؔ਺Ҿ਺ͰࢦఆͰ͖Δ ૸ࠪதʹ෼͔ͬͨ͜ͱΛه࿥͢Ε͹ɺίʔυͷղੳॲཧʹͳΔ ֤subform͸ walkerͷฦΓ஋Ͱஔ׵͞ΕΔ(ม׵ॲཧ͕Մೳ) 14 

Major APIs of SBCL code walker [1,14] [Macro] (define-walker-template name &optional template)
 syntax formͷධՁنଇΛڭ͑ࠐΉͨΊͷϚΫϩ
 
 15 
 
 
 
 [Function] (walk-form form &optional env walk-fn)
 ΤϯτϦAPI: walk-fnΛenvʹՃ͑ͯ walk-form-internalΛݺͿ
 
 
 
 
 
 
 ҎԼͷAPI΋export͞Ε͓ͯΓɺwalk-fnͷதͳͲͰ࢖͑Δ
 [Variable] *walk-form-expand-macros-p* (nil by default)
 macro formʹର͢ΔwalkerͷฦΓ஋Λల։݁Ռʹ͢Δ͔Ͳ͏͔?
 [Function] (var-lexical-p var-name env)
 [Function] (var-special-p var-name env)
 [Function] (var-globally-special-p var-name)
 [Function] (var-declaration decl-sym var-name env) 

(define-walker-template name template) pre-defined walker Λ༩͑ΔςϯϓϨʔτϚΫϩ (ҎԼ͸هड़ྫ) 
 (define-walker-template if walk-if)
 (define-walker-template function (nil :call))
 (define-walker-template setq (nil :repeat (:set :eval)))
 (define-walker-template block (nil nil :repeat (:eval)))
 (define-walker-template catch (nil :eval :repeat (:eval))) if ʹ͸ɺઐ༻ؔ਺walk-ifΛ࢖͏(SBCLͰ͸໿൒਺͕͜ͷλΠϓ) function/setq/block/catchʹ͸ɺsubformͷධՁنଇΛද͢
 ϛχݴޠΠϯλϓϦλʹ༩͑ΔcontextγϯϘϧ͕ྻه͞Ε͍ͯΔ
 :repeat͸ɺ௚ޙͷྻ͕0ճҎ্܁Γฦ͞ΕΔ͜ͱΛද͠ɺ
 setqͷྫ͸ɺߏจ͕ (setq {place val}*)Ͱ͋Δ͜ͱΛ͍ࣔͯ͠Δ ˞ ݱߦͷίʔυͰ͸ɺsetqͷwalker-template͸ઐ༻ؔ਺ʹͳ͍ͬͯ·͢ 16 

(walk-form-internal form context env) => form’ walkerͷmain driver
 ɾ ॴ༩ͷ؀ڥ(env: lexical؀ڥ΍walk-fn౳)Ͱ formΛwalk͢Δ
 ɾ context(:eval, :set, :call, etc)Λิॿ৘ใͱͯ͠࢖༻͍ͯ͠Δ 1. walk-fnͰformΛwalk͢Δ ( walk-form-internalͷuser hook )
 ୈ2ฦΓ஋͕non-nilͷ৔߹͸ 2.ʹਐ·ͣearly return͢Δ
 2. walk-fnͷୈ1ฦΓ஋Λର৅formͱͯ͠ walk͢Δ
 2a. setq for var ΍ symbol macro˞ ͷ৔߹
 2b. setf for compound formͷ৔߹
 2c. templateఆٛ͞Εͨform(i.e. special form)ͷ৔߹
 2d. macro form˞ͷ৔߹
 2e. function call formͷ৔߹ ˞ ϚΫϩ͸૸ࠪ௚લʹల։͞ΕΔ 17 

walk-fnɿ3rd param for walk-form ࠶ܝ: walk-fn ͸ɺwalk-form-internal ͷ user hook ʹ૬౰͢Δ
 ɾ Ҿ਺ͷܗࣜͱҙຯ͚ͮ͸walk-form-internalͱಉ͡ʂ
 ɾ contextΛ࢖༻͢Δͷ΋ࣗ༝
 ɾ subformʹରͯ͠walk-form(-internal)Λ࠶ؼతʹݺΜͰΑ͍ walk-formݺग़͠Ͱলུ࣌͸ɺҾ਺ͷformΛͨͩฦ͚ͩ͢ͷؔ਺ ୈ1ฦΓ஋ʹՃ͑ͯɺୈ2ฦΓ஋Λฦ͢͜ͱ͕Ͱ͖Δ
 ɾ non-nilΛฦͨ͠৔߹ɺwalk-form-internal͕early return ͢Δ
 (sbclͰ͸macroexpand-all͕quasiquoteͷॲཧͰ࢖͍ͬͯΔ)
 ɾ nil Λฦͨ͠৔߹ɺୈ1ฦΓ஋͕Ҿ͖ଓ͖walk͞ΕΔ
 㱺 walk-fnͰม׵Λࢪͨ͠formΛwalkͤ͞Δ͜ͱ΋Ͱ͖Δ 18 

Where to use a Lisp code walker? ҙ֎ʹ࢖͍Ͳ͜Ζ͕೉͍͠ʁ
 ɾ ߴϨϕϧˠ௿ϨϕϧͷશҬม׵(e.g. ී௨ͷίϯύΠϥ)ͷ৔߹ɺ
 ɹ macroexpand-all͔ͯ͠Βॲཧ͢Ε͹ࡁΉ͜ͱ΋ଟ͍ [3]
 ɾ code walkingແ͠ʹैདྷख๏ͷϚΫϩͰॻ͚ΔͳΒɺͦΕͰे෼ Ͱ͋Ε͹ɺҎԼͷΑ͏ͳॲཧͰ࢖͏͜ͱ͕༗ޮ… ͱߟ͑ΒΕΔ
 ɾ macroexpand-all ͦͷ΋ͷ
 ɾ code walkingແ͠ʹ͸ॻ͖ͮΒ͍macro expander
 ɾจ๏஌ࣝΛ༻͍ͨղੳ/ม׵
 ɾෳ਺ϨϕϧͷSࣜʹލΔղੳ/ม׵ (ؒʹsyntax form͕ڬ·Γ͏Δ)
 ɾ ιʔεϨϕϧͷ··௿Ϩϕϧ΁ม׵ͤͣʹߦ͍͍ͨղੳ/ม׵ 19 

Examples
 ʙ macroexpand-all (1/2) ʙ ;; sbcl/src/contrib/sb-cltl2/macroexpand.lisp (note: a bit modified)
 (defun sb-cltl2:macroexpand-all (form &optional env)
 (flet ((walk-fn (subform context env2)
 (aif (and (eq context :eval)
 (listp subform) (symbolp (car subform))
 (get (car subform) :partial-macroexpander))
 ;; The partial expander, only for QUASIQUOTE as of Aug 2019,
 ;; must return T as its 2nd value to stop the walk
 (funcall it subform env2)
 subform))
 ;; just call WALK-FORM after binding *walk-…-macros-p* to t
 (let ((*WALK-FORM-EXPAND-MACROS-P* t ))
 (WALK-FORM form env #’walk-fn))))
 
 ɾ walk-fn͸ɺquasiquoteͷͱ͖Ҏ֎͸ ୯ʹsubformࣗ਎(ͱnil)Λฦ͢
 ɾ ଈͪ *walk-…-macros-p* Λม͚͑ͨͩͷ४ඪ४ಈ࡞͕ɺϚΫϩશల։ 20 

Examples
 ʙ macroexpand-all (2/2) ʙ (defvar *my-repl*
 '(defun my-read-eval-print-loop (level)
 (with-simple-restart (abort "Exit command level ~D." level)
 (loop
 (with-simple-restart (abort "Return to command level ~D." level)
 (let ((form (prog2 (fresh-line) (read) (fresh-line))))
 (prin1 (eval form))))))))
 
 (defun mex-test (level &optional (form (fourth *my-repl*)))
 (ecase level
 (0 (macroexpand-1 form))
 (1 (macroexpand form))
 (2 (sb-cltl2:macroexpand-all form)))) 
 
 ɾ ࣮ࡍʹల։ͯ͠ΈΔͱɺ(locally (declare ...)) ͱ͔৭ʑͬ͘෇͍ͯͯɺ
 ɹ ίʔυղੳπʔϧͱ͔࡞Δͷ͸ɺͦΕ΄Ͳ؆୯Ͱ͸ͳ͍ͱ෼͔Δ 21 


 (defmacro with-constant-folding (&body body &environment env)
 (WALK-FORM `(progn ,@body) env
 (lambda (form context env2)
 (if (and (eq context :eval) (consp form)
 (member (car form) '(+ - * /))
 (every #'constantp (cdr form)))
 (eval form)
 form))))
 
 ɾશҾ਺͕ఆ਺ͳ࢛ଇԋࢉϑΥʔϜΛ Τόͬͯ͠·͏ɺͱ͍͏΋ͷ
 ɾೖΕࢠʹͳͬͨ਺ࣜ΋৞ΈࠐΊΔΑ͏ʹɺ(cdr form)ͷ֤ཁૉʹ
 ࠶ؼతʹwalk-form-internalΛݺͼग़͢ྫ΋ڍ͛ΒΕ͍ͯΔ[1] 
 ɾeval ͸௨ྫϦϦʔε͢ΔΑ͏ͳίʔυͰ͸࢖Θͳ͍͠ɺSBCL্Ͱ͸
 (constantp ‘(* 3 4)) => T ͷΑ͏ͳ࠷దԽ͕ߦΘΕΔͷͰɺ͜ͷྫࣗମʹ
 ࣮༻ੑ͕͋ΔΘ͚Ͱ͸ͳ͍(͜͜ΒลͰ࠷దԽͯ͠ΔՄೳੑ΋͋Δ͕…) Examples
 ʙ constant folding at source code level [1] ʙ 22 


 (defmacro anonymous-walker ( …… &environment env)
 (WALK-FORM 
 env
 (lambda (form context env2 ) …… )))
 
 
 ɾ લทͷΑ͏ʹɺWALK-FORMΛexpanderʹ༻͍Δ৔߹ͷύλʔϯ
 ɾ sb-cltl2:macroexpand-all ͷྫͰ͸ defun Λ࢖͍ͬͯΔ͕ɺ
 ࢖ΘΕͲ͜Ζ͕ະఆͰ caller contextͷimplicitͳ؀ڥʹΞΫηε͢Δ
 ৔߹͸ɺ্ྫͷΑ͏ʹdefmacroΛ࢖͑͹ɺmacro lambda listͷ
 &environment(CLHS 3.4.4) ܦ༝Ͱɺ؀ڥΛҾ͖ܧ͍ͰऔΓग़ͤΔ
 ˞ ͜Ε͸code walkerͱ͍͏ΑΓ defmacroͷ࿩Ͱ͸͋Γ·͢ Examples
 ʙ a general pattern on WALK-FORM based macroexpander ʙ 23 


 (defvar *sample* '(let ((x (list Y)))
 (tagbody x
 (setq x (cons 'x X))
 (if (< (length X) 8) (go x)))
 (print X)))
 
 (defun replace-varref (&optional (form *sample*) env)
 (WALK-FORM form env 
 (lambda (form context env)
 (if (and (eq context :eval) form (symbolp form)
 ;; (VAR-LEXICAL-P form env)
 )
 "Foo" form)))) 
 
 ɾେจࣈ/੺ࣈͷม਺ࢀর͚ͩஔ׵͞ΕΔ (ࠨล஋, λά, ఆ਺͸ର৅֎)
 ɾίϝϯτ෦Λ༗ޮʹ͢Δͱɺࣗ༝ม਺Ͱ͋ΔY͸ஔ׵͞Εͳ͘ͳΔ Examples
 ʙ replacing every variable reference with "Foo" [1] ʙ 24 

(defun same-var-p (var env1 env2) ; variable-same-p in Fig.10 [2]
 (eq (VAR-LEXICAL-P var env1) (VAR-LEXICAL-P var env2)))
 
 ;; lambda bodyͷ؀ڥͱ lambda formͷஔ͔Εͨ؀ڥͰɺม਺Λൺֱ͍ͯ͠Δ
 (defun aggregate (lmd-form lmd-env &aux (pool '()))
 (WALK-FORM lmd-f lmd-e
 (lambda (f c e) 
 (when (and (eq c :eval) f (symbolp f) (same-var-p f e lmd-env))
 (pushnew f pool))
 f))
 pool)
 (defun main (&optional (form '(let (x y z) (lambda (y) (foo x y z))) env))
 (WALK-FORM form env 
 (lambda (f c e) 
 (when (lambda-form-p f) (return-from main (aggregate f e)))
 f)))
 ;; (main) => (Z X) -- Ϋϩʔδϟม਺͕(ͦ͏Ͱͳ͍yΛআ͍ͯ)ूΊΒΕΔ Examples
 ʙ aggregating closure variables in a lambda form ʙ 25 


 (defun binding-name (b) (if (consp b) (car b) b))
 (defun gen-subst (b &aux (n (binding-name b))) (cons n (gensym (string n)))
 (defun gen-substs (bindings) (mapcar #'gen-subst bindings))
 
 (defun rename-variables (form alist env) ; from Fig.10 [2]
 (WALK-FORM form env ; using "same-var-p" in the previous page
 (lambda (f c e &aux pair) 
 (if (and f (symbolp f) (setq pair (assoc f alist)) (same-var-p f e env))
 (cdr pair) f))))
 (defun alphatize-let-body (let-form &optional env)
 (destructuring-bind (let bindings . body) let-form
 (let ((substs (gen-substs bindings)))
 (rename-variables `(progn ,@body) substs env))))
 
 ɾεϖʔεͷؔ܎Ͱ [2] ʹ฿ͬͯ let bodyΛม׵͢ΔॲཧͷྫͷΈࣔͨ͠
 ɾ͜ͷํ๏͸ɺݸʑͷlet͝ͱͷalistʹconsult͕ཁΔͷͰΠέͯͳ͍͔΋
 (΋ͱ΋ͱ[2]͸ԶʑϧʔϓϚΫϩͷల։໨తͰɺ൚༻Ћม׵Ͱ͸ͳ͍) Examples
 ʙ alpha conversion (1/2) ʙ 26 


 (alphatize '(with-open-file (st st) ; stream(bound-var) and string(free-var)
 (do ((l (read-line st) (read-line st nil 'eof)))
 ((eq l 'eof) "Reached end.")
 (format t "~&*** ~A~%" l))))
 => (with-open-file (#:st1 st)
 (do ((l (read-line #:st1) (read-line #:st1 nil 'eof)))
 ((eq l 'eof) "Reached end.")
 (format t "~&*** ~A~%" l)))
 
 ɾ࣮͸macroexpand-allͷޙ(͋Δ͍͸ηοτͰ)ͳΒЋ-conversion͸༰қ
 ɾ্هͷΑ͏ͳ source-to-source Ћ-conversionΛ͍͕ͨ͠ Մೳ͔?
 ɹɾ ࠓͷ SBCL code walkerʹ͸खΛೖΕͣʹɺwalk-fn ͷهड़͚ͩͰɺ
 ɹɹ ߴʑ2-pass͔3-passͰɺ͘Β͍ͷറΓͷ΋ͱͰ…
 ɹɾ ݸʑͷม਺ࢀরΛผΦϒδΣΫτͱͯ͠هԱҬΛઃఆ͍͕ͨ͠
 ɹɹ γϯϘϧͳͷͰͰ͖ͳ͍… 㱺 গ͠ߟ͕͑ͨૉ௚ʹ͸Ϝζͦ͏ Examples
 ʙ alpha conversion (2/2) ʙ 27 

(let ((i 0)) (defmacro iter (&body body)
 (iter (with-gensyms (iter-block acc start)
 (while (< i 5)) `(block ,iter-block
 (incf i) (let ((,acc nil))
 (print acc) (macrolet ((while (test) …)
 (collect (* 4 i) :into acc))) (collect (thing) …)
 (tagbody ,start ,@body (go ,start)))))))
 
 ɾiterΛdefmacroͰɺwhile΍collectΛiterఆٛதͷmacroletͰॻ͖͍͕ͨɺ
 iterͷల։ܗ͕ collectͷల։ΑΓઌ͔ͭผʹఆ·ΔͷͰ೉͍͠[8]
 (iterͱcollectͷϚΫϩఆ͕ٛɺద੾ʹίϛϡχέʔτ͢Δඞཁ͕͋Δ͕ɺ
 ͦͷΑ͏ͳػߏ͸ݱߦͷdefmacro/macroletʹ͸ແ͍)
 
 ɾiterateϚΫϩ͸ɺࣗલcode walkerͰcollectઅͷࣄલղੳΛ͍ͯ͠ΔΒ͍͠
 㱺 Ұྫ͕ͩɺSBCL code walkerͰ΋ ࣍ทͷΑ͏ͳײ͡Ͱॻ͚ͦ͏ Examples
 ʙ transforming across multi-level forms (1/3) ʙ 28 

(defmacro iter (… &environment env)
 (let ((analysis-info (make-analysis-info-object))
 (labels ((walk-fn (subform context subenv)
 (cond ((iter-form-p subform) (walk-iter-form …)) … (t subform))))
 ;; w-i-f͔Βw-c-fΛ(ඞཁͳΒ͹walk-form-internal౳Λܦͯ)ݺΜͰΑ͍
 (walk-iter-form (subform context subenv) …)
 (walk-collect-form (subform context subenv) …)
 (expand-iter-form () ...)) 
 (WALK-FORM form env #’walk-fn) ; analyze iter-macro form
 (expand-iter-form))) ; pass-2: expand using analysis-info 
 
 ɾWALK-FORMͷwalk-fnͷݺग़͠͸ɺmacroletͷbodyͱҧͬͯɺಛఆͷ
 formʹؔ࿈෇͚ΒΕ͍ͯͳ͍ (ͱ͍͏͔ શsubformʹରͯ͠ݺ͹ΕΔ) 
 㱺 iter formͷղੳͷதͰɺ಺ଆͷcollect formͷղੳΛͯ͠΋Α͍
 ɾ্ྫ͸2-pass͕ͩɺՄೳͳΒWALK-FORMͷ1-pass͚ͩͰల։ͯ͠΋Α͍ Examples
 ʙ transforming across multi-level forms (2/3) ʙ 29 


 ɾҎԼͷwebจॻ[11]ʹ΋ɺલทͱಉ͡झࢫͷࢦఠ͕͞Ε͍ͯΔ 
 (code walkerͷํ͕ɺࣗ੹ͷ୅ঈʹmacroΑΓࣗ༝౓͕େ͖͍Θ͚Ͱ͢Ͷ)
 What code walkers can do that macro can’t
 
 The main difference between a macro and a code walker is coverage. 
 A macro can change only the parts of the program that call the
 macro (*1). Whereas a code walker can change the whole program.
 
 *1: One reason macros can’t walk over a whole program is the 
 runtime system doesn’t let them. …
 ɾSBCLʹ࢒͍ͬͯΔ walk-formͷݺग़Օॴʹ΋ɺmulti-levelͰ͸ͳ͍͚Ͳ
 ෳ਺ͷsubformղੳΛdispatch͍ͯ͠Δͱ͜Ζ(boot.lisp)͕͋Γ·͢ Examples
 ʙ transforming across multi-level forms (3/3) ʙ 30 


 (macroexpand-1 
 '(-> "THREE" 
 string-downcase (char 0) char-code (complex (1+ $) (1- $))))
 
 => (LET (($ "THREE"))
 (SETQ $ (STRING-DOWNCASE $)) ; ؔ਺໊͚ͩͷͱ͖΍
 (SETQ $ (CHAR $ 0)) ; $͕ͳ͍ͱ͖͸ɺ
 (SETQ $ (CHAR-CODE $)) ; $Λ1st-argʹิ͍
 (SETQ $ (COMPLEX $ (1+ $) (1- $)))) ; $͕͋Δͱ͖͸ɺͦͷ·· 
 
 ɾSBCLͷϝϯςφʹΑΔSBCL walkerΛ࢖ͬͨॴҦ pipe operatorͷྫ
 ɾwalk-fn ޙͷ early return Λ࢖ͬͨྫʹ΋ͳ͍ͬͯΔ
 ɾଞʹ code walker Ͱ “defmacro/g! ͷχονͳ໰୊Λਖ਼͘͠ॲཧ͢Δ”
 ɹͱ͍ͬͨझࢫͷهࣄ[9]΋ॻ͍͓ͯΒΕ·͢ Examples
 ʙ writing pipe operator [10] ʙ 31 

;; (json:decode-json-from-string "{\"a\": 1, \"b\": {\"bb\": 2}, \"c\": 3}")
 ;; => ((:A . 1) (:B (:BB . 2)) (:C . 3))
 (defmacro with-json (json &body forms &environment env)
 (with-gensyms (decoded)
 `(let ((,decoded (json:decode-json-from-string ,json)))
 ,@(mapcar (lambda (form) (walk-with-json-body decoded form env)) forms))))
 
 (defun test-with-json () ; nesting with-json works properly
 (with-json "{\"a\": 1, \"b\": {\"bb\": 2}, \"c\": 3}"
 (let ((@c 999))
 ;; Here, @a and @b.bb take their json-value, i.e. 1 and 2,
 ;; however @c keeps a value 999 bound by let, not 3 in json description.
 (list @a @b.bb @c
 (with-json "{\"a\": 10, \"b\": {\"bb\": 20}, \"c\": 30}"
 (let ((@c 9990))
 (list @a @b.bb @c))))))) ; => (1 2 999 (10 20 9990))
 
 ɾjsonΞΫηε༻ͷsymbol macro (@quek͞Μ࡞) ͷಈ࡞ྫΛࣔͨ͠΋ͷ
 Examples
 ʙ a tiny symbol macro converted to an alist accessor [7] (1/2) ʙ 32 

(defun free-varref-p (form ctxt env)
 (and (eq ctxt :eval) form (symbolp form) 
 (cond ((VAR-LEXICAL-P form env) nil)
 ((SB-WALKER::VARIABLE-SYMBOL-MACRO-P form env) nil)
 (t t))))
 
 ;; a macro expander intrinsic function using SB-WALKER:WALK-FORM
 (defun walk-with-json-body (decoded-str form env) 
 (WALK-FORM form env
 (lambda (subform ctxt subenv)
 (if (and (free-varref-p subform ctxt subenv)
 (@-prefixed-symbol-p subform))
 (symbol-to-assoc-form subform decoded-str) ; make an alist accessor
 subform))))
 
 ɾhu.duim.walkerͰ࡞ΒΕ͍ͯͨΦϦδφϧ Λ SBCL walkerʹҠ২ͨ͠΋ͷ
 ɾ࣮૷࢓༷΋΄΅ಉ౳ʹ͠ɺࠩ෼΍ࣗ໌Ͱͳ͍ίʔυͷΈ͍ࣔͯ͠Δ Examples
 ʙ a tiny symbol macro converted to an alist accessor [7] (2/2) ʙ 33 

࡞ΔϝϦοτ͕͋Δ͔ "ʁ" ͕ͩɺҎԼͷΑ͏ͳ΋ͷ΋ࢥ͍౰ͨΔ semantic checker෇͖embedded HTML/XML macro
 ✔ λάΛϚΫϩͰදͭͭ͠ɺҟͳΔϨϕϧͷλάؒʹ࣮ߦformΛ
 ڬΊͯ(ؚsyntax form)ɺλά੍ؒ໿(͜ͷλάͷԼʹ͋ͷλά͕
 ɹ 1ϲҎ্དྷΔ/དྷͳ͍౳)ͷݕࠪΛ࡞ΓࠐΉͷʹɺcode walkerΛ࢖͑Δ
 ☓ λάΛϚΫϩͰͳ͘ɺCL-WHOͷΑ͏ʹσʔλγϯϘϧʹ͢Δ
 ɹ ͜ͱ͕͋Δ͔΋(׶͑ͯෆ׬શͳλά࢓༷Λड͚༰Ε͍ͨ৔߹ͳͲ)
 ☓ ࡞ΓํΑΓ΋ɺଞʹॏࢹ͢΂͖͜ͱ͕ଟͦ͏ͳԠ༻Ͱ͢͠… source code cross reference tool
 ✔ ιʔεϨϕϧͷ৘ใղੳʹɺcode walkerΛ࢖͑Δ
 ☓ طʹslime͕XrefΛऔࠐΈࡁͳͷͰɺγϣϘ͍΋ͷ͸ཁΒͳ͛͞ Other Usages? 34 

Ͳͷcode walker(PCL/SBCL, Arnesi, …)ʹ΋ɺ؀ڥΞΫηεAPI͕ඞཁ CLtL2ͷ؀ڥΞΫηεAPIͷΑ͏ͳ΋ͷ͸ɺANSI CL͸نఆ͍ͯ͠ͳ͍͕ ॲཧܥʹΑͬͯ͸ఏڙ͓ͯ͠Γɺportability layer࡞੒ͷࢼΈ[13]΋͋Δ SBCL͸ɺҎԼͷPCL༝དྷͷ؀ڥΞΫηεAPIΛɺαϙʔτॲཧܥ(10छྨ Ҏ্)Λࣗ਎༻ͷΈʹॖݮͯ͠ఏڙ͍ͯ͠Δɿ
 
 [Macro] (with-augmented-environment 
 ((new-env old-env &key functions macros) &body body) 
 ৽ͨͳϨϕϧͷ؀ڥΛnew-envʹଋറ͠ɺbodyΛ࣮ߦ͢Δ 
 [Function] (environment-macro env key-symbol)
 envதͷlexicalཁૉ(ϚΫϩɺม਺ɺwalk-fn౳)ͷྖҬ΁ͷΞΫηοα Implementation of SBCL code walker ˞ ໊લ(…-macro) ͕গ͓͔͍͕͠͠ɺެ։APIͰ΋ͳ͍͍͔ͤݩͷ··Ͱ͢ 35 

Arnesi code walkerখ࢙ (as of Aug 2019)
 ɾMarco BaringerࢯͷArnesi utility(github্ͷอશ)ͷҰ෦Ͱ͋Δ
 ɹcode walker͕ factored out͞Εͯ cl-walkerʹͳͬͨ
 ɾcl-walker(gitlab)ͷAttila LendvaiࢯʹΑΔ࠷ऴߋ৽͸໿10೥લ
 ɾͦͷޙɺLendvaiࢯʹΑΔhu.dwim(darcs)ͷҰ෦ͷhu.dwim.walker
 ɹͱͯ͠ܧଓ͠ɺquicklispʹ͸2015೥ͷϦϦʔε͕ऩ࿥͞Ε͍ͯΔ
 ɾArnesi ˠ cl-walker ˠ hu.dwim.walkerͷؒɺ؀ڥAPI౳ʹߋ৽͋Γ
 (ͨͩ͠hu.dwim͸Զ༷ϚΫϩଟ༻Ͱશମ͕ີ݁߹ͱ͍͏໰୊͕͋Δ) ಛ௃ɿCLOS object node ͷ AST Λ࢖༻
 ɾParserͰASTੜ੒ ˠ User's methodsͰղੳ/ม׵ ˠ Sࣜʹٯม׵ 
 ɾBrowsable document͸͜ͷลΓ(Arnesi, hu.dwim.walker) Arnesi code walker and its descendants ݱ୅తʂ 36 (PCLΑΓ͸…) 

@guicho271828͞Μͷهࣄ[8]ͷຊࢫ͸ɺcode wakingΛͯ͠ҟͳΔ
 ϨϕϧʹލΔม׵Λߦ͏ͨΊʹʮdefmacro(΍macrolet)͕Ҿ਺Ͱ continuationΛड͚औΕΔͷ͕ɺ͍͍Μ͡ΌͶʁʯͱ͍͏ఏҊ 㱺 [1] (Fig.12) ʹ΋ɺPCL code walkerͷwalk-fn͕ continuationΛ
 ɹͱΕΔΑ͏ʹ͢Δͱ৭ʑॊೈͳ੍ޚ͕Ͱ͖Δɺͱ͍͏ಉझࢫ
 ɹهࡌ͕͋ͬͨ (ͦͷޙͷਐల͸ެ։͞Εͳ͔ͬͨ໛༷) portable code walkerͷࢼΈͱͯ͠ɺmacroexpand-dammit[4]΍ agnostic lizard[12]͕͋Γɺ͍ͣΕ΋·ͣ͸macroexpand-all͔Β
 Ξϓϩʔν͍ͯ͠Δ͕ɺޙऀ͸callback΋ॻ͚ΔΒ͍͠ !?
 (ࠓճ͸ௐࠪෆ଍ʹ͖ͭɺ͜͜ʹ͸ه͍ͯ͠ͳ͍͕ɺίʔυ͸
 gitlab্Ͱެ։͞Ε͓ͯΓɺquicklispʹ΋ऩ࿥͞Ε͍ͯΔ༝) Miscellaneous notes 37 

Personal View code walker͸ ίʔυ/ߏจ໦ͷ૸ࠪϥΠϒϥϦͱ͍͏͜ͱʹͳΔ͕ɺ
 Lispͷ৔߹ɺsyntax formͷ஌ࣝ(p.14)͚ͩͰ૸ࠪΛنଇԽͰ͖ɺ1st class 
 functionΛ࢖͑ͨͨΊɺPCLҎલ͔ΒϑϨʔϜϫʔΫԽ͕ਐΊΒΕͨҹ৅ ͕͢͞ʹ PCL walkerΑΓ͸ Arnesi walkerͷํ͕༏Ґ఺͕ଟ͍ؾ΋͢Δ
 ɾ ద੾ͳ(ந৅)ΫϥεͷαϒΫϥεԽͰɺⅰ)hook pointΛॊೈʹ૿΍ͤͯɺ
 ɹ ⅱ)ద੾ͳଐੑهԱҬΛ࡞Εͯɺⅲ)AST nodeͷจ๏஌ࣝΛ࢖͍қ͍
 ɾ CLOSͷڧྗͳmethod combinationΛ࢖͑Δ(before, after, around౳)
 ˞ ⅰɿݶఆతͳҐஔͷhook͚ͩͰΧελϜԽ͢Δύζϧʹ೰·ͳͯ͘Α͍
 ˞ ⅱ,ⅲɿྫ͑͹ɺม਺ͷoccurence͝ͱʹҟͳΔ৘ใΛ࣋ͨͤ΍͍͢౳ ໬΋ɺ༗ྗॲཧܥͷҰ෦ͱͯ͠ҡ࣋͞Ε͍ͯΔ͜ͱ͸େ͖͍
 (ͳͷͰɺຊࢿྉͰ͸ "ݹ͞͸֮ޛͰ" SBCL code walkerΛௐ΂ͯΈͨ) 38 

Summary CL code walker͸ɺcode walking (ίʔυ͋Δ͍͸ߏจ໦Λ૸ࠪͭͭ͠
 ߦ͏ॲཧ)ʹඞཁͳɺspecial form΍macro formͷॲཧΛ୅ߦͯ͘͠ΕΔ CLͷίʔυղੳɺϚΫϩఆٛɺݴޠ֦ு/಺෦DSL࡞੒ͳͲʹར༻Ͱ͖Δ PCLʹىݯΛ࣋ͭ SBCLͷcode walkerʹ͍ͭͯ؆୯ʹ঺հͨ͠
 (ଞͷ ”·ͱ·ͬͨ΋ͷ” ͱͯ͠͸ɺArnesiͷwalkerͱͦͷࢠଙ͕͋Δ) ANSI CLࡦఆ࣌ʹɺ؀ڥΞΫηεAPIͷ࠾༻͕υϩοϓ͕ͨ͠ɺ
 ॲཧܥґଘ෦ͷແ͍CL code walkerΛॻ͘ࢼΈ΋͋Δ
 ˞walkerͷػೳ࢓༷ΛͲ͜·Ͱͱଊ͑Δ͔…ʹ͸෯͕͋Γͦ͏ ͋Δcode walkerΛ࢖͏ࡍɺಛఆͷCLͰಈ࡞͢Δ͔Ͳ͏͔ʹཁ஫ҙ͕ͩɺ
 prototypicalͳ࣮ݧ/ղੳ/։ൃͷπʔϧͱͯ͠ͳΒ͹࢖͍ॲ͸͋Γͦ͏ 39 

Why are macros so important to Lisp programmers?
 Not merely for the syntactic convenience they provide, but because they are programs that manipulate programs, which has always been a central theme in the Lisp community. 
 If FORTRAN is the language that pushes numbers around, and C is the language that pushes characters and pointers around, then 
 Lisp is the language that pushes programs around.
 
 ˞ underline mine
 ˞ ਂொ͞Μͷϒϩά ʹ΋ ๜༁ͱڞʹࡌ͍ͬͯ·͢ Finally...
 ʙ A quote from “The Evolution of Lisp” ʙ 40 

Questions? 41 

[1] P. Curtis, (algorithms), Lisp Pointers, ACM SIGPLAN, 
 3(1):48-61, July 1989 (an intro. for PCL code walker)
 [2] B. v. Melle, Implementation of an Iteration Macro, Lisp Pointers, ACM SIGPLAN, 3(2-4):29-40, Apr 1990
 [3] R. C. Waters, Macroexpand-All: An example of a Simple Lisp Code Walker, Lisp Pointers, ACM SIGPLAN, 6(1):25-32, Jan 1993
 [4] J. Fremlin, Portable Common Lisp code walking with macroexpand-dammit, Jul 2009 (g000001, a Japanese intro.) [5] Hacker News, news.ycombinator.com/item?id=3607707, Feb 2012
 [6] Z. Beane, The tree-walkers of CL, CL Tips, Feb 2013
 [7] @quek, Using hu.dwim.walker (in Japanese), May 2013 References (1/2) 42 

References (2/2) [8] @guicho271828, Common Lisp is still hard to satisfy (in Japanese), Feb 2014
 [9] C. Rhodes, naive vs proper code-walking, Sep 2014
 [10] C. Rhodes, code walking for pipe sequencing, Sep 2014
 (showing examples using sbcl code walker, so does [9])
 [11] What Code Walkers can do that macros can’t, Aug 2016
 [12] M. Raskin, Writing a best-effort portable code walker in Common Lisp, Euro Lisp Symp. (gitlab repo.), Apr 2017
 [13] A. Gutev, cl-environments (github repo.), Sep 2017
 [14] github.com/sbcl/sbcl/blob/master/src/pcl/walk.lisp
 [15] @nfunato, companion code for this presentation (gist) 43