code-walkers-for-lisp.pdf

A presentation slide at "Kansai Lisp Users' Meeting" on 2019-09-28

Transcript

1. Code Walkers for Lisp ʙ or How to implement an inside DSL in Common Lisp ʙ Kansai Lisp Users’ Meeting 2019-09-28 @nfunato 1
2. Summary Abstract code walkerͱ͍͏ϢʔςΟϦςΟ଒ʹ͍ͭͯɺ΍͘͞͠঺հ Lispͷcode walkerͬͯɺ͍ͭ/ͳͥཁΔͷ? Ͳ͏໾ཱͭͷ? యܕతͳ༻్: ͪΐ͍ڽΓϚΫϩ, ղੳπʔϧ, ಺෦DSL౳ ࣮ྫͱͯ͠ɺࠓͦ͜ʹ͋Δ Common Lisp code walker Ͱ͋Δ
 PCL code walker (ported to SBCL) ͷ࢖͍ํΛ֓؍͢Δ code walkerͬͯɺ࢖ͬͯΔͱ͜Ζ͋Μ·Γݟͳ͘Ͷ? 
 ͱ͍͏ٙ࿭ʹ͍ͭͯগ͠ߟ͑ͯΈΔ 2
3. Who’re you? Why Lisp?
 ʙ it started from PCL ʙ ֶੜ࣌୅͸ॴଐݚڀࣨͰίϯύΠϥपลΛֶͿ ͳ͓ɺ35೥Ҏ্લʹ͖ͭ࣌ޮͰ͢ :-) ৬ྺ͸֓Ͷ୺຤ػثͷ௨৴ܥ(IP૚ΑΓԼ͕ଟ͍) ࢓ࣄ͸௿ϨΠϠਓͩͬͨͷʹ Կނ͔Lispʹ͸ັͤΒΕΔ
 CLʹັ͔ΕͨΩοΧέ: ’87ࠒ comp.sourcesʹྲྀΕͯͨ PCL
 ɹ̋ Portable Common Loops ✕ Practical Common Lisp FORTH, Haskell(Gofer), Smalltalk, OCaml, Erlang౳΋טͬͨ ஶऀԕӨ 3
4. BTW, Portable Common Loops is… ͘͢͝େࡶ೺ʹӠ͏ͱ CLOSͷϓϩτλΠϓ
 ɾen.wikipedia.org/wiki/CommonLoops (ྺ࢙)
 ɾwww.cs.cmu.edu/Groups/AI/lang/lisp/oop/clos/pcl/ 
 (ຊՈXerox൛ͷ ’92ͷarchive)
 ɾgithub.com/sbcl/sbcl/tree/master/src/pcl/ Authors: Gregor Kiczales, Jim des Rivieres
 (AMOPຊ΍AOPͷݯྲྀ) গͳ͔Β͵CLॲཧܥͷ“CLOS෦෼”ͷίʔυϕʔε
 including cmucl, sbcl, ecl, … 4
5. Back on Track: a little opportunity twitter.com/nfunato/status/1113851067333103617 ˞ ਖ਼֬ʹ͸ɺεϨશମ΍ຊࢿྉ(ͱ࣮ࡍͷwalker)Λޚཡ͍ͩ͘͞ 5
6. code walker : “ίʔυ”Λา͖ճΔͨΊͷϢʔςΟϦςΟͷ૯শ า͘(૸ࠪ͢Δ)աఔͰɺίʔυͷղੳ΍ม׵Λߦ͏͜ͱΛ
 ໨తͱͨ͠ϑϨʔϜϫʔΫ Lispͷ৔߹ɺ૸ࠪର৅͸ Sࣜίʔυͦͷ΋ͷͩͬͨΓɺ
 ରԠ͢ΔAST(CLOS object nodeͷtreeͱ͔)ͩͬͨΓ ݹయతLisp༻walker(ؚPCL’s)͸ɺ௚઀ SࣜίʔυΛ૸ࠪͯ͠Δ So, what is a code walker? ͜Ε͕ code walking ͷग़͔ࣗͱࢥ͍·͢ (ASTͷ૸ࠪࣗମ͸
 ίϯύΠϥͷsemantic analysis౳͕ී௨ʹߦ͍ͬͯΔ͜ͱͰ͢) 6
7. Unfortunately, in practice, it’s not really the case that “it’s really easy to parse Lisp code.” …… (ঞ༁)
 SࣜΛread͢Δͷ͸؆୯΍ɻ͚ͲɺίʔυΛparse͢Δͷ͸ඞͣ ͠΋ͦ͏΍ͳ͍ɻS্ࣜʹఆٛ͞ΕͨCommon Lispͷ “ߏจ” Λ ཧղͤͳ͔͋Μ͔Β΍ɻͨ·ͨ·νϟνϟͬͱͰ͚Δ৔߹΋
 ͋Δ͔΋΍͚Ͳɺ׬શʹparse͢Δͷ͸େม΍Ͱ! ஌ΒΜ͚Ͳɻ SࣜҎ֎ʹLispͷ “ߏจ” ͳΜ͚ͯ͋ͬͨͬ ??? Somewhat infamous 
 “Lisp code walker” problem [5] 7
8. Simple tree walkers [6] ;; શͯͷsubform(cons·ͨ͸atom)ʹରͯ͠ ؔ਺fnΛݺͿ (defun walk-tree (tree fn) (subst-if t (constantly nil) tree :key fn) ;; subform͕atomͷ৔߹ʹ ؔ਺fn ΛݺͿ (defun walk-tree-atoms (tree fn) (tree-equal tree tree 
 :test (lambda (atom1 atom2) (declare (ignore atom2)) (funcall fn atom1) t))) ͜ΕΒ͸SࣜΛίʔυͰͳ͘୯ͳΔσʔλͱͯ͠ݟ͍ͯΔ͚ͩ ྫ͑͹ɺࣗ༝ม਺Λநग़͍ͨ͠ɺͱ͍ͬͨ৔߹͸ Ͳ͏͢Δͷʁ
 ؤுͬͯ fn Λॻ͘ͷ? 8
9. Another Problem posed by [3] ˞ ЕΛlambdaʹஔ׵ͯ͠[3]ͱશ͘ಉ͡ίʔυʹ͢ΔͱɺANSI CLͰ͸
 ɹ ແݶ࠶ؼݺग़ʹͳΔ (ޙड़ͷsb-cltl2:macroexpand-all ͳΒେৎ෉) (defmacro kond (&rest cls) ; (if cls `(if ,(caar cls) (progn ,@(cdar cls)) (kond ,@(cdr cls)))))
 
 (defun my-macroexpand-all (tree); (let ((x (macroexpand tree))) (if (atom x) x (mapcar #’my-macroexpand-all x)))) condͷnaiveͳϚΫϩఆٛ ໰୊ͷ͋ΔԶʑmacroexpand-all 9 ;;
 (my-macroexpand-all ’(mapcar #’(λ (kond) (car kond)) list)) ҎԼΛ࣮ߦ͢ΔͱͲ͏ͳΔͷ͔ʁ => (mapcar #’(λ nil (car kond)) list)
10. So, WTF is a “Lisp” code walker!? Common Lispͷcode walker͸ “ίʔυ” Λద੾ʹ૸ࠪ͢ΔͨΊʹ
 CLͷsyntax formͷ஌ࣝ Λ͍࣋ͬͯΔʂ ্ه “஌ࣝ” ͸ ௨ৗͷݴޠͰ͍͏ "semantics" ʹ૬౰͢Δɺͱ͍͏ͷ͕
 ଥ౰Ͱ͠ΐ͏͕ɺCLͷ৔߹͸ reader macro, macrolet, symbol-macrolet ౳΋͋ΔͷͰɺparse΍syntaxͱ͔͍ͬͨݴ༿ͷ࢖͍ํʹ஫ҙ͕ཁΔ͔΋ ͠Ε·ͤΜ (͜͜Ͱ͸ਂೖΓ͠·ͤΜ)
 10
11. Review: what is a syntax form? Common Lispͷߏจنଇʹ͓͚Δform (ධՁର৅ͱͳΔSࣜ) atom: γϯϘϧɺϦςϥϧ(਺஋ɺจࣈɺ…) compound form: Χοίͷ͋ΔSࣜ function form: ؔ਺ద༻ — શҾ਺͕ؔ਺ݺग़͠લʹධՁ͞ΕΔ syntax form: ධՁ͞ΕΔҾ਺΍ධՁॱ͸ݸผʹఆٛ͞ΕΔ! special form: led by known 25 operators (CLHS 3.1.2.1.2.1) built-in macro form: ୔ࢁ͋Γͦ͏͚ͩͲ Ͳ͏͢Δͷ? user-defined macro form: ޙ͔Β௥Ճ͞ΕͨΒ Ͳ͏ͳΔͷ? 11
12. So, WTF is a “Lisp” code walker!? Common Lispͷcode walker͸ “ίʔυ” Λద੾ʹ૸ࠪ͢ΔͨΊʹ
 ANSI CLͷ25छྨͷsyntax operatorͷ஌ࣝ Λ͍࣋ͬͯΔʂ ϢʔβఆٛϚΫϩϑΥʔϜͷධՁنଇ͸Ͳ͏͢Δͷ?
 ૊ࠐΈϚΫϩϑΥʔϜͷධՁنଇ͸Ͳ͏͢Δͷ?
 PCL͸ɺCLtL1্ͷ಺෦DSL (ྲྀߦΓݴ༿ͰӠ͑͹ CLOS͔ΒCLtL1΁ͷ
 τϥϯεύΠϥ)Ͱɺ࠷Լ૚ʹ code walkerͷϨΠϠΛ͍࣋ͬͯͨʂ
 12 ˠ ࣮͸ ૊ࠐΈϚΫϩϑΥʔϜ΋expandͰ͖Δʂ 㱺 SBCLʹ΋ sb-walkerύοέʔδͱͯ͠࢒͓ͬͯΓɺ࢖͑·͢ʂ ˠ expand͢Ε͹ɺεϖγϟϧϑΥʔϜʴؔ਺ϑΥʔϜʹͳΔʂ
13. You could even expand a defclass form! ࣮͸ ૊ࠐΈϚΫϩϑΥʔϜ΋expandͰ͖Δʂ (cf. CLHS 3.1.2.1.2.2) 13
14. Functionality of the PCL code walker [1] subform૸ࠪͷࡍɺ໘౗ͳॲཧΛ୅ߦͯ͘͠ΕΔϑϨʔϜϫʔΫ special formͷ૸ࠪํ๏Λ஌͍ͬͯΔ ධՁنଇΛ஌͍ͬͯΔ (ධՁ͢΂͖subform ΍ ධՁͷॱং) ૸ࠪதʹ let౳ʹૺ۰͢Ε͹ɺࢀর͢Δ؀ڥΛద੾ʹߋ৽͢Δ macro formʹૺ۰͢Ε͹ɺͦͷ৔Ͱల։͔ͯ͠Β૸ࠪͯ͘͠ΕΔ macrolet ΍ symbol-macrolet ͱ͍ͬͨہॴϚΫϩʹ΋ରԠ Ϣʔβ͕ߦ͍͍ͨॲཧ͸ɺؔ਺Ҿ਺ͰࢦఆͰ͖Δ ૸ࠪதʹ෼͔ͬͨ͜ͱΛه࿥͢Ε͹ɺίʔυͷղੳॲཧʹͳΔ ֤subform͸ walkerͷฦΓ஋Ͱஔ׵͞ΕΔ(ม׵ॲཧ͕Մೳ) 14
15. Major APIs of SBCL code walker [1,14] [Macro] (define-walker-template name &optional template)
 syntax formͷධՁنଇΛڭ͑ࠐΉͨΊͷϚΫϩ
 
 15 
 
 
 
 [Function] (walk-form form &optional env walk-fn)
 ΤϯτϦAPI: walk-fnΛenvʹՃ͑ͯ walk-form-internalΛݺͿ
 
 
 
 
 
 
 ҎԼͷAPI΋export͞Ε͓ͯΓɺwalk-fnͷதͳͲͰ࢖͑Δ
 [Variable] *walk-form-expand-macros-p* (nil by default)
 macro formʹର͢ΔwalkerͷฦΓ஋Λల։݁Ռʹ͢Δ͔Ͳ͏͔?
 [Function] (var-lexical-p var-name env)
 [Function] (var-special-p var-name env)
 [Function] (var-globally-special-p var-name)
 [Function] (var-declaration decl-sym var-name env)
16. (define-walker-template name template) pre-defined walker Λ༩͑ΔςϯϓϨʔτϚΫϩ (ҎԼ͸هड़ྫ) 
 (define-walker-template if walk-if)
 (define-walker-template function (nil :call))
 (define-walker-template setq (nil :repeat (:set :eval)))
 (define-walker-template block (nil nil :repeat (:eval)))
 (define-walker-template catch (nil :eval :repeat (:eval))) if ʹ͸ɺઐ༻ؔ਺walk-ifΛ࢖͏(SBCLͰ͸໿൒਺͕͜ͷλΠϓ) function/setq/block/catchʹ͸ɺsubformͷධՁنଇΛද͢
 ϛχݴޠΠϯλϓϦλʹ༩͑ΔcontextγϯϘϧ͕ྻه͞Ε͍ͯΔ
 :repeat͸ɺ௚ޙͷྻ͕0ճҎ্܁Γฦ͞ΕΔ͜ͱΛද͠ɺ
 setqͷྫ͸ɺߏจ͕ (setq {place val}*)Ͱ͋Δ͜ͱΛ͍ࣔͯ͠Δ ˞ ݱߦͷίʔυͰ͸ɺsetqͷwalker-template͸ઐ༻ؔ਺ʹͳ͍ͬͯ·͢ 16
17. (walk-form-internal form context env) => form’ walkerͷmain driver
 ɾ ॴ༩ͷ؀ڥ(env: lexical؀ڥ΍walk-fn౳)Ͱ formΛwalk͢Δ
 ɾ context(:eval, :set, :call, etc)Λิॿ৘ใͱͯ͠࢖༻͍ͯ͠Δ 1. walk-fnͰformΛwalk͢Δ ( walk-form-internalͷuser hook )
 ୈ2ฦΓ஋͕non-nilͷ৔߹͸ 2.ʹਐ·ͣearly return͢Δ
 2. walk-fnͷୈ1ฦΓ஋Λର৅formͱͯ͠ walk͢Δ
 2a. setq for var ΍ symbol macro˞ ͷ৔߹
 2b. setf for compound formͷ৔߹
 2c. templateఆٛ͞Εͨform(i.e. special form)ͷ৔߹
 2d. macro form˞ͷ৔߹
 2e. function call formͷ৔߹ ˞ ϚΫϩ͸૸ࠪ௚લʹల։͞ΕΔ 17
18. walk-fnɿ3rd param for walk-form ࠶ܝ: walk-fn ͸ɺwalk-form-internal ͷ user hook ʹ૬౰͢Δ
 ɾ Ҿ਺ͷܗࣜͱҙຯ͚ͮ͸walk-form-internalͱಉ͡ʂ
 ɾ contextΛ࢖༻͢Δͷ΋ࣗ༝
 ɾ subformʹରͯ͠walk-form(-internal)Λ࠶ؼతʹݺΜͰΑ͍ walk-formݺग़͠Ͱলུ࣌͸ɺҾ਺ͷformΛͨͩฦ͚ͩ͢ͷؔ਺ ୈ1ฦΓ஋ʹՃ͑ͯɺୈ2ฦΓ஋Λฦ͢͜ͱ͕Ͱ͖Δ
 ɾ non-nilΛฦͨ͠৔߹ɺwalk-form-internal͕early return ͢Δ
 (sbclͰ͸macroexpand-all͕quasiquoteͷॲཧͰ࢖͍ͬͯΔ)
 ɾ nil Λฦͨ͠৔߹ɺୈ1ฦΓ஋͕Ҿ͖ଓ͖walk͞ΕΔ
 㱺 walk-fnͰม׵Λࢪͨ͠formΛwalkͤ͞Δ͜ͱ΋Ͱ͖Δ 18
19. Where to use a Lisp code walker? ҙ֎ʹ࢖͍Ͳ͜Ζ͕೉͍͠ʁ
 ɾ ߴϨϕϧˠ௿ϨϕϧͷશҬม׵(e.g. ී௨ͷίϯύΠϥ)ͷ৔߹ɺ
 ɹ macroexpand-all͔ͯ͠Βॲཧ͢Ε͹ࡁΉ͜ͱ΋ଟ͍ [3]
 ɾ code walkingແ͠ʹैདྷख๏ͷϚΫϩͰॻ͚ΔͳΒɺͦΕͰे෼ Ͱ͋Ε͹ɺҎԼͷΑ͏ͳॲཧͰ࢖͏͜ͱ͕༗ޮ… ͱߟ͑ΒΕΔ
 ɾ macroexpand-all ͦͷ΋ͷ
 ɾ code walkingແ͠ʹ͸ॻ͖ͮΒ͍macro expander
 ɾจ๏஌ࣝΛ༻͍ͨղੳ/ม׵
 ɾෳ਺ϨϕϧͷSࣜʹލΔղੳ/ม׵ (ؒʹsyntax form͕ڬ·Γ͏Δ)
 ɾ ιʔεϨϕϧͷ··௿Ϩϕϧ΁ม׵ͤͣʹߦ͍͍ͨղੳ/ม׵ 19
20. Examples
 ʙ macroexpand-all (1/2) ʙ ;; sbcl/src/contrib/sb-cltl2/macroexpand.lisp (note: a bit modified)
 (defun sb-cltl2:macroexpand-all (form &optional env)
 (flet ((walk-fn (subform context env2)
 (aif (and (eq context :eval)
 (listp subform) (symbolp (car subform))
 (get (car subform) :partial-macroexpander))
 ;; The partial expander, only for QUASIQUOTE as of Aug 2019,
 ;; must return T as its 2nd value to stop the walk
 (funcall it subform env2)
 subform))
 ;; just call WALK-FORM after binding *walk-…-macros-p* to t
 (let ((*WALK-FORM-EXPAND-MACROS-P* t ))
 (WALK-FORM form env #’walk-fn))))
 
 ɾ walk-fn͸ɺquasiquoteͷͱ͖Ҏ֎͸ ୯ʹsubformࣗ਎(ͱnil)Λฦ͢
 ɾ ଈͪ *walk-…-macros-p* Λม͚͑ͨͩͷ४ඪ४ಈ࡞͕ɺϚΫϩશల։ 20
21. Examples
 ʙ macroexpand-all (2/2) ʙ (defvar *my-repl*
 '(defun my-read-eval-print-loop (level)
 (with-simple-restart (abort "Exit command level ~D." level)
 (loop
 (with-simple-restart (abort "Return to command level ~D." level)
 (let ((form (prog2 (fresh-line) (read) (fresh-line))))
 (prin1 (eval form))))))))
 
 (defun mex-test (level &optional (form (fourth *my-repl*)))
 (ecase level
 (0 (macroexpand-1 form))
 (1 (macroexpand form))
 (2 (sb-cltl2:macroexpand-all form)))) 
 
 ɾ ࣮ࡍʹల։ͯ͠ΈΔͱɺ(locally (declare ...)) ͱ͔৭ʑͬ͘෇͍ͯͯɺ
 ɹ ίʔυղੳπʔϧͱ͔࡞Δͷ͸ɺͦΕ΄Ͳ؆୯Ͱ͸ͳ͍ͱ෼͔Δ 21
22. 
 (defmacro with-constant-folding (&body body &environment env)
 (WALK-FORM `(progn ,@body) env
 (lambda (form context env2)
 (if (and (eq context :eval) (consp form)
 (member (car form) '(+ - * /))
 (every #'constantp (cdr form)))
 (eval form)
 form))))
 
 ɾશҾ਺͕ఆ਺ͳ࢛ଇԋࢉϑΥʔϜΛ Τόͬͯ͠·͏ɺͱ͍͏΋ͷ
 ɾೖΕࢠʹͳͬͨ਺ࣜ΋৞ΈࠐΊΔΑ͏ʹɺ(cdr form)ͷ֤ཁૉʹ
 ࠶ؼతʹwalk-form-internalΛݺͼग़͢ྫ΋ڍ͛ΒΕ͍ͯΔ[1] 
 ɾeval ͸௨ྫϦϦʔε͢ΔΑ͏ͳίʔυͰ͸࢖Θͳ͍͠ɺSBCL্Ͱ͸
 (constantp ‘(* 3 4)) => T ͷΑ͏ͳ࠷దԽ͕ߦΘΕΔͷͰɺ͜ͷྫࣗମʹ
 ࣮༻ੑ͕͋ΔΘ͚Ͱ͸ͳ͍(͜͜ΒลͰ࠷దԽͯ͠ΔՄೳੑ΋͋Δ͕…) Examples
 ʙ constant folding at source code level [1] ʙ 22
23. 
 (defmacro anonymous-walker ( …… &environment env)
 (WALK-FORM <pattern-to-be-expanded>
 env
 (lambda (form context env2 ) …… )))
 
 
 ɾ લทͷΑ͏ʹɺWALK-FORMΛexpanderʹ༻͍Δ৔߹ͷύλʔϯ
 ɾ sb-cltl2:macroexpand-all ͷྫͰ͸ defun Λ࢖͍ͬͯΔ͕ɺ
 ࢖ΘΕͲ͜Ζ͕ະఆͰ caller contextͷimplicitͳ؀ڥʹΞΫηε͢Δ
 ৔߹͸ɺ্ྫͷΑ͏ʹdefmacroΛ࢖͑͹ɺmacro lambda listͷ
 &environment(CLHS 3.4.4) ܦ༝Ͱɺ؀ڥΛҾ͖ܧ͍ͰऔΓग़ͤΔ
 ˞ ͜Ε͸code walkerͱ͍͏ΑΓ defmacroͷ࿩Ͱ͸͋Γ·͢ Examples
 ʙ a general pattern on WALK-FORM based macroexpander ʙ 23
24. 
 (defvar *sample* '(let ((x (list Y)))
 (tagbody x
 (setq x (cons 'x X))
 (if (< (length X) 8) (go x)))
 (print X)))
 
 (defun replace-varref (&optional (form *sample*) env)
 (WALK-FORM form env 
 (lambda (form context env)
 (if (and (eq context :eval) form (symbolp form)
 ;; (VAR-LEXICAL-P form env)
 )
 "Foo" form)))) 
 
 ɾେจࣈ/੺ࣈͷม਺ࢀর͚ͩஔ׵͞ΕΔ (ࠨล஋, λά, ఆ਺͸ର৅֎)
 ɾίϝϯτ෦Λ༗ޮʹ͢Δͱɺࣗ༝ม਺Ͱ͋ΔY͸ஔ׵͞Εͳ͘ͳΔ Examples
 ʙ replacing every variable reference with "Foo" [1] ʙ 24
25. (defun same-var-p (var env1 env2) ; variable-same-p in Fig.10 [2]
 (eq (VAR-LEXICAL-P var env1) (VAR-LEXICAL-P var env2)))
 
 ;; lambda bodyͷ؀ڥͱ lambda formͷஔ͔Εͨ؀ڥͰɺม਺Λൺֱ͍ͯ͠Δ
 (defun aggregate (lmd-form lmd-env &aux (pool '()))
 (WALK-FORM lmd-f lmd-e
 (lambda (f c e) 
 (when (and (eq c :eval) f (symbolp f) (same-var-p f e lmd-env))
 (pushnew f pool))
 f))
 pool)
 (defun main (&optional (form '(let (x y z) (lambda (y) (foo x y z))) env))
 (WALK-FORM form env 
 (lambda (f c e) 
 (when (lambda-form-p f) (return-from main (aggregate f e)))
 f)))
 ;; (main) => (Z X) -- Ϋϩʔδϟม਺͕(ͦ͏Ͱͳ͍yΛআ͍ͯ)ूΊΒΕΔ Examples
 ʙ aggregating closure variables in a lambda form ʙ 25
26. 
 (defun binding-name (b) (if (consp b) (car b) b))
 (defun gen-subst (b &aux (n (binding-name b))) (cons n (gensym (string n)))
 (defun gen-substs (bindings) (mapcar #'gen-subst bindings))
 
 (defun rename-variables (form alist env) ; from Fig.10 [2]
 (WALK-FORM form env ; using "same-var-p" in the previous page
 (lambda (f c e &aux pair) 
 (if (and f (symbolp f) (setq pair (assoc f alist)) (same-var-p f e env))
 (cdr pair) f))))
 (defun alphatize-let-body (let-form &optional env)
 (destructuring-bind (let bindings . body) let-form
 (let ((substs (gen-substs bindings)))
 (rename-variables `(progn ,@body) substs env))))
 
 ɾεϖʔεͷؔ܎Ͱ [2] ʹ฿ͬͯ let bodyΛม׵͢ΔॲཧͷྫͷΈࣔͨ͠
 ɾ͜ͷํ๏͸ɺݸʑͷlet͝ͱͷalistʹconsult͕ཁΔͷͰΠέͯͳ͍͔΋
 (΋ͱ΋ͱ[2]͸ԶʑϧʔϓϚΫϩͷల։໨తͰɺ൚༻Ћม׵Ͱ͸ͳ͍) Examples
 ʙ alpha conversion (1/2) ʙ 26
27. 
 (alphatize '(with-open-file (st st) ; stream(bound-var) and string(free-var)
 (do ((l (read-line st) (read-line st nil 'eof)))
 ((eq l 'eof) "Reached end.")
 (format t "~&*** ~A~%" l))))
 => (with-open-file (#:st1 st)
 (do ((l (read-line #:st1) (read-line #:st1 nil 'eof)))
 ((eq l 'eof) "Reached end.")
 (format t "~&*** ~A~%" l)))
 
 ɾ࣮͸macroexpand-allͷޙ(͋Δ͍͸ηοτͰ)ͳΒЋ-conversion͸༰қ
 ɾ্هͷΑ͏ͳ source-to-source Ћ-conversionΛ͍͕ͨ͠ Մೳ͔?
 ɹɾ ࠓͷ SBCL code walkerʹ͸खΛೖΕͣʹɺwalk-fn ͷهड़͚ͩͰɺ
 ɹɹ ߴʑ2-pass͔3-passͰɺ͘Β͍ͷറΓͷ΋ͱͰ…
 ɹɾ ݸʑͷม਺ࢀরΛผΦϒδΣΫτͱͯ͠هԱҬΛઃఆ͍͕ͨ͠
 ɹɹ γϯϘϧͳͷͰͰ͖ͳ͍… 㱺 গ͠ߟ͕͑ͨૉ௚ʹ͸Ϝζͦ͏ Examples
 ʙ alpha conversion (2/2) ʙ 27
28. (let ((i 0)) (defmacro iter (&body body)
 (iter (with-gensyms (iter-block acc start)
 (while (< i 5)) `(block ,iter-block
 (incf i) (let ((,acc nil))
 (print acc) (macrolet ((while (test) …)
 (collect (* 4 i) :into acc))) (collect (thing) …)
 (tagbody ,start ,@body (go ,start)))))))
 
 ɾiterΛdefmacroͰɺwhile΍collectΛiterఆٛதͷmacroletͰॻ͖͍͕ͨɺ
 iterͷల։ܗ͕ collectͷల։ΑΓઌ͔ͭผʹఆ·ΔͷͰ೉͍͠[8]
 (iterͱcollectͷϚΫϩఆ͕ٛɺద੾ʹίϛϡχέʔτ͢Δඞཁ͕͋Δ͕ɺ
 ͦͷΑ͏ͳػߏ͸ݱߦͷdefmacro/macroletʹ͸ແ͍)
 
 ɾiterateϚΫϩ͸ɺࣗલcode walkerͰcollectઅͷࣄલղੳΛ͍ͯ͠ΔΒ͍͠
 㱺 Ұྫ͕ͩɺSBCL code walkerͰ΋ ࣍ทͷΑ͏ͳײ͡Ͱॻ͚ͦ͏ Examples
 ʙ transforming across multi-level forms (1/3) ʙ 28
29. (defmacro iter (… &environment env)
 (let ((analysis-info (make-analysis-info-object))
 (labels ((walk-fn (subform context subenv)
 (cond ((iter-form-p subform) (walk-iter-form …)) … (t subform))))
 ;; w-i-f͔Βw-c-fΛ(ඞཁͳΒ͹walk-form-internal౳Λܦͯ)ݺΜͰΑ͍
 (walk-iter-form (subform context subenv) …)
 (walk-collect-form (subform context subenv) …)
 (expand-iter-form () ...)) 
 (WALK-FORM form env #’walk-fn) ; analyze iter-macro form
 (expand-iter-form))) ; pass-2: expand using analysis-info 
 
 ɾWALK-FORMͷwalk-fnͷݺग़͠͸ɺmacroletͷbodyͱҧͬͯɺಛఆͷ
 formʹؔ࿈෇͚ΒΕ͍ͯͳ͍ (ͱ͍͏͔ શsubformʹରͯ͠ݺ͹ΕΔ) 
 㱺 iter formͷղੳͷதͰɺ಺ଆͷcollect formͷղੳΛͯ͠΋Α͍
 ɾ্ྫ͸2-pass͕ͩɺՄೳͳΒWALK-FORMͷ1-pass͚ͩͰల։ͯ͠΋Α͍ Examples
 ʙ transforming across multi-level forms (2/3) ʙ 29
30. 
 ɾҎԼͷwebจॻ[11]ʹ΋ɺલทͱಉ͡झࢫͷࢦఠ͕͞Ε͍ͯΔ 
 (code walkerͷํ͕ɺࣗ੹ͷ୅ঈʹmacroΑΓࣗ༝౓͕େ͖͍Θ͚Ͱ͢Ͷ)
 What code walkers can do that macro can’t
 
 The main difference between a macro and a code walker is coverage. 
 A macro can change only the parts of the program that call the
 macro (*1). Whereas a code walker can change the whole program.
 
 *1: One reason macros can’t walk over a whole program is the 
 runtime system doesn’t let them. …
 ɾSBCLʹ࢒͍ͬͯΔ walk-formͷݺग़Օॴʹ΋ɺmulti-levelͰ͸ͳ͍͚Ͳ
 ෳ਺ͷsubformղੳΛdispatch͍ͯ͠Δͱ͜Ζ(boot.lisp)͕͋Γ·͢ Examples
 ʙ transforming across multi-level forms (3/3) ʙ 30
31. 
 (macroexpand-1 
 '(-> "THREE" 
 string-downcase (char 0) char-code (complex (1+ $) (1- $))))
 
 => (LET (($ "THREE"))
 (SETQ $ (STRING-DOWNCASE $)) ; ؔ਺໊͚ͩͷͱ͖΍
 (SETQ $ (CHAR $ 0)) ; $͕ͳ͍ͱ͖͸ɺ
 (SETQ $ (CHAR-CODE $)) ; $Λ1st-argʹิ͍
 (SETQ $ (COMPLEX $ (1+ $) (1- $)))) ; $͕͋Δͱ͖͸ɺͦͷ·· 
 
 ɾSBCLͷϝϯςφʹΑΔSBCL walkerΛ࢖ͬͨॴҦ pipe operatorͷྫ
 ɾwalk-fn ޙͷ early return Λ࢖ͬͨྫʹ΋ͳ͍ͬͯΔ
 ɾଞʹ code walker Ͱ “defmacro/g! ͷχονͳ໰୊Λਖ਼͘͠ॲཧ͢Δ”
 ɹͱ͍ͬͨझࢫͷهࣄ[9]΋ॻ͍͓ͯΒΕ·͢ Examples
 ʙ writing pipe operator [10] ʙ 31
32. ;; (json:decode-json-from-string "{\"a\": 1, \"b\": {\"bb\": 2}, \"c\": 3}")
 ;; => ((:A . 1) (:B (:BB . 2)) (:C . 3))
 (defmacro with-json (json &body forms &environment env)
 (with-gensyms (decoded)
 `(let ((,decoded (json:decode-json-from-string ,json)))
 ,@(mapcar (lambda (form) (walk-with-json-body decoded form env)) forms))))
 
 (defun test-with-json () ; nesting with-json works properly
 (with-json "{\"a\": 1, \"b\": {\"bb\": 2}, \"c\": 3}"
 (let ((@c 999))
 ;; Here, @a and @b.bb take their json-value, i.e. 1 and 2,
 ;; however @c keeps a value 999 bound by let, not 3 in json description.
 (list @a @b.bb @c
 (with-json "{\"a\": 10, \"b\": {\"bb\": 20}, \"c\": 30}"
 (let ((@c 9990))
 (list @a @b.bb @c))))))) ; => (1 2 999 (10 20 9990))
 
 ɾjsonΞΫηε༻ͷsymbol macro (@quek͞Μ࡞) ͷಈ࡞ྫΛࣔͨ͠΋ͷ
 Examples
 ʙ a tiny symbol macro converted to an alist accessor [7] (1/2) ʙ 32
33. (defun free-varref-p (form ctxt env)
 (and (eq ctxt :eval) form (symbolp form) 
 (cond ((VAR-LEXICAL-P form env) nil)
 ((SB-WALKER::VARIABLE-SYMBOL-MACRO-P form env) nil)
 (t t))))
 
 ;; a macro expander intrinsic function using SB-WALKER:WALK-FORM
 (defun walk-with-json-body (decoded-str form env) 
 (WALK-FORM form env
 (lambda (subform ctxt subenv)
 (if (and (free-varref-p subform ctxt subenv)
 (@-prefixed-symbol-p subform))
 (symbol-to-assoc-form subform decoded-str) ; make an alist accessor
 subform))))
 
 ɾhu.duim.walkerͰ࡞ΒΕ͍ͯͨΦϦδφϧ Λ SBCL walkerʹҠ২ͨ͠΋ͷ
 ɾ࣮૷࢓༷΋΄΅ಉ౳ʹ͠ɺࠩ෼΍ࣗ໌Ͱͳ͍ίʔυͷΈ͍ࣔͯ͠Δ Examples
 ʙ a tiny symbol macro converted to an alist accessor [7] (2/2) ʙ 33
34. ࡞ΔϝϦοτ͕͋Δ͔ "ʁ" ͕ͩɺҎԼͷΑ͏ͳ΋ͷ΋ࢥ͍౰ͨΔ semantic checker෇͖embedded HTML/XML macro
 ✔ λάΛϚΫϩͰදͭͭ͠ɺҟͳΔϨϕϧͷλάؒʹ࣮ߦformΛ
 ڬΊͯ(ؚsyntax form)ɺλά੍ؒ໿(͜ͷλάͷԼʹ͋ͷλά͕
 ɹ 1ϲҎ্དྷΔ/དྷͳ͍౳)ͷݕࠪΛ࡞ΓࠐΉͷʹɺcode walkerΛ࢖͑Δ
 ☓ λάΛϚΫϩͰͳ͘ɺCL-WHOͷΑ͏ʹσʔλγϯϘϧʹ͢Δ
 ɹ ͜ͱ͕͋Δ͔΋(׶͑ͯෆ׬શͳλά࢓༷Λड͚༰Ε͍ͨ৔߹ͳͲ)
 ☓ ࡞ΓํΑΓ΋ɺଞʹॏࢹ͢΂͖͜ͱ͕ଟͦ͏ͳԠ༻Ͱ͢͠… source code cross reference tool
 ✔ ιʔεϨϕϧͷ৘ใղੳʹɺcode walkerΛ࢖͑Δ
 ☓ طʹslime͕XrefΛऔࠐΈࡁͳͷͰɺγϣϘ͍΋ͷ͸ཁΒͳ͛͞ Other Usages? 34
35. Ͳͷcode walker(PCL/SBCL, Arnesi, …)ʹ΋ɺ؀ڥΞΫηεAPI͕ඞཁ CLtL2ͷ؀ڥΞΫηεAPIͷΑ͏ͳ΋ͷ͸ɺANSI CL͸نఆ͍ͯ͠ͳ͍͕ ॲཧܥʹΑͬͯ͸ఏڙ͓ͯ͠Γɺportability layer࡞੒ͷࢼΈ[13]΋͋Δ SBCL͸ɺҎԼͷPCL༝དྷͷ؀ڥΞΫηεAPIΛɺαϙʔτॲཧܥ(10छྨ Ҏ্)Λࣗ਎༻ͷΈʹॖݮͯ͠ఏڙ͍ͯ͠Δɿ
 
 [Macro] (with-augmented-environment 
 ((new-env old-env &key functions macros) &body body) 
 ৽ͨͳϨϕϧͷ؀ڥΛnew-envʹଋറ͠ɺbodyΛ࣮ߦ͢Δ 
 [Function] (environment-macro env key-symbol)
 envதͷlexicalཁૉ(ϚΫϩɺม਺ɺwalk-fn౳)ͷྖҬ΁ͷΞΫηοα Implementation of SBCL code walker ˞ ໊લ(…-macro) ͕গ͓͔͍͕͠͠ɺެ։APIͰ΋ͳ͍͍͔ͤݩͷ··Ͱ͢ 35
36. Arnesi code walkerখ࢙ (as of Aug 2019)
 ɾMarco BaringerࢯͷArnesi utility(github্ͷอશ)ͷҰ෦Ͱ͋Δ
 ɹcode walker͕ factored out͞Εͯ cl-walkerʹͳͬͨ
 ɾcl-walker(gitlab)ͷAttila LendvaiࢯʹΑΔ࠷ऴߋ৽͸໿10೥લ
 ɾͦͷޙɺLendvaiࢯʹΑΔhu.dwim(darcs)ͷҰ෦ͷhu.dwim.walker
 ɹͱͯ͠ܧଓ͠ɺquicklispʹ͸2015೥ͷϦϦʔε͕ऩ࿥͞Ε͍ͯΔ
 ɾArnesi ˠ cl-walker ˠ hu.dwim.walkerͷؒɺ؀ڥAPI౳ʹߋ৽͋Γ
 (ͨͩ͠hu.dwim͸Զ༷ϚΫϩଟ༻Ͱશମ͕ີ݁߹ͱ͍͏໰୊͕͋Δ) ಛ௃ɿCLOS object node ͷ AST Λ࢖༻
 ɾParserͰASTੜ੒ ˠ User's methodsͰղੳ/ม׵ ˠ Sࣜʹٯม׵ 
 ɾBrowsable document͸͜ͷลΓ(Arnesi, hu.dwim.walker) Arnesi code walker and its descendants ݱ୅తʂ 36 (PCLΑΓ͸…)
37. @guicho271828͞Μͷهࣄ[8]ͷຊࢫ͸ɺcode wakingΛͯ͠ҟͳΔ
 ϨϕϧʹލΔม׵Λߦ͏ͨΊʹʮdefmacro(΍macrolet)͕Ҿ਺Ͱ continuationΛड͚औΕΔͷ͕ɺ͍͍Μ͡ΌͶʁʯͱ͍͏ఏҊ 㱺 [1] (Fig.12) ʹ΋ɺPCL code walkerͷwalk-fn͕ continuationΛ
 ɹͱΕΔΑ͏ʹ͢Δͱ৭ʑॊೈͳ੍ޚ͕Ͱ͖Δɺͱ͍͏ಉझࢫ
 ɹهࡌ͕͋ͬͨ (ͦͷޙͷਐల͸ެ։͞Εͳ͔ͬͨ໛༷) portable code walkerͷࢼΈͱͯ͠ɺmacroexpand-dammit[4]΍ agnostic lizard[12]͕͋Γɺ͍ͣΕ΋·ͣ͸macroexpand-all͔Β
 Ξϓϩʔν͍ͯ͠Δ͕ɺޙऀ͸callback΋ॻ͚ΔΒ͍͠ !?
 (ࠓճ͸ௐࠪෆ଍ʹ͖ͭɺ͜͜ʹ͸ه͍ͯ͠ͳ͍͕ɺίʔυ͸
 gitlab্Ͱެ։͞Ε͓ͯΓɺquicklispʹ΋ऩ࿥͞Ε͍ͯΔ༝) Miscellaneous notes 37
38. Personal View code walker͸ ίʔυ/ߏจ໦ͷ૸ࠪϥΠϒϥϦͱ͍͏͜ͱʹͳΔ͕ɺ
 Lispͷ৔߹ɺsyntax formͷ஌ࣝ(p.14)͚ͩͰ૸ࠪΛنଇԽͰ͖ɺ1st class 
 functionΛ࢖͑ͨͨΊɺPCLҎલ͔ΒϑϨʔϜϫʔΫԽ͕ਐΊΒΕͨҹ৅ ͕͢͞ʹ PCL walkerΑΓ͸ Arnesi walkerͷํ͕༏Ґ఺͕ଟ͍ؾ΋͢Δ
 ɾ ద੾ͳ(ந৅)ΫϥεͷαϒΫϥεԽͰɺⅰ)hook pointΛॊೈʹ૿΍ͤͯɺ
 ɹ ⅱ)ద੾ͳଐੑهԱҬΛ࡞Εͯɺⅲ)AST nodeͷจ๏஌ࣝΛ࢖͍қ͍
 ɾ CLOSͷڧྗͳmethod combinationΛ࢖͑Δ(before, after, around౳)
 ˞ ⅰɿݶఆతͳҐஔͷhook͚ͩͰΧελϜԽ͢Δύζϧʹ೰·ͳͯ͘Α͍
 ˞ ⅱ,ⅲɿྫ͑͹ɺม਺ͷoccurence͝ͱʹҟͳΔ৘ใΛ࣋ͨͤ΍͍͢౳ ໬΋ɺ༗ྗॲཧܥͷҰ෦ͱͯ͠ҡ࣋͞Ε͍ͯΔ͜ͱ͸େ͖͍
 (ͳͷͰɺຊࢿྉͰ͸ "ݹ͞͸֮ޛͰ" SBCL code walkerΛௐ΂ͯΈͨ) 38
39. Summary CL code walker͸ɺcode walking (ίʔυ͋Δ͍͸ߏจ໦Λ૸ࠪͭͭ͠
 ߦ͏ॲཧ)ʹඞཁͳɺspecial form΍macro formͷॲཧΛ୅ߦͯ͘͠ΕΔ CLͷίʔυղੳɺϚΫϩఆٛɺݴޠ֦ு/಺෦DSL࡞੒ͳͲʹར༻Ͱ͖Δ PCLʹىݯΛ࣋ͭ SBCLͷcode walkerʹ͍ͭͯ؆୯ʹ঺հͨ͠
 (ଞͷ ”·ͱ·ͬͨ΋ͷ” ͱͯ͠͸ɺArnesiͷwalkerͱͦͷࢠଙ͕͋Δ) ANSI CLࡦఆ࣌ʹɺ؀ڥΞΫηεAPIͷ࠾༻͕υϩοϓ͕ͨ͠ɺ
 ॲཧܥґଘ෦ͷແ͍CL code walkerΛॻ͘ࢼΈ΋͋Δ
 ˞walkerͷػೳ࢓༷ΛͲ͜·Ͱͱଊ͑Δ͔…ʹ͸෯͕͋Γͦ͏ ͋Δcode walkerΛ࢖͏ࡍɺಛఆͷCLͰಈ࡞͢Δ͔Ͳ͏͔ʹཁ஫ҙ͕ͩɺ
 prototypicalͳ࣮ݧ/ղੳ/։ൃͷπʔϧͱͯ͠ͳΒ͹࢖͍ॲ͸͋Γͦ͏ 39
40. Why are macros so important to Lisp programmers?
 Not merely for the syntactic convenience they provide, but because they are programs that manipulate programs, which has always been a central theme in the Lisp community. 
 If FORTRAN is the language that pushes numbers around, and C is the language that pushes characters and pointers around, then 
 Lisp is the language that pushes programs around.
 
 ˞ underline mine
 ˞ ਂொ͞Μͷϒϩά ʹ΋ ๜༁ͱڞʹࡌ͍ͬͯ·͢ Finally...
 ʙ A quote from “The Evolution of Lisp” ʙ 40
41. Questions? 41
42. [1] P. Curtis, (algorithms), Lisp Pointers, ACM SIGPLAN, 
 3(1):48-61, July 1989 (an intro. for PCL code walker)
 [2] B. v. Melle, Implementation of an Iteration Macro, Lisp Pointers, ACM SIGPLAN, 3(2-4):29-40, Apr 1990
 [3] R. C. Waters, Macroexpand-All: An example of a Simple Lisp Code Walker, Lisp Pointers, ACM SIGPLAN, 6(1):25-32, Jan 1993
 [4] J. Fremlin, Portable Common Lisp code walking with macroexpand-dammit, Jul 2009 (g000001, a Japanese intro.) [5] Hacker News, news.ycombinator.com/item?id=3607707, Feb 2012
 [6] Z. Beane, The tree-walkers of CL, CL Tips, Feb 2013
 [7] @quek, Using hu.dwim.walker (in Japanese), May 2013 References (1/2) 42
43. References (2/2) [8] @guicho271828, Common Lisp is still hard to satisfy (in Japanese), Feb 2014
 [9] C. Rhodes, naive vs proper code-walking, Sep 2014
 [10] C. Rhodes, code walking for pipe sequencing, Sep 2014
 (showing examples using sbcl code walker, so does [9])
 [11] What Code Walkers can do that macros can’t, Aug 2016
 [12] M. Raskin, Writing a best-effort portable code walker in Common Lisp, Euro Lisp Symp. (gitlab repo.), Apr 2017
 [13] A. Gutev, cl-environments (github repo.), Sep 2017
 [14] github.com/sbcl/sbcl/blob/master/src/pcl/walk.lisp
 [15] @nfunato, companion code for this presentation (gist) 43