Master Tooling by Hacking the Angular Compiler

Nowadays, we can hardly imagine the web without tooling. There are tools for code generation, transformations, and, most importantly, build tools. Build tools play an important role in modern web development. They make frameworks like Angular, React, or Vue more approachable and simplify the entire process from start to end. For example, a build tool manipulates our code in powerful ways; it downlevels our code, makes it smaller, and creates a production-grade bundle that is ready for deployment. But how does that work? To understand this, we will specifically look at Angular's build system to implement our own custom template syntax. Along the way, we explore tools such as Webpack, the Angular Compiler, and, most importantly, an Abstract Syntax Tree (AST). ASTs are the basic building blocks of most web tooling that empowers tools like Angular Schematics, Prettier, Babel, or TSLint to do their job. This talk should give you an insight into how Angular builds our applications, and act as an intro to Abstract Syntax Trees, how they work, and what we can do with them.

Special thanks to Kwinten Pisman (https://twitter.com/KwintenP) for helping me putting this talk together.

Links:
https://github.com/typebytes/ngx-template-streams
https://astexplorer.net/
https://github.com/phenomnomnominal/tsquery
https://github.com/urish/tsquery-playground

Transcript

1. Master Tooling by Hacking the Angular Compiler by Dominic Elm
3. AST @elmd_
4. Abstract Syntax Tree @elmd_
5. @elmd_ NGULAR CLI
6. @elmd_ NGULAR CLI Schematics
7. @elmd_ ESLint NGULAR CLI Schematics
8. @elmd_ ESLint NGULAR CLI Schematics
9. @elmd_ ESLint NGULAR CLI Schematics
10. @elmd_ AST ESLint NGULAR CLI Schematics
13. @elmd_ Dominic Elm thoughtram https://thoughtram.io StackBlitz https://stackblitz.com Google Developer Expert https://bit.ly/2nAqRrZ
14. Angular Checklist @elmd_
15. @elmd_ @KwintenP Kwinten Pisman Thanks!
16. ngx-template-streams? @elmd_
17. Angular library that tries to embrace reactivity and supercharges templates with Observables. @elmd_ “
18. <button (click)=“clicks$.next($event)”>Click Me</button> @Component({...}) export class AppComponent implements OnInit { clicks$ = new Subject(); ngOnInit() { // we can either manually subscribe // or use the async pipe this.clicks$.subscribe(console.log); } }
19. <button (click)=“clicks$.next($event)”>Click Me</button> @Component({...}) export class AppComponent implements OnInit { clicks$ = new Subject(); ngOnInit() { // we can either manually subscribe // or use the async pipe this.clicks$.subscribe(console.log); } }
20. <button (click)=“clicks$.next($event)”>Click Me</button> @Component({...}) export class AppComponent implements OnInit { clicks$ = new Subject(); ngOnInit() { // we can either manually subscribe // or use the async pipe this.clicks$.subscribe(console.log); } }
22. <button (click)=“clicks$.next($event)”>Click Me</button> @Component({...}) export class AppComponent implements OnInit { clicks$ = new Subject(); ngOnInit() { // we can either manually subscribe // or use the async pipe this.clicks$.subscribe(console.log); } }
23. <button (*click)=“clicks$”>Click Me</button> @Component({...}) export class AppComponent implements OnInit { @ObservableEvent() clicks$: Observable<any>; ngOnInit() { // we can either manually subscribe // or use the async pipe this.clicks$.subscribe(console.log); } }
24. <button (*click)=“clicks$”>Click Me</button> @Component({...}) export class AppComponent implements OnInit { @ObservableEvent() clicks$: Observable<any>; ngOnInit() { // we can either manually subscribe // or use the async pipe this.clicks$.subscribe(console.log); } }
25. The Angular Build System @elmd_
26. @elmd_ Angular CLI
27. Webpack @elmd_
28. Webpack @elmd_ Ahead of Time Compiler for the Browser
29. Webpack @elmd_ Ahead of Time Compiler for the Browser Common Misconception: It’s not only for JavaScript ⚠
30. Webpack *.bundle.js *.bundle.js main.bundle.js App @elmd_ Ahead of Time Compiler for the Browser Common Misconception: It’s not only for JavaScript ⚠
31. Webpack @elmd_ webpack.config.js
32. Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin AngularCompilerPlugin @elmd_ webpack.config.js
33. Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Angular Code AngularCompilerPlugin @elmd_ webpack.config.js
34. Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Angular Code <waits for> AngularCompilerPlugin @elmd_ webpack.config.js
35. Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin @elmd_ webpack.config.js
36. Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Angular Code <waits for> Resolve Dependencies, Bundle Bundle AngularCompilerPlugin @elmd_ webpack.config.js
37. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin @elmd_ WebpackCompilerHost ts.CompilerHost
38. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin ngProgram AOT ts.Program JIT @elmd_ WebpackCompilerHost ts.CompilerHost
39. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin Analysis AOT ngProgram AOT ts.Program JIT @elmd_ WebpackCompilerHost ts.CompilerHost
40. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin Resolve AOT Analysis AOT ngProgram AOT ts.Program JIT @elmd_ WebpackCompilerHost ts.CompilerHost
41. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin Resolve AOT Analysis AOT ngProgram AOT ts.Program JIT @elmd_ WebpackCompilerHost ts.CompilerHost Type Checking AOT
42. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin Resolve AOT Analysis AOT ngProgram AOT ts.Program JIT Transformer @elmd_ WebpackCompilerHost ts.CompilerHost Type Checking AOT Emit
43. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin Angular Code Resolve AOT Analysis AOT ngProgram AOT ts.Program JIT Transformer @elmd_ WebpackCompilerHost ts.CompilerHost Type Checking AOT Emit
44. Cool. But what’s next? @elmd_
45. Loader .html @elmd_
46. <button (click)=“"__clicks$.next()”> Angular rocks! "</button> Loader .html <button (*click)=“clicks$”> Angular rocks! "</button> @elmd_
47. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Angular Code <waits for> Resolve Dependencies, Bundle Bundle AngularCompilerPlugin @elmd_
48. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Angular Code <waits for> Resolve Dependencies, Bundle Bundle AngularCompilerPlugin HTML Loader Custom Loader @elmd_
49. Builders @elmd_
50. ngx-build-plus @elmd_ Thanks Manfred Steyer
51. $ @elmd_
52. $ ng add ngx-build-plus $ @elmd_
53. $ ng add ngx-build-plus touch extra-webpack-config.js $ $ @elmd_
54. module.exports = { module: { rules: [ { test: /\.html$/, use: [ { loader: path.resolve('path/to/custom/html/loader') }, { loader: 'raw-loader' "// import file as a string } ] } ] } }; @elmd_
56. $ ng add ngx-build-plus touch extra-webpack-config.js $ $ @elmd_
57. $ ng add ngx-build-plus touch extra-webpack-config.js $ touch our-plugin.js $ $ @elmd_
58. export default { pre() { "// before build }, post() { "// after build }, config(config: webpack.Configuration) { "// after webpack config was built by the CLI } }; @elmd_
59. export default { pre() { "// before build }, post() { "// after build }, config(config: webpack.Configuration) { const acp = findAngularCompilerPlugin(config) as AngularCompilerPlugin; if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); @elmd_
60. export default { pre() { "// before build }, post() { "// after build }, config(config: webpack.Configuration) { const acp = findAngularCompilerPlugin(config) as AngularCompilerPlugin; if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); @elmd_
61. if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); config.plugins.push(newCompilerPlugin); return config; } }; @elmd_
62. if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); config.plugins.push(newCompilerPlugin); return config; } }; @elmd_
63. if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); config.plugins.push(newCompilerPlugin); return config; } }; @elmd_
64. if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); config.plugins.push(newCompilerPlugin); return config; } }; @elmd_
65. if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); config.plugins.push(newCompilerPlugin); return config; } }; @elmd_
66. Inline templates? @elmd_
67. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin Angular Code Resolve AOT Analysis AOT ngProgram AOT ts.Program JIT Transformer @elmd_ WebpackCompilerHost ts.CompilerHost Type Checking AOT Emit
68. webpack.config.js Webpack Loader Loader Loader ngc <register> <register> Plugin Plugin AngularCompilerPlugin <emits> Generated Angular Code <waits for> Resolve Dependencies, Bundle AngularCompilerPlugin Angular Code Resolve AOT Analysis AOT ngProgram AOT ts.Program JIT Transformer @elmd_ WebpackCompilerHost ts.CompilerHost Type Checking AOT Emit Transformer Custom TS Transformer
69. TypeScript Transformers @elmd_
70. export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { "// change sourceFile (AST) }; } @elmd_
71. let a = 1; @elmd_ VariableStatement VariableDeclaration VariableDeclarationList NumericalLiteral text = “1” text = “a” Identifier SourceFile
72. let a = 1; SourceFile VariableStatement VariableDeclaration VariableDeclarationList text = “a” Identifier @elmd_ NumericalLiteral text = “1”
73. let a = 1; SourceFile VariableStatement VariableDeclaration VariableDeclarationList NumericalLiteral text = “1” @elmd_ text = “a” Identifier
74. let a = 1; SourceFile VariableStatement VariableDeclarationList NumericalLiteral text = “1” @elmd_ text = “a” Identifier VariableDeclaration
75. let a = 1; SourceFile VariableStatement VariableDeclarationList NumericalLiteral text = “1” @elmd_ text = “a” Identifier VariableDeclaration
76. let a = 1; SourceFile VariableStatement VariableDeclarationList NumericalLiteral text = “1” @elmd_ text = “a” Identifier VariableDeclaration
77. SourceFile VariableStatement VariableDeclarationList NumericalLiteral text = “1” @elmd_ text = “a” Identifier VariableDeclaration let a = 1;
78. @elmd_
79. export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { /** * Todos: * 1. Find `template` property within @Component decorator * 2. Apply HTML transformation * 3. Update AST node * 4. Return updated SourceFile "*/ }; } @elmd_
80. export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; } @elmd_
81. export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; } function findInlineTemplate(sourceFile: ts.SourceFile) { return sourceFile.statements[1].decorators[0].expression .arguments[0].properties[0].initializer.text; } @elmd_
82. export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; } function findInlineTemplate(sourceFile: ts.SourceFile) { return sourceFile.statements[1].decorators[0].expression .arguments[0].properties[0].initializer.text; } @elmd_
83. TSQuery @elmd_ Thanks Craig Spence CSS-like Selectors for the TS AST
84. @elmd_ export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; } function findInlineTemplate(sourceFile: ts.SourceFile) { return tsquery.query(sourceFile, INLINE_TEMPLATE_QUERY); }
85. @elmd_ export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; }
86. @elmd_ export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; } function updateInlineTemplate( nodes: Array<ts.Node>, sourceFile: ts.SourceFile, context: ts.TransformationContext) { const visitor: ts.Visitor = (node: ts.Node) "=> { "// ast walker function return ts.visitEachChild(node, visitor, context); }; return ts.visitNode(sourceFile, visitor); }
87. @elmd_ export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; } function updateInlineTemplate( nodes: Array<ts.Node>, sourceFile: ts.SourceFile, context: ts.TransformationContext) { const visitor: ts.Visitor = (node: ts.Node) "=> { "// ast walker function return ts.visitEachChild(node, visitor, context); }; return ts.visitNode(sourceFile, visitor); }
88. @elmd_ export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; } function updateInlineTemplate( nodes: Array<ts.Node>, sourceFile: ts.SourceFile, context: ts.TransformationContext) { const visitor: ts.Visitor = (node: ts.Node) "=> { "// ast walker function return ts.visitEachChild(node, visitor, context); }; return ts.visitNode(sourceFile, visitor); }
89. @elmd_ export function transformer(context: ts.TransformationContext) { return (sourceFile: ts.SourceFile) "=> { const template = findInlineTemplate(sourceFile); return updateInlineTemplate(template, sourceFile, context); }; } function updateInlineTemplate( nodes: Array<ts.Node>, sourceFile: ts.SourceFile, context: ts.TransformationContext) { const visitor: ts.Visitor = (node: ts.Node) "=> { "// ast walker function return ts.visitEachChild(node, visitor, context); }; return ts.visitNode(sourceFile, visitor); }
90. @elmd_ function updateInlineTemplate( nodes: Array<ts.Node>, sourceFile: ts.SourceFile, context: ts.TransformationContext) { const visitor: ts.Visitor = (node: ts.Node) "=> { "// ast walker function return ts.visitEachChild(node, visitor, context); }; return ts.visitNode(sourceFile, visitor); }
91. function updateInlineTemplate( nodes: Array<ts.Node>, sourceFile: ts.SourceFile, context: ts.TransformationContext) { const visitor: ts.Visitor = (node: ts.Node) "=> { if (nodes.includes(node)) { const prop = node as ts.PropertyAssignment; const currentTpl = prop.initializer.getFullText(); const inlineTpl = ts.createNoSubstitutionTemplateLiteral( updateDirectives(currentTpl, true) ); return ts.updatePropertyAssignment(prop, prop.name, inlineTpl); } return ts.visitEachChild(node, visitor, context); }; return ts.visitNode(sourceFile, visitor); @elmd_
92. if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); config.plugins.push(newCompilerPlugin); return config; } }; @elmd_
93. if (!acp) { throw new Error('AngularCompilerPlugin not found'); } const options: AngularCompilerPluginOptions = { ""...acp.options, directTemplateLoading: false "// ""<-- IMPORTANT }; config.plugins = removeCompilerPlugin(config.plugins, acp); const newCompilerPlugin = new AngularCompilerPlugin(options); acp._transformers = [inlineTplTransformer, ""...acp._transformers]; config.plugins.push(newCompilerPlugin); return config; } }; @elmd_
95. https://github.com/typebytes/ngx-template-streams @elmd_
96. Key Takeaways @elmd_
97. Key Takeaways @elmd_ The Angular Compiler doesn’t need to be scary
98. Key Takeaways @elmd_ TypeScript Transformers can be used to transform our own source code The Angular Compiler doesn’t need to be scary
99. Key Takeaways @elmd_ TypeScript Transformers can be used to transform our own source code ASTs are powerful and the foundation of many tools The Angular Compiler doesn’t need to be scary
100. Key Takeaways @elmd_ TypeScript Transformers can be used to transform our own source code ASTs are powerful and the foundation of many tools Tooling and code automation can be so much fun and save lots of time The Angular Compiler doesn’t need to be scary
101. Thank You NG-BE! @elmd_