How We Foster Rapid Development Workflows with a Performant and Extensible Build System @ TECHPULSE 2023

- Speaker: Colin Lin
- Event: http://techpulse.line.me/

我們的團隊始終致力於提供最優質的技術與服務，而隨著各式服務的數量、規模迅速成長，開發流程及系統架構等挑戰也接踵而來。快跟著我們一起探索，開發團隊是如何運用高性能且可擴展的構建系統來優化原有的開發流程，在提升 DX (Developer Experience) 的同時，也協助我們更能確保產品交付的效率及品質。

Transcript

2. Challenges Rapidly Growing Services 2 LIFF Web Server
3. Challenges Rapidly Growing Services 3 LIFF Web Server
4. Code Repository
5. State & Data Libraries Single-repo Monolithic 5 Project B Project C … … Project A Reusable Components … App Pros › Easy to Implement › Easy to Manage a Small Amount of Mini Projects Cons › Release All Projects Together › A Big Ball of Mud › Hard to Scale Repository
6. State & Data Libraries Single-repo Monolithic 6 Project B Project C … … Project A Reusable Components … App Pros › Easy to Implement › Easy to Manage a Small Amount of Mini Projects Cons › Release All Projects Together › A Big Ball of Mud › Hard to Scale Repository
7. Polyrepo 7 Pros › Precise Ownership › Less Number of Conflicts Cons › Cumbersome Code Sharing › Code Duplication › Costly Cross-repo Changes › Inconsistent Tooling App Repo Project A App App Repo Project B App App Repo … App Shared Repo
8. Polyrepo 8 Pros › Precise Ownership › Less Number of Conflicts Cons › Cumbersome Code Sharing › Code Duplication › Costly Cross-repo Changes › Inconsistent Tooling App Repo Project A App App Repo Project B App App Repo … App Shared Repo
9. Monorepo 9 Pros › Atomic Commits › Consistent Tooling › Enforce Project Boundaries › … Cons › More Sophisticated CI Setup › Need to Think More About Large-scale Changes Repository Project A App Project B App Project C App … App WebView Manager Reusable Components Plugins API Services Recoil Request … MSW Utils
10. Monorepo 10 Pros › Atomic Commits › Consistent Tooling › Enforce Project Boundaries › … Cons › More Sophisticated CI Setup › Need to Think More About Large-scale Changes Repository Project A App Project B App Project C App … App WebView Manager Reusable Components Plugins API Services Recoil Request … MSW Utils
11. In the Course of Evolution Single-repo Monolithic 11 Polyrepo Monorepo
12. Not Just “Code Repository”
13. Solutions for Different Needs 13
14. Nx Next Generation Build System with First Class Monorepo Support Plugable Tooling Task Orchestration Workspace Analysis Detecting Affected Computation Caching 14
15. Computation Caching Never Build or Test The Same Thing Twice 15 CLI Command Flags > nx run seller-center:build --stage=alpha Relevant Source App Lib1 Lib2 Runtime External Dependencies React ProseMirror Global Config Hasher Computation Hash
16. Computation Caching Never Build or Test The Same Thing Twice 16 CLI Command Flags > nx run seller-center:build --stage=alpha Relevant Source App Lib1 Lib2 Runtime External Dependencies React ProseMirror Global Config Hasher Computation Hash
17. Computation Caching Store and Replay File and Process Output 17 NX Running target build for project seller-center and 3 task(s) it depends on > nx run services:build Compiling TypeScript files for project "services"... Done compiling TypeScript files for project "services". > nx run recoil-request:build Compiling TypeScript files for project "recoil-request"... Done compiling TypeScript files for project "recoil-request". > nx run shared:build Compiling TypeScript files for project "shared"... Done compiling TypeScript files for project "shared". > nx run seller-center:build:production ... NX Successfully ran target build for project seller-center and 3 task(s) it depends on < Command > Processing Command Result Cached fi rst time (no cache) second time (cached)
18. Computation Caching Store and Replay File and Process Output 18 < Command > Processing Command Result Cached fi rst time (no cache) second time (cached) NX Running target build for project seller-center and 3 task(s) it depends on > nx run seller-center:build:production ... NX Successfully ran target build for project seller-center Nx read the output from the cache instead of running the command for 3 out of 4 tasks.
19. Task Orchestration Analyze Run Tasks app lIb lib app-e2e lIb Project Graph app-e2e:test Task Graph app:serve app:build app:test lib:test lib:test lib:test ex. Run All Tests Source Source Code Installed Dependencies Cached Graph
20. Task Orchestration Run Tasks in the Correct Order and in Parallel 20 0 1 2 3 4 app:build app:serve app-e2e:test app:test lib:test lib:test lib:test lib:test lib:test …. …. …. Parallel Processes Execution Time
21. Detecting Affected Realistic Project Graph 21 Git Branches Commit #1 Commit #2 Commit #3 Commit #4 Commit #5 Commit #6 Develop Branch Feature Branch app lIb lib lIb lib lIb lIb app app app Changed
22. Detecting Affected Only Run Build/Test for Affected Projects 22 app lIb lib lIb lib lIb lIb app app app Changed Change Detection Commit #1 Commit #2 Commit #3 Commit #4 Commit #5 Commit #6 Develop Branch Feature Branch
23. Detecting Affected Continuous Integration 23 develop feature checkout commits pull request automated build & test commits approve merge Unit Test E2E Test SonarQube Unit Test Checkout Setup Environment Install Dependencies Test Affected Restore Cache Detect Affected Set SHAs CI Work fl ows
24. Detecting Affected Continuous Integration 24 develop feature checkout commits pull request automated build & test commits approve merge Unit Test E2E Test SonarQube Unit Test Checkout Setup Environment Install Dependencies Test Affected Restore Cache Detect Affected Set SHAs CI Work fl ows
25. Workspace Analysis Graph Visualization 25 › High Level View of the Code Architecture › Search, Filter, Hide, and Focus the Nodes in the Graph › Detailed Information of Each Dependency Link › Task Graph Support
26. Workspace Analysis Circular Dependency 26 Consequences › Unexpected Side Effects › Make Nx’s Affected Algorithm Less Effective Strategies to Resolve › Move Pieces of Code from One Project to the Other › Move Code that Both Libraries Depend on into a New Library › Combine Projects into One Library
27. Workspace Analysis Project Boundaries 27 { "@nrwl/nx/enforce-module-boundaries": [ "error", { "enforceBuildableLibDependency": true, "allowCircularSelfDependency": true, "allow": [], "depConstraints": [ { "sourceTag": "scope:app", "onlyDependOnLibsWithTags": ["scope:shared"] }, { "sourceTag": "scope:shared", "onlyDependOnLibsWithTags": ["scope:shared"] }, { "sourceTag": "scope:plugin", "onlyDependOnLibsWithTags": ["scope:plugin", "scope:shared"] } ] } ] } Scope Tags app1 routing recoil app1-e2e playwright app app shared shared plugin app2 app implicit implicit .eslintrc.json
28. Plugable Tooling Executors 28 › Executing Similar Tasks on Unrelated Projects › Leveraging this Consistency to Run the Specific Target Across Multiple Projects › Metadata to Define the Available Options › Composable Executors { "name": "app1", "$schema": "../../node_modules/nx/schemas/project-schema.json", "sourceRoot": "apps/app1/src", "projectType": "application", "targets": { "build": { "executor": "@mono-web/webpack:build", ... }, "serve": { "executor": “@mono-web/webpack:serve", ... }, "lint": { "executor": "@nrwl/linter:eslint", ... }, "test": { "executor": “@nrwl/jest:jest", ... }, "analyze": { "executor": "@mono-web/webpack:analyze", ... } }, "tags": ["scope:app"] } project.json
29. Plugable Tooling Generators 29 › Generating, Updating and Structuring Code in a Certain Way › Migration Generator › AST Manipulation › EJS Syntax for Control flow in the Template › Composable Generators Generated Files src/index.html src/main.ts src/components/ src/pages/ tsconfig.json jest.config.ts … Template Files src/index.html src/main.ts__tmpl__ src/components/ src/pages/ tsconfig.json__tmpl__ jest.config.ts__tmpl__ … Generate create/update fi les create/update project con fi g create/update TS con fi g compose other generators update package dependencies format fi les …
30. Plugable Tooling Create Our Own Plugin: React Application Generator 30 Template Files Normalize Options Create Application Files Workspace Configurations Create Project Configurations Update Workspace Configurations Compose Other Generators Generator Entry Linting Files Format Files …
31. Plugable Tooling Create Our Own Plugin: Playwright Executor 31 Normalize Options Start Target Server Chokidar Watcher Playwright CLI Executor Entry E2E Test Codegen Trace Viewer
32. What’s Next Distributed Computation Caching 32 CI Agent > build, test, lint, … Remote Cache Developer > build, test, lint, … Developer > build, test, lint, … Developer > build, test, lint, …
33. Thank you 33