How We Foster Rapid Development Workflows with a Performant and Extensible Build System @ TECHPULSE 2023

- Speaker: Colin Lin
- Event: http://techpulse.line.me/

我們的團隊始終致力於提供最優質的技術與服務，而隨著各式服務的數量、規模迅速成長，開發流程及系統架構等挑戰也接踵而來。快跟著我們一起探索，開發團隊是如何運用高性能且可擴展的構建系統來優化原有的開發流程，在提升 DX (Developer Experience) 的同時，也協助我們更能確保產品交付的效率及品質。

Transcript

1. View Slide

2. Challenges
   Rapidly Growing Services
   2
   LIFF
   Web Server
   

   View Slide

3. Challenges
   Rapidly Growing Services
   3
   LIFF
   Web Server
   

   View Slide

4. Code Repository
   

   View Slide

5. State & Data
   Libraries
   Single-repo Monolithic
   5
   Project B
   Project C
   …
   …
   Project A
   Reusable
   Components
   …
   App
   Pros
   › Easy to Implement
   › Easy to Manage a Small Amount of
   Mini Projects
   Cons
   › Release All Projects Together
   › A Big Ball of Mud
   › Hard to Scale
   Repository
   

   View Slide

6. State & Data
   Libraries
   Single-repo Monolithic
   6
   Project B
   Project C
   …
   …
   Project A
   Reusable
   Components
   …
   App
   Pros
   › Easy to Implement
   › Easy to Manage a Small Amount of
   Mini Projects
   Cons
   › Release All Projects Together
   › A Big Ball of Mud
   › Hard to Scale
   Repository
   

   View Slide

7. Polyrepo
   7
   Pros
   › Precise Ownership
   › Less Number of Conflicts
   Cons
   › Cumbersome Code Sharing
   › Code Duplication
   › Costly Cross-repo Changes
   › Inconsistent Tooling
   App Repo
   Project A
   App
   App Repo
   Project B
   App
   App Repo
   …
   App
   Shared Repo
   

   View Slide

8. Polyrepo
   8
   Pros
   › Precise Ownership
   › Less Number of Conflicts
   Cons
   › Cumbersome Code Sharing
   › Code Duplication
   › Costly Cross-repo Changes
   › Inconsistent Tooling
   App Repo
   Project A
   App
   App Repo
   Project B
   App
   App Repo
   …
   App
   Shared Repo
   

   View Slide

9. Monorepo
   9
   Pros
   › Atomic Commits
   › Consistent Tooling
   › Enforce Project Boundaries
   › …
   Cons
   › More Sophisticated CI Setup
   › Need to Think More About Large-scale
   Changes
   Repository
   Project A
   App
   Project B
   App
   Project C
   App
   …
   App
   WebView
   Manager
   Reusable
   Components
   Plugins
   API
   Services
   Recoil
   Request
   …
   MSW Utils
   

   View Slide

10. Monorepo
    10
    Pros
    › Atomic Commits
    › Consistent Tooling
    › Enforce Project Boundaries
    › …
    Cons
    › More Sophisticated CI Setup
    › Need to Think More About Large-scale
    Changes
    Repository
    Project A
    App
    Project B
    App
    Project C
    App
    …
    App
    WebView
    Manager
    Reusable
    Components
    Plugins
    API
    Services
    Recoil
    Request
    …
    MSW Utils
    

    View Slide

11. In the Course of Evolution
    Single-repo Monolithic
    11
    Polyrepo Monorepo
    

    View Slide

12. Not Just “Code Repository”
    

    View Slide

13. Solutions for Different Needs
    13
    

    View Slide

14. Nx
    Next Generation Build System with First Class Monorepo Support
    Plugable Tooling
    Task Orchestration
    Workspace Analysis
    Detecting Affected
    Computation Caching
    14
    

    View Slide

15. Computation Caching
    Never Build or Test The Same Thing Twice
    15
    CLI Command Flags
    > nx run seller-center:build --stage=alpha
    Relevant Source
    App
    Lib1
    Lib2
    Runtime
    External
    Dependencies
    React
    ProseMirror
    Global Config
    Hasher Computation Hash
    

    View Slide

16. Computation Caching
    Never Build or Test The Same Thing Twice
    16
    CLI Command Flags
    > nx run seller-center:build --stage=alpha
    Relevant Source
    App
    Lib1
    Lib2
    Runtime
    External
    Dependencies
    React
    ProseMirror
    Global Config
    Hasher Computation Hash
    

    View Slide

17. Computation Caching
    Store and Replay File and Process Output
    17
    NX Running target build for project seller-center and 3
    task(s) it depends on
    > nx run services:build
    Compiling TypeScript files for project "services"...
    Done compiling TypeScript files for project "services".
    > nx run recoil-request:build
    Compiling TypeScript files for project "recoil-request"...
    Done compiling TypeScript files for project "recoil-request".
    > nx run shared:build
    Compiling TypeScript files for project "shared"...
    Done compiling TypeScript files for project "shared".
    > nx run seller-center:build:production
    ...
    NX Successfully ran target build for project seller-center and
    3 task(s) it depends on
    < Command >
    Processing
    Command
    Result Cached
    fi
    rst time
    (no cache)
    second time
    (cached)
    

    View Slide

18. Computation Caching
    Store and Replay File and Process Output
    18
    < Command >
    Processing
    Command
    Result Cached
    fi
    rst time
    (no cache)
    second time
    (cached)
    NX Running target build for project seller-center and 3
    task(s) it depends on
    > nx run seller-center:build:production
    ...
    NX Successfully ran target build for project seller-center
    Nx read the output from the cache instead of running the
    command for 3 out of 4 tasks.
    

    View Slide

19. Task Orchestration
    Analyze Run Tasks
    app
    lIb
    lib
    app-e2e
    lIb
    Project Graph
    app-e2e:test
    Task Graph
    app:serve
    app:build
    app:test
    lib:test
    lib:test
    lib:test
    ex. Run All Tests
    Source
    Source Code
    Installed
    Dependencies
    Cached Graph
    

    View Slide

20. Task Orchestration
    Run Tasks in the Correct Order and in Parallel
    20
    0 1 2 3 4
    app:build app:serve app-e2e:test
    app:test
    lib:test lib:test
    lib:test lib:test
    lib:test
    ….
    ….
    ….
    Parallel Processes
    Execution Time
    

    View Slide

21. Detecting Affected
    Realistic Project Graph
    21
    Git Branches
    Commit
    #1
    Commit
    #2
    Commit
    #3
    Commit
    #4
    Commit
    #5
    Commit
    #6
    Develop
    Branch
    Feature
    Branch
    app
    lIb
    lib lIb
    lib lIb lIb
    app app
    app
    Changed
    

    View Slide

22. Detecting Affected
    Only Run Build/Test for Affected Projects
    22
    app
    lIb
    lib lIb
    lib lIb lIb
    app app
    app
    Changed
    Change Detection
    Commit
    #1
    Commit
    #2
    Commit
    #3
    Commit
    #4
    Commit
    #5
    Commit
    #6
    Develop
    Branch
    Feature
    Branch
    

    View Slide

23. Detecting Affected
    Continuous Integration
    23
    develop feature
    checkout
    commits
    pull request
    automated
    build & test
    commits
    approve
    merge
    Unit Test
    E2E Test
    SonarQube
    Unit Test
    Checkout
    Setup
    Environment
    Install
    Dependencies
    Test
    Affected
    Restore
    Cache
    Detect
    Affected
    Set SHAs
    CI Work
    fl
    ows
    

    View Slide

24. Detecting Affected
    Continuous Integration
    24
    develop feature
    checkout
    commits
    pull request
    automated
    build & test
    commits
    approve
    merge
    Unit Test
    E2E Test
    SonarQube
    Unit Test
    Checkout
    Setup
    Environment
    Install
    Dependencies
    Test
    Affected
    Restore
    Cache
    Detect
    Affected
    Set SHAs
    CI Work
    fl
    ows
    

    View Slide

25. Workspace Analysis
    Graph Visualization
    25
    › High Level View of the Code
    Architecture
    › Search, Filter, Hide, and Focus
    the Nodes in the Graph
    › Detailed Information of Each
    Dependency Link
    › Task Graph Support
    

    View Slide

26. Workspace Analysis
    Circular Dependency
    26
    Consequences
    › Unexpected Side Effects
    › Make Nx’s Affected Algorithm Less
    Effective
    Strategies to Resolve
    › Move Pieces of Code from One
    Project to the Other
    › Move Code that Both Libraries
    Depend on into a New Library
    › Combine Projects into One Library
    

    View Slide

27. Workspace Analysis
    Project Boundaries
    27
    {
    "@nrwl/nx/enforce-module-boundaries": [
    "error",
    {
    "enforceBuildableLibDependency": true,
    "allowCircularSelfDependency": true,
    "allow": [],
    "depConstraints": [
    {
    "sourceTag": "scope:app",
    "onlyDependOnLibsWithTags": ["scope:shared"]
    },
    {
    "sourceTag": "scope:shared",
    "onlyDependOnLibsWithTags": ["scope:shared"]
    },
    {
    "sourceTag": "scope:plugin",
    "onlyDependOnLibsWithTags": ["scope:plugin", "scope:shared"]
    }
    ]
    }
    ]
    }
    Scope Tags
    app1
    routing
    recoil
    app1-e2e
    playwright
    app
    app
    shared shared plugin
    app2
    app
    implicit
    implicit
    .eslintrc.json
    

    View Slide

28. Plugable Tooling
    Executors
    28
    › Executing Similar Tasks on
    Unrelated Projects
    › Leveraging this Consistency to
    Run the Specific Target Across
    Multiple Projects
    › Metadata to Define the Available
    Options
    › Composable Executors
    {
    "name": "app1",
    "$schema": "../../node_modules/nx/schemas/project-schema.json",
    "sourceRoot": "apps/app1/src",
    "projectType": "application",
    "targets": {
    "build": {
    "executor": "@mono-web/webpack:build",
    ...
    },
    "serve": {
    "executor": “@mono-web/webpack:serve",
    ...
    },
    "lint": {
    "executor": "@nrwl/linter:eslint",
    ...
    },
    "test": {
    "executor": “@nrwl/jest:jest",
    ...
    },
    "analyze": {
    "executor": "@mono-web/webpack:analyze",
    ...
    }
    },
    "tags": ["scope:app"]
    }
    project.json
    

    View Slide

29. Plugable Tooling
    Generators
    29
    › Generating, Updating and
    Structuring Code in a Certain
    Way
    › Migration Generator
    › AST Manipulation
    › EJS Syntax for Control flow in
    the Template
    › Composable Generators
    Generated Files
    src/index.html
    src/main.ts
    src/components/
    src/pages/
    tsconfig.json
    jest.config.ts
    …
    Template Files
    src/index.html
    src/main.ts__tmpl__
    src/components/
    src/pages/
    tsconfig.json__tmpl__
    jest.config.ts__tmpl__
    …
    Generate
    create/update
    fi
    les
    create/update project con
    fi
    g
    create/update TS con
    fi
    g
    compose other generators
    update package dependencies
    format
    fi
    les
    …
    

    View Slide

30. Plugable Tooling
    Create Our Own Plugin: React Application Generator
    30
    Template
    Files
    Normalize
    Options
    Create
    Application
    Files
    Workspace
    Configurations
    Create
    Project
    Configurations
    Update
    Workspace
    Configurations
    Compose
    Other
    Generators
    Generator
    Entry
    Linting Files
    Format Files
    …
    

    View Slide

31. Plugable Tooling
    Create Our Own Plugin: Playwright Executor
    31
    Normalize
    Options
    Start Target
    Server
    Chokidar
    Watcher
    Playwright CLI
    Executor
    Entry
    E2E Test
    Codegen
    Trace Viewer
    

    View Slide

32. What’s Next
    Distributed Computation Caching
    32
    CI Agent
    > build, test, lint, …
    Remote Cache
    Developer
    > build, test, lint, …
    Developer
    > build, test, lint, …
    Developer
    > build, test, lint, …
    

    View Slide

33. Thank you
    33
    

    View Slide