impguard - protect your project structure

Transcript

1. Impguard
   protect your
   project structure
   Warsaw, November 2020
   Oleg Kovalov
   

   View Slide

2. • Gopher for ~5 years
   • Open source contributor
   • Engineer at GoGoApps
   Hi, i’m Oleg
   olegk.dev
   @oleg_kovalov
   @cristaloleg
   

   View Slide

3. At GoGoApps we’re...
   

   View Slide

4. At GoGoApps we’re...
   • (re)building a product for a big company™
   

   View Slide

5. At GoGoApps we’re...
   • (re)building a product for a big company™
   • with a lot of microservices
   • and a complicated business logic
   

   View Slide

6. At GoGoApps we’re...
   • (re)building a product for a big company™
   • with a lot of microservices
   • and a complicated business logic
   • with a care on our code and architecture
   • CI, linters, code reviews, everything as it should be
   

   View Slide

7. At GoGoApps we’re...
   • (re)building a product for a big company™
   • with a lot of microservices
   • and a complicated business logic
   • with a care on our code and architecture
   • CI, linters, code reviews, everything as it should be
   • ...and everything is fine!
   

   View Slide

8. At GoGoApps we’re...
   But there is one thing is running in my mind for a long time…
   • (re)building a product for a big company™
   • with a lot of microservices
   • and a complicated business logic
   • with a care on our code and architecture
   • CI, linters, code reviews, everything as it should be
   • ...and everything is fine!
   

   View Slide

9. Define me a problem first
   

   View Slide

10. Define me a problem first
    • linters give you suggestions on code level
    • mighty golangci-lint is The Beast
    

    View Slide

11. Define me a problem first
    • linters give you suggestions on code level
    • mighty golangci-lint is The Beast
    • but let’s look level above
    

    View Slide

12. Define me a problem first
    • linters give you suggestions on code level
    • mighty golangci-lint is The Beast
    • but let’s look level above
    • how your packages are organized?
    

    View Slide

13. Setup, goals, solution
    

    View Slide

14. Setup, goals, solution
    • you’ve a monorepo
    • or just a bunch of repos
    

    View Slide

15. Setup, goals, solution
    • you’ve a monorepo
    • or just a bunch of repos
    • and you want to keep them similar
    • same folder structure
    • same filenames
    • same imports
    • same something
    

    View Slide

16. Setup, goals, solution
    • you’ve a monorepo
    • or just a bunch of repos
    • and you want to keep them similar
    • same folder structure
    • same filenames
    • same imports
    • same something
    • custom tooling solves this!
    

    View Slide

17. Example
    

    View Slide

18. Let’s talk about imports
    

    View Slide

19. Let’s talk about imports
    • core idea of (re)using others code
    • staying on the shoulders of giants as Newton said
    

    View Slide

20. Let’s talk about imports
    • core idea of (re)using others code
    • staying on the shoulders of giants as Newton said
    • reducing repeating work
    • it’s simpler to integrate something than build from scratch
    • ^^^ citation needed
    

    View Slide

21. Let’s talk about imports
    • core idea of (re)using others code
    • staying on the shoulders of giants as Newton said
    • reducing repeating work
    • it’s simpler to integrate something than build from scratch
    • ^^^ citation needed
    • but nothing is free
    • others code needs to be verified and used carefully
    • you can rely on any code and use it everywhere
    • but that might be a ticking bomb (LeftPad-like problem)
    

    View Slide

22. When in doubts - see golang/go
    

    View Slide

23. When in doubts - see golang/go
    • Go doesn’t allow import cycles - that’s cool
    • simpler compilation routine and code organization
    

    View Slide

24. When in doubts - see golang/go
    • Go doesn’t allow import cycles - that’s cool
    • simpler compilation routine and code organization
    • But that’s not enough
    • compiling that you don’t really need is time and cpu wasteful
    • See: compiler bomb
    

    View Slide

25. When in doubts - see golang/go
    • Go doesn’t allow import cycles - that’s cool
    • simpler compilation routine and code organization
    • But that’s not enough
    • compiling that you don’t really need is time and cpu wasteful
    • See: compiler bomb
    • Stdlib has a test for imports
    https://github.com/golang/go/blob/release-branch.go1.14/src/go/build/deps_test.go
    https://github.com/golang/go/blob/release-branch.go1.15/src/go/build/deps_test.go
    

    View Slide

26. View Slide

27. View Slide

28. What does it mean?
    

    View Slide

29. What does it mean?
    • in other words you’ve a rigid code structure
    • you cannot import something without an explicit change
    

    View Slide

30. What does it mean?
    • in other words you’ve a rigid code structure
    • you cannot import something without an explicit change
    • this can be a solution for some rare changed code
    • but not for feature rich and fast moving packages
    

    View Slide

31. What does it mean?
    • in other words you’ve a rigid code structure
    • you cannot import something without an explicit change
    • this can be a solution for some rare changed code
    • but not for feature rich and fast moving packages
    • what we can do?
    • check Github!
    

    View Slide

32. What do we’ve right now?
    

    View Slide

33. What do we’ve right now?
    • `$ go mod why`
    • out of the box solution
    

    View Slide

34. What do we’ve right now?
    • `$ go mod why`
    • out of the box solution
    • https://github.com/OpenPeeDeeP/depguard
    • allow/block list for deps (presented in golangci-lint)
    

    View Slide

35. What do we’ve right now?
    • `$ go mod why`
    • out of the box solution
    • https://github.com/OpenPeeDeeP/depguard
    • allow/block list for deps (presented in golangci-lint)
    • https://github.com/tailscale/depaware
    • know what is imported by whom (via golden file)
    

    View Slide

36. What do we’ve right now?
    • `$ go mod why`
    • out of the box solution
    • https://github.com/OpenPeeDeeP/depguard
    • allow/block list for deps (presented in golangci-lint)
    • https://github.com/tailscale/depaware
    • know what is imported by whom (via golden file)
    • https://github.com/divan/depscheck
    • helps to find (and copy-paste) small dependencies
    

    View Slide

37. What do we’ve right now?
    • `$ go mod why`
    • out of the box solution
    • https://github.com/OpenPeeDeeP/depguard
    • allow/block list for deps (presented in golangci-lint)
    • https://github.com/tailscale/depaware
    • know what is imported by whom (via golden file)
    • https://github.com/divan/depscheck
    • helps to find (and copy-paste) small dependencies
    • … (sorry, I cannot copy-paste all the Github ):
    

    View Slide

38. go mod why
    

    View Slide

39. go mod why
    > go mod why github.com/go-ole/go-ole
    # github.com/go-ole/go-ole
    tailscale.com/wgengine/router
    github.com/go-ole/go-ole
    

    View Slide

40. go mod why
    > go mod why github.com/go-ole/go-ole
    # github.com/go-ole/go-ole
    tailscale.com/wgengine/router
    github.com/go-ole/go-ole
    > go mod why github.com/RoaringBitmap/roaring
    # github.com/RoaringBitmap/roaring
    code.gitea.io/gitea/modules/indexer/code
    github.com/blevesearch/bleve
    github.com/blevesearch/bleve/index/scorch
    github.com/RoaringBitmap/roaring
    

    View Slide

41. tailscale/depaware
    

    View Slide

42. divan/depscheck
    

    View Slide

43. What is not ok? (only for me?)
    

    View Slide

44. What is not ok? (only for me?)
    • solving author’s problem
    • let’s look at a broader problem
    

    View Slide

45. What is not ok? (only for me?)
    • solving author’s problem
    • let’s look at a broader problem
    • not extensible
    • golden file or nothing
    • allow/block list
    

    View Slide

46. What is not ok? (only for me?)
    • solving author’s problem
    • let’s look at a broader problem
    • not extensible
    • golden file or nothing
    • allow/block list
    • already feature complete
    • boooooooo, we need a room for the improvements!
    

    View Slide

47. Welcome impguard
    

    View Slide

48. Welcome impguard
    • read as „import guard”
    

    View Slide

49. Welcome impguard
    • read as „import guard”
    • helps to verify
    • what is imported
    • what files are presented
    • what rules are passed
    

    View Slide

50. Welcome impguard
    • read as „import guard”
    • helps to verify
    • what is imported
    • what files are presented
    • what rules are passed
    • community driven!
    • have an idea? - make an issue or ping me!
    

    View Slide

51. Welcome impguard
    • read as „import guard”
    • helps to verify
    • what is imported
    • what files are presented
    • what rules are passed
    • community driven!
    • have an idea? - make an issue or ping me!
    • also no relations to „imperial guard” found
    

    View Slide

52. Welcome impguard
    • read as „import guard”
    • helps to verify
    • what is imported
    • what files are presented
    • what rules are passed
    • community driven!
    • have an idea? - make an issue or ping me!
    • also no relations to „imperial guard” found yet?…
    

    View Slide

53. How does it work?
    

    View Slide

54. How does it work?
    • go list -json all | impguard project.yaml
    

    View Slide

55. How does it work?
    • go list -json all | impguard project.yaml
    • go list -json all
    • returns a loooooot of things
    

    View Slide

56. How does it work?
    • go list -json all | impguard project.yaml
    • go list -json all
    • returns a loooooot of things
    • |
    • simple Unix magic
    

    View Slide

57. How does it work?
    • go list -json all | impguard project.yaml
    • go list -json all
    • returns a loooooot of things
    • |
    • simple Unix magic
    • impguard project.yaml
    • checks results based on config file
    

    View Slide

58. How does it work?
    • go list -json all | impguard project.yaml
    • go list -json all
    • returns a loooooot of things
    • |
    • simple Unix magic
    • impguard project.yaml
    • checks results based on config file
    • simple, isn’t ?
    

    View Slide

59. go list gitea
    ❯ time go list -json all > 1.txt
    go list -json all > 1.txt
    3.07s user 2.63s system 376% cpu 1.516
    total
    ❯ du -sh 1.txt
    4.0M 1.txt
    

    View Slide

60. Show me examples!
    

    View Slide

61. Show me examples!
    "*":
    - name: must have doc.go
    checks:
    type: has_file
    pattern: doc.go
    

    View Slide

62. Show me examples!
    "*":
    - name: must have doc.go
    checks:
    type: has_file
    pattern: doc.go
    - name: explicit runtime is forbidden
    checks:
    type: has_import
    pattern: runtime
    exclude: github.com/company/runtimeutil
    

    View Slide

63. Show me more examples!
    github.com/company:
    - name: don't use bad UUID libs
    checks:
    type: no_import
    patterns:
    - github.com/oh-please-no/uuid
    - github.com/this-too-please/go-uuid
    

    View Slide

64. Show me more examples!
    github.com/company:
    - name: don't use bad UUID libs
    checks:
    type: no_import
    patterns:
    - github.com/oh-please-no/uuid
    - github.com/this-too-please/go-uuid
    - name: no vendors, just modules
    checks:
    type: no_dir
    pattern: vendor
    

    View Slide

65. Show me some more examples!
    github.com/company/project/cmd/*:
    - name: no subpackages for cmd
    checks:
    - type: no_folders
    pattern: "*"
    - type: has_file
    patterns:
    - Dockerfile
    - main.go
    - README.md
    

    View Slide

66. Show me some more examples!
    github.com/company/project/cmd/*:
    - name: no subpackages for cmd
    checks:
    - type: no_folders
    pattern: "*"
    - type: has_file
    patterns:
    - Dockerfile
    - main.go
    - README.md
    github.com/company/project/services/*:
    - name: services should be independent
    checks:
    - type: no_import
    pattern: github.com/company/project/services/*
    

    View Slide

67. Where is the code Lebowski?
    

    View Slide

68. Where is the code Lebowski?
    • good question
    

    View Slide

69. Where is the code Lebowski?
    • good question
    • will be available
    • https://github.com/cristaloleg/impguard
    

    View Slide

70. Where is the code Lebowski?
    • good question
    • will be available
    • https://github.com/cristaloleg/impguard
    • feel free to share your ideas!
    

    View Slide

71. Where is the code Lebowski?
    • good question
    • will be available
    • https://github.com/cristaloleg/impguard
    • feel free to share your ideas!
    • no, really
    

    View Slide

72. Thanks
    

    View Slide

73. Thanks
    • Yegor Myskin
    

    View Slide

74. Thanks
    • Yegor Myskin
    • Iskander (Alex) Sharipov
    • Bohdan Storozhuk
    • Alexey Palazhchenko
    

    View Slide

75. Thanks
    • Yegor Myskin
    • Iskander (Alex) Sharipov
    • Bohdan Storozhuk
    • Alexey Palazhchenko
    • Sylwia Barbachowska
    • Roman Bystrytskyi
    

    View Slide

76. Thanks
    • Yegor Myskin
    • Iskander (Alex) Sharipov
    • Bohdan Storozhuk
    • Alexey Palazhchenko
    • Sylwia Barbachowska
    • Roman Bystrytskyi
    • you and Golang Poland <3
    

    View Slide

77. Also see
    • https://github.com/go-critic/go-critic
    • https://github.com/quasilyte/go-ruleguard
    • https://github.com/cristalhq
    

    View Slide

78. ruleguard
    

    View Slide

79. ruleguard
    // +build ignore
    package gorules
    import "github.com/quasilyte/go-ruleguard/dsl/fluent"
    

    View Slide

80. ruleguard
    // +build ignore
    package gorules
    import "github.com/quasilyte/go-ruleguard/dsl/fluent"
    func _(m fluent.Matcher) {
    m.Match(`$x || $x`,
    `$x && $x`).
    

    View Slide

81. ruleguard
    // +build ignore
    package gorules
    import "github.com/quasilyte/go-ruleguard/dsl/fluent"
    func _(m fluent.Matcher) {
    m.Match(`$x || $x`,
    `$x && $x`).
    Where(m["x"].Pure).
    Report(`suspicious identical LHS and RHS`)
    

    View Slide

82. ruleguard
    // +build ignore
    package gorules
    import "github.com/quasilyte/go-ruleguard/dsl/fluent"
    func _(m fluent.Matcher) {
    m.Match(`$x || $x`,
    `$x && $x`).
    Where(m["x"].Pure).
    Report(`suspicious identical LHS and RHS`)
    m.Match(`!($x != $y)`).Suggest(`$x == $y`)
    m.Match(`!($x == $y)`).Suggest(`$x != $y`)
    }
    

    View Slide

83. Thank you
    Questions?
    @oleg_kovalov
    @cristaloleg
    

    View Slide