[EN] Golang Package Composition for Web Application: The Case of Mercari Kauru

Transcript

1. Mercari Tech Conf 2017 Software Engineer Osamu Tonomori Golang Package Composition for Web Applications: The Case of Mercari Kauru
2. Contents 1. About Me 2. About Mercari Kauru 3. About Package Composition 4. Usefulness of the Interface 5. Future Developments 6. Conclusion
3. About Me
4. About Me • 主森 理 (Osamu Tonomori) Screen name: @osamingo • Souzoh Inc. Mercari Kauru team • Software Engineer Server-side, Gopher ʕ◔ϖ◔ʔ
5. GitHub - https://github.com/osamingo
6. I am the funny looking guy in the middle
8. About Mercari Kauru
11. May September Overall Listing Numbers Over 2 million Released! Yay!
12. I presented on how to use GAE at the TECH PLAY Conference 2017 • Google “gae 新規 web” to see the slides
13. About Package Composition
14. Survey Results ʕ◔ϖ◔ʔ from Golang.Tokyo Participants Choose three topics that you thought were interesting Package separation 29 responses
15. Advanced Testing in Go - GopherCon 2017 • Google “gophercon testing” to see it
16. The three types of package composition • One Package When the repository itself is an individual package • Flat Packages Makes the purpose of each package clear, and separates accordingly. • Multiple Packages Separated by each design pattern, such as MVC, DDD etc.
17. One Package • Simple is Best (๑•̀ㅂ•́)و✧ • Easy to get coverage • Good for things that can be done with a simple structure like libraries etc. . ├── ctx.go ├── ctx_test.go ├── debug.go ├── debug_test.go ├── error.go ├── error_test.go ├── handler.go ├── handler_func.go ├── handler_func_test.go ├── jsonrpc.go ├── jsonrpc_test.go ├── method.go ├── method_test.go ├── parse.go └── parse_test.go
18. Difference between Flat Packages and Multiple Packages Feature A Feature B Feature C Feature A, Feature B, Feature C Feature A, Feature B, Feature C Feature A, Feature B, Feature C Flat Packages Categorized based on different features Multiple Packages Categorized based on design patterns
19. Flat Packages • Similar to the standard Golang package structure • Even if you traverse the layers, it settles down at the second level • As roles are clear for each package, they are loose coupling • Good for Micro Services and Middleware etc. . ├── ctx │ ├── ctx.go │ └── ctx_test.go ├── debug │ ├── debug.go │ └── debug_test.go ├── error │ ├── error.go │ └── error_test.go ├── handler │ ├── handler.go │ ├── handler_func.go │ └── handler_func_test.go ├── jsonrpc.go ├── jsonrpc_test.go ├── parse │ ├── parse.go │ └── parse_test.go └── method ├── method.go └── method_test.go
20. Multiple Packages • Package design that is closely linked to design patterns • Good for monolithic web services . ├── controller │ ├── debug │ │ ├── debug.go │ │ └── debug_test.go │ └── handler.go ├── jsonrpc.go ├── jsonrpc_test.go ├── model │ ├── ctx │ │ ├── ctx.go │ │ └── ctx_test.go │ ├── error │ │ ├── error.go │ │ └── error_test.go │ ├── marshal │ │ ├── unmarshal.go │ │ └── unmarshal_test.go │ └── method │ ├── method.go │ └── method_test.go └── view └── index.html.tpl
21. Mercari Kauru uses Multiple Packages • A monolithic web service managed with one repository 　As it uses GAE, it works well with a Project Based GOPATH. • Uses various design patterns that simplify the DDD 　It is close to the composition of our previous Mercari Atte application and accounts for human resource mobility
22. About the Google App Engine • Platform as a Service (PaaS) offered by GCP • Competitors include Heroku, Engine Yard etc.
23. Repository Structure • Uses a gb that works well with GAE projects • $ gb vendor … 　Dependent library management • $ gb gae … 　gae-sdk wrap command group • Environment variables used for the test should be injected with direnv. . ├── Makefile ├── appengine # File that defines GAE│ ├── admin-api │ ├── api │ ├── cron.yaml │ ├── default │ ├── dispatch.yaml │ ├── index.yaml │ ├── management │ └── queue.yaml ├── cmd # Handmade convenient command group ├── docs # uml, sql etc. design docs ├── src │ └── kauru # Application itself │ ├── application │ ├── domain │ ├── infrastructure │ ├── interfaces │ └── library ├── test # Defines variables for test environment │ └── .envrc └── vendor # Dependent library management ├── manifest └── src
24. • Made from 4 layers 　Application 　Domain 　Infrastructure 　Interface(s) • Existence of the library directory 　Package group forms a complete unit Lightweight DDD . └── kauru ├── application ├── domain ├── infrastructure ├── interfaces └── library
25. • Gathers up domain layer processes and provides them to the interface layer • Separates packages with same concept as Flat Packages • Used with internal packages and covers up the common logic Application Layer . └── kauru ├── application │ ├── audio │ │ └── audio.go │ ├── book │ │ └── book.go │ ├── exhibit │ ├── install │ ├── internal │ ├── product │ ├── ranking │ ├── search │ ├── stock │ ├── user │ └── visual ├── domain ├── infrastructure ├── interfaces └── library
26. Domain Layer • The Domain layer does not exist in other layers • Provides Entity, Enum, Interface etc. . └── kauru ├── application ├── domain │ ├── bookmaster │ │ └── repository.go │ ├── bookmaster.go │ ├── configuration.go │ ├── domain.go │ ├── error.go │ ├── mercari.go │ ├── pager.go │ ├── product.go │ ├── time.go │ ├── user │ │ └── repository.go │ └── user.go ├── infrastructure ├── interfaces └── library
27. Infrastructure Layer • Wrap the GAE library • Has basic processing capabilities　 Logging 　Validation Rule 　Word Filtering 　Value Get/Set in Context 　Persistence . └── kauru ├── application ├── domain ├── infrastructure │ ├── backoff │ │ └── backoff.go │ ├── configuration │ │ └── configuration.go │ ├── crypto │ ├── ctx │ ├── identifier │ ├── log │ ├── persistence │ ├── queue │ ├── storage │ ├── tx │ ├── validation │ └── wordfilter ├── interfaces └── library
28. Interface Layer • JSON-RPC’s Handler group 　Search.FindProduct etc. • Original Error definition 　-32044: Not Found etc. • Interceptor group 　Packs a lot into context 　Action Logger etc. • Other 　Startup processing 　Obtains movement condition . └── kauru ├── application ├── domain ├── infrastructure ├── interfaces │ ├── api │ │ └── search │ │ └── find_product.go │ ├── error.go │ ├── interceptor │ │ └── recovery.go │ ├── interfaces.go │ ├── management │ │ └── cron │ │ └── backup_datastore.go │ ├── stats.go │ └── warmup.go └── library
29. Library Directory • Package group that is not dependent on the application • Many have been made into OSS packages 　github.com 　　/osamingo - 5 packages 　　/tenntenn - 3 packages . └── kauru ├── application ├── domain ├── infrastructure ├── interfaces └── library ├── device │ └── device.go ├── errutil │ └── temporary.go ├── io └── strrecord
30. Points to Be Careful of • Each underlying layer is composed of flat packages 　The most important thing is cutting the packages appropriately • Even within the layer, we need to protect the domain layer above all else 　If the Domain fails, we’ll be in cycle import hell and it will become weak to changes. • If you’re lost, choose the most simple path 　I like the quote “Keep It Simple Stupid”
31. Usefulness of the Interface
32. Background • The design pattern used for Multiple Packages often waits for object oriented movement • With Golang, we can use the interface and fillers to expect object oriented movement.
33. So does Golang even have objects? • “Is Golang an object oriented language?” 　http://postd.cc/is-go-object-oriented/ 　This post from @spf13 is really helpful. • What does “object oriented” here refer to? a. It doesn’t refer to the program as code and data, but uses the concept of an “object” to integrate the two b. An object is an abstract data type that has a condition (data) and a behavior (code)
34. type ( Duck interface { Waddler Quacker } Waddler interface { Waddle() (lat, lng float64) } Quacker interface { Quack() string } ) • Defined method list • An interface without a method is interface{} • All defined types fulfill interface{} How to Define the Interface
35. type Osamingo struct { Lat float64 Lng float64 } func (o *Osamingo) Waddle() (float64, float64) { return o.Lat, o.Lng } func (o *Osamingo) Quack() string { return "Hey" } func main() { var o interface{} = &Osamingo{ Lat: 35.662056, Lng: 139.728451, } switch o.(type) { case Duck: fmt.Println("Hi, duck!") } } How to Implement the Interface • Similar to Duck Typing • If you implement Waddler and Quacker, it will become Duck.
36. An Example of the Flow of Registering User Info in the Database • An example of what we are actually doing with Mercari Kauru. • Will introduce with the following steps a. Entity interface design b. User entity implementation c. UserRepository interface design d. UserRepository implementation e. Cloud Datastore library function wrap
37. About Cloud Datastore • A fully-managed NoSQL service offered by GCP GCP • Used by Pokémon GO etc. • “Google Cloud Datastore Inside-Out” 　https://www.slideshare.net/enakai/google-cloud-datastore-insideout
38. // Domain layer - domain/domain.go type ( EntityBehavior interface { Identifier datastore.PropertyLoadSaver } Identifier interface { GetID() string SetID(string) } UpdateTimestamper interface { GetUpdatedAt() time.Time SetUpdatedAt(time.Time) } ) • EntityBehavior 　Condition for storing in the database • Identifier Getter/Setter for the ID datastore. • PropertyLoadSaver 　Manages load, save from the datastore • CreateTimestamper 　Manages CreatedAt • UpdateTimestamper 　Manages UpdatedAt Entity Interface
39. User Entity Architecture • Construct User Entity 　Nothing complicated // Domain layer - domain/user.go type User struct { ID string `json:"id" datastore:"-" validate:”required”` ConnectedAt time.Time `json:"connected_at" datastore:"connected_at" validate:”required”` }
40. User Entity Implementation // Domain layer - domain/user.go func (u *User) GetID() string { return u.ID } func (u *User) SetID(id string) { u.ID = id } func (u *User) Load(p []datastore.Property) error { return Load(u, p) } func (u *User) Save() ([]datastore.Property, error) { return Save(u) } • Implementing EntityBehavior for the user • For Load and Save, prepare a common function within the entity
41. Entity’s Load Function // Domain layer - domain/domain.go func Load(dst interface{}, p []datastore.Property) error { err := datastore.LoadStruct(dst, p) if err != nil { return err } return LoadTimestamps(dst, p) } func LoadTimestamps(i interface{}, p []datastore.Property) error { ut, isUT := i.(UpdateTimestamper) if !isUT { return nil } for i := range p { if p[i].Name == "updated_at" { ut.SetUpdatedAt(p[i].Value.(time.Time)) } } return nil } • Uses App Engine library’s LoadStruct • If xTimestamper is implemented, it will be loaded
42. // Domain layer - domain/domain.go func Save(src interface{}) ([]datastore.Property, error) { p, err := datastore.SaveStruct(src) if err != nil { return nil, err } return append(p, SaveTimestamps(src)...), nil } func SaveTimestamps(i interface{}) []datastore.Property { p := make([]datastore.Property, 0, 1) if ut, ok := i.(UpdateTimestamper); ok { ut.SetUpdatedAt(time.Now()) return append(nil, datastore.Property{ Name: "updated_at", Value: ut.GetUpdatedAt(), }) } return nil } • Uses the App Engine library’s savestruct • If xTimestamper is implemented, a time will be set Entity’s Save Function
43. Repository’s Interface // Domain layer - domain/user/user.go type repository interface { Get(c context.Context, id int64) (*domain.User, error) Save(c context.Context, u *domain.User) error } var Repository repository • Java Interface definition • The Infrastructure layer’s persistence package is in charge of implementation
44. Repository Implementation // Infrastructure layer - // infrastructure/persistence/user_repository.go type UserRepository datastoreRepository func NewUserRepository() *UserRepository { return &UserRepository{ kind: "User", } } func (r *UserRepository) Get(c context.Context, id int64) (*domain.User, error) { u := &domain.User{} u.SetID(id) return u, get(c, r.kind, u) } func (r *UserRepository) Save(c context.Context, u *domain.User) error { return put(c, r.kind, u) } • DI the domain layer from the Infrastructure layer • For get and put, prepare a common function within the repository.
45. Datastore Function Wrap // Infrastructure layer - // infrastructure/persistence/persistence.go func get(c context.Context, kind string, dst domain.EntityBehavior) error { key := datastore.NewKey(c, kind, dst.GetID(), 0, nil) return datastore.Get(c, key, dst) } func put(c context.Context, kind string, src domain.EntityBehavior) error { if err := validation.Check(src); err != nil { return err } key :-= datastore.NewKey(c, kind, src.GetID(), 0, nil) _, err := datastore.Put(c, key, src) return err } • For each parameter, set a kind and EntityBehavior • Use the Get, Put in the app engine library
46. Unique Points • Separating use of an interface as an entity and as a repository 　For the Entity, expect an action 　For the Repository, expect DI • Wrap the App Engine library and use it 　Use the interface as an entity 　
47. Future Developments
48. Moving Towards Clearer DDD • As the application grows, the logic expands into the application layer and enlarges internal packages more than required. => In order to prioritize development speed, I tried my best to implement DRY => However, it is true that DDD and DRY are not compatible => Remove the reference from the interface, Application layers to the infrastructure layer.
49. Streamlining the Initialization Process • As an application grows, the initialization processing becomes split up into interface layer and infrastructure layer parts. => Register JSON-RPC method in the interface layer => DI of domain repository in the infrastructure layer => You’re really not supposed to initialize on each layer… (ヽ´ω`) => Start an application and streamline initialization processing within the main function.
50. Knowledge Extraction • Of the interfaces introduced, identifier and TimeStamper are some of the more commonly used => Extracting the Datastore library as a loosely wrapped construct makes it easier to generalize and maintain => Most new services within Souzoh use GAE and Datastore, which helps improve the synergy => Will continue to create libraries for sharing, adding test functionality as well
51. Conclusion
52. Conclusion • It is best to consider the programming language you are going to use before designing. Dev will be smoother if you consider what each language is best at beforehand • It is hard to be perfect from the start. However, it is possible to make it adaptable to change. 　It is not good to create a condition where special cases are not accepted right from the start.
53. “ ” Clear is better than clever. Rob Pike Go Proverts, Google, Inc.
54. Thanks for your attention!