A scalable Metadata Ecosystem powered by Golang

Metadata is important because structured data can make up for human fallibilities. In the context of an online C2C Marketplace, with millions ad millions of users and items, this effect is amplified at its maximum. This talk covers our journey in leveraging Golang to build a Metadata Ecosystem that scales.
Overview of agenda:
- Introduction
- Why is Metadata important?
- Our journey: from zero to hero
- Conclusion

By Marco Ganci, Engineering Manager @mercari_jp

This presentation was presented at Go Bangalore Meetup 69.

Please follow us on our LinkedIn account for the latest information!
https://www.linkedin.com/company/mercari-india/

#Engineering #MercariIndia #Backend

Transcript

1. 1 A scalable Metadata Ecosystem powered by Golang Marco Ganci Engineering Manager
2. 2 Introduction
3. 3 Create value in a global marketplace where anyone can buy & sell Mission
4. 4 Businesses
5. 5 　　 What Is Mercari? • Service launch: July 2013 • Operating systems: Android, iOS *Can also be accessed through web browsers • Usage fee: Free *Sales fee for sold items: 10% of the sales price • Regions/languages supported: Base specs for Japan/Japanese • Total number of listings to date: More than 2.5 billion *As of December 2021 The Mercari app is a C2C marketplace where individuals can easily sell used items. We want to provide both buyers and sellers with a service where they can enjoy safe and secure transactions. Mercari offers a unique customer experience, with a transaction environment that uses an escrow system, where Mercari temporarily holds payments, and simple and affordable shipping options.
6. 6 　　 Future Growth Large potential interest in listing 36.1 million People who want to list but haven’t done so yet* 20.4 million Mercari MAU Would like to list if Mercari improved some of its features, or if listing was easier - Want to list items - Would like to list items someday Total number of listings While the total number of listings is growing steadily, we estimate that there are 36.1 million people who want to list items but haven’t done so yet. 　 2.5 billion December 4, 2021 Million items
7. 7 Why is metadata important?
8. 8 Item Seller Buyer
9. 9 Marketplace
10. 10 Enhance
11. 11 Retrieve
12. 12 Understand
13. 13 Why Metadata is important? Unstructured sets of user-generated data + Data that provides information about other data Improve Sellability Buyer Tailored experiences through personalization, and better matching AI More sophisticated marketplaces analysis. Understand connections between sellers and buyers Seller Suggest what kind of item information is useful for potential buyers
14. 14 Our journey: From zero to hero
15. 15 1. Gather requirements and form a rough design idea 2. Check existing service list to see what’s already available 3. Write your design using the template 4. Ask for review & improve 0. Start with a design doc Item Attribute ItemID Attributes Values AttributeID OptionID OR User input text
16. 16 1. Think about the interface of your service 2. Follow the guidelines for designing (and implementing) your APIs. a. gRPC is the recommended protocol for APIs in Mercari 1. Design the interface
17. 17 1. Clone go template project for Mercari microservices 2. The project is providing ▪ easy to develop code structure ▪ naming convention ▪ default CI setting ▪ Dockerfile practice ▪ common observability code 2. Clone the skeleton service
18. 18 1. GRPC: It is the outside interface for the service. It implements the GRPC service as required by the proto. It has no business logic 2. Service: This package contains business logic. It contains methods that can be used to serve the endpoints in the GRPC package 3. Repository: Each resource has its own repository which handles storage/retrieval of entities 4. Infrastructure: It is the wrapper/helper to perform database operations 3. Implement your service GRPC Service Repository Infrastructure Since each layer depends only on the layer below it, the packages can be mocked easily to write a unit test for each package
19. 19 4. Environment lint microservices-terraform IaaS SaaS Google Cloud
20. 20 5. Build test lint code coverage Build docker image Store docker image Google Cloud Build Google Container Registry Item-attributes MS
21. 21 6. Scale Google Cloud Storage lint generate manifest file microservices-kubernetes
22. 22 7. Deploy Google Container Registry Google Cloud Storage dev/prod Build Scale
23. 23 8. Operate dev/prod Alert Monitoring Metric Log Alert Error Report
24. 24 ∞ Iterate
25. 25 Requests and errors per second Zero Hero
26. 26 Conclusions
27. 27 • Build fast, reliable, and efficient software at scale ◦ Go is an open source programming language supported by Google ◦ Easy to learn and get started with ◦ Built-in concurrency and a robust standard library ◦ Growing ecosystem of partners, communities, and tools Go Reference: https://go.dev/
28. 28 • Simple problem, big scale ◦ Focus on the business logic ◦ Quickly deliver value ◦ Easy to expand using modules and cloud services ◦ Perfect for newcomers Go and Metadata Ecosystem