MTC2018 - Listing Service: From Monolith to Microservices

Speaker: Morikuni Taihei

One of the areas that Mercari is working to implement microservices in is the item listing feature. In this session, we will discuss how we’re handling the switch from monolithic PHP code to microservices using Go while maintaining as much compatibility as possible.

Transcript

1. Listing Service: From Monolith to Microservices Taihei Morikuni Software Engineer (Backend)
2. Software Engineer Backend Taihei Morikuni
3. Listing Service: From Monolithic to Microservices
4. Listing Service • Microservice for managing the listing function on Mercari • This is the first project for transitioning from monolithic (PHP) to microservices (Go) • Currently under development for one HTTP endpoint • Use gRPC for connecting with other microservices
5. Strategy to Transition to Microservices Split services while maintaining API compatibility Enable independent development for each team • The goal of transitioning to microservices is not to split the code, but to make a scalable organization • Enable Canary Release by maintaining API compatibility, lowering the risk
6. • Implement only logic related to listings • Link the portion left in PHP with Webhook Technological Strategy for Listing Service • Automated app tests for the existing API app • Manual checking by the QA team • AQA team test tools (under development) Split services while maintaining API compatibility Enable independent development for each team
7. Challenges of Transitioning to Microservices Connecting from GCP to MySQL at Sakura Handling Additions/Modifications to Features Splitting of Transactions Compatibility with PHP Exceptions
8. Connecting from GCP to MySQL at Sakura • Share PHP and DB until the transition to microservices has been completed • In terms of security, don’t want to make MySQL ports available externally Problem Solution • Develop the SQL on gRPC service and a Go library ◦ In general, handle only CRUD, don’t handle Join and Transaction
9. Connecting from GCP to MySQL at Sakura message SelectRequest { repeated string columns = 1; string table = 2; Where where = 3; ... } message SelectResponse { repeated Row rows = 1; int64 count = 2; }
10. Connecting from GCP to MySQL at Sakura req := Select("id", "name", "age"). From("users"). Where(Eq("lang", "ja"), And(Gt("age", "18"))). OrderBy(ASC("created_at")). ToRequest() err := dbClient.Select(ctx, &user, req)
11. Splitting of Transactions • Updates of items, shipping information, and analysis data were included in a single transaction Problem Solution • For item data, Item Service (Sakura) will process the transaction • For data that the user can see, process as much as possible within the request • For internal data, aim for asynchronous eventual consistency using Cloud PubSub
12. Splitting of Transactions → Transaction: Unnatural if the item has been listed but there is no transaction → Synchronous: Unnatural if no image of the item was included → Asynchronous: No shipment until item sold → Asynchronous: Customer cannot see the data for analysis function listing() { $db->begin(); $item->save(); $photo->save(); $shipping->save(); $analysis->save(); $db->commit(); }
13. Handling Additions/Modifications to Features • As listing is a core Mercari function, functions will be added and modified during the transition from PHP to Go Problem Solution • Create an update tracking branch in the PHP repository, and delete the code transitioned to Go ◦ If an update has been made, the code will conflict when merged ◦ Once all of the code disappears the process is complete
14. Handling Additions/Modifications to Features function do_something() { if (is_x()) { return 1; } return hello(); } func DoSomething() int { }
15. function do_something() { if (is_x()) { return 1; } return hello(); } Handling Additions/Modifications to Features func DoSomething() int { if IsX() { return 1 } }
16. Handling Additions/Modifications to Features function do_something() { if (is_x()) { return 1; } return hello(); } func DoSomething() int { if IsX() { return 1 } return Hello() }
17. Handling Additions/Modifications to Features function do_something() { if (is_x() || is_y()) { return 1; } return hello(); } func DoSomething() int { if IsX() || IsY() { return 1 } return Hello() } Merge conflict!!
18. Handling Additions/Modifications to Features function do_something() { if (is_x() || is_y()) { return 1; } additional_process(); return hello(); } func DoSomething() int { if IsX() || IsY(){ return 1 } AdditionalProcess() return Hello() } Not migrated!!
19. Compatibility with PHP Exceptions • A PHP exception name is included in the API response as an error code ◦ The client is using PHP exception names ◦ Must also return the PHP exception names from Go • A part of the response JSON is different in each type of exception • By transitioning to microservices, exceptions are also distributed out among each service ◦ Each service must maintain compatibility with the PHP exceptions Problem
20. PHP Exception Compatiblity Solution • Develop a shared Go library to maintain compatibility with PHP exceptions • Add PHP exception information for handling errors in gRPC Status.details • Retrieve the exception information from PHP-compatible HTTP endpoints, and directly generate the response JSON
21. PHP Exception Compatiblity func DoSomething(ctx context.Context, r *pb.Request) (*pb.Response, error) { ... if err != nil { s, err := exception.WithException(codes.NotFound, exception.NotFoundException) return nil, s.Err() } }
22. PHP Exception Compatiblity func HTTPServer(r http.ResponseWriter, r *http.Requeset) { res, err := client.DoSomething(ctx, &pb.Request{}) if err != nil { e, isException, err := exception.FromError(err) body, err := exception.ToJSON(e) r.Write(body) } }
23. Challenges of Transitioning to Microservices Connecting from GCP to MySQL at Sakura Handling Additions/Modifications to Features Splitting of Transactions Compatibility with PHP Exceptions
24. Summary • In order to solve long-term problems (and not increase debt), we are keeping these issues in mind as we transition to microservices • Because this is the first project for transitioning from monolithic (PHP) to microservices (Go), we haven’t fully organized our assets, meaning most parts of the process are difficult • The pursuit of accuracy, when there is no correct answer, can be enjoyable ◦ Believe in the setup in which I believe!