Why my team is stepping out of sharing DB in multiple products

Transcript

1. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 Production Issue No.2 Why my team is stepping out of sharing DB in multiple products 障害シリーズ第二弾 なぜ複数プロダクトによるDB利用から脱却しようとしているのか
2. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 Agenda 1. Quick Self-Introduction 2. What happened? 3. Why did it happen? 4. How did we fix it? 5. Conclusion Presentation Language: English プレゼン資料：日本語メイン Confidential
3. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 Quick Self-Introduction Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 Teruhiko Yatomi - ANDPADに入社して3年が経過しました - ANDPAD受発注というプロダクトの開発リードをしています Github: teruhiky
4. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 What happened? Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 1. アンチパターンと知りながらも新規プロダクトでANPDAD本体のDBを参照した 2. 対象のテーブルにANDPAD本体側でパーティション分割の変更を入れた 3. 新規プロダクトで主キーの齟齬があり500エラーになった
5. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 What happened? Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 1. アンチパターンと知りながらも新規プロダクトでANPDAD本体のDBを参照した ◦ 通知テーブルをANDPAD本体がWritable, ANDPAD受発注がReadableで接続 本体DB （通知テーブル) ANDPAD本体 ANDPAD受発注 Write Read Read
6. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 What happened? Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 2. 対象のテーブルにANDPAD本体側でパーティション分割の変更を入れた ◦ ANDPAD本体とANDPAD受発注で同じ変更を入れなくてはいけなかった 本体DB （通知テーブル) ANDPAD本体 ANDPAD受発注 Write Read Read # app/models/notification.rb self.primary_key = :id # app/models/notification.rb self.primary_key = :id データ量が多いのでid, created_atを主 キーとしてパーティション分割 実装の追従ができなかった
7. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 What happened? Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 3. 新規プロダクトで主キーの齟齬があり500エラーになった ◦ IDでfindをかけていたが500エラー # app/controllers/…/notifications_controller.rb 27 notification = Notification.find(notification_id) ↓↓↓↓↓↓ ActiveRecord::UnknownPrimaryKey Unknown primary key for table notifications in model Notification.
8. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 Why did it happen? Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 1. 新規プロダクトの立ち上げを超急ピッチでやっていた（2ヶ月で初回リリースのスケジュール） 2. 暫定対応ででもマルチDBで作らざるを得なかった ◦ ANDPAD本体は成熟したプロダクト、新規プロダクトは毎日がリリースみたいなもの ▪ ANDPAD本体にAPIを実装するリリースサイクルと、実装負荷が取れなかった ◦ PubSubのようなサービス間で同期する様な構造がまだなかった 本体DB （通知テーブル) ANDPAD本体 ANDPAD受発注 Write Read Read 受発注DB （通知テーブル) Write 本体DB （通知テーブル) ANDPAD本体 ANDPAD受発注 Write Read Read
9. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 How did we fix it? (tentative solution) Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 1. 本体と同様の実装を入れてリリース！ # app/models/notification.rb self.primary_key = :id
10. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 How did we fix it? (solutions) Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 • そもそも同じテーブルを見る様にし続けていたのが原因！（今回の変更の進め方は割愛） ◦ 本体にAPIを実装してそれを参照するようにする ◦ PubSubで受発注用DBに同期テーブルを作る 本体DB （通知テーブル) ANDPAD本体 ANDPAD受発注 Write Read Read 受発注DB （通知テーブル) Write 本体DB （通知テーブル) ANDPAD本体 ANDPAD受発注 Write Read Read
11. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 Conclusion Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止 • 暫定対応は事業戦略上回避できないところもあるので、リスクは回避できないがその分正し く制御するもの ◦ 急成長企業では社員が爆増なので、Github Actions等で暫定的にセーフガードをおく べきだった ◦ 今回はもう恒久対応にリソースを割きに行く！ • 今後他プロダクトにおいても必要となる施策はきっちり対応していきたい
12. Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. Proprietary & Confidential 無断転載・無断複製の禁止 Q&A Confidential Copyright © 2020 Present ANDPAD Inc. This information is confidential and was prepared by ANDPAD Inc. for the use of our client. It is not to be relied on by and 3rd party. 無断転載・無断複製の禁止