Database Lounge Tokyo #6 LT - Jepsen Test Introduction

How Jepsen works

Transcript

1. Jepsen Test Introduction DBLTokyo #6 25th June 2020 Yuji Ito
2. 自己紹介 • Architect@Scalar ◦ Scalar DLT の開発、性能評価、検証 ◦ Cassandra への貢献 ▪ 機能提案、バグ調査、 Jepsen テスト ◦ Cosmos DB 対応中 • 以前は、SSD の R&D や firmware 開発 ◦ Enterprise SSD, Consumer SSD, UFS 2
3. Jepsen • A framework for distributed systems verification, with fault injection ◦ https://github.com/jepsen-io/jepsen ◦ Jepsen Test という共通テストや評価基準があるわけではない ▪ 代表的なテストはある : ${JEPSEN}/jepsen/src/jepsen/tests • Why “Jepsen”? ◦ The name comes from Carly Rae Jepsen ◦ The author explains Jepsen by using her songs like “Call Me Maybe” 3
4. 4 https://aphyr.com/posts/281-jepsen-on-the-perils-of-network-partitions
5. Break databases • Many databases are tested with Jepsen • Jepsen found bugs and reported them to developers ◦ The author offers tech talks, training classes, consulting services 5 https://jepsen.io/analyses
6. Why Jepsen? • Jepsen framework provides not only a controller for DB setup and requesting but also checking consistency and complex failure injection mechanisms ◦ Developers can customize all of them for their system ▪ DB 開発元が作成したり、 Jepsen 作者に依頼することでテストは作られている 6
7. Caution • Jepsen 0.2.0 has been released ◦ This is a significant compatibility-breaking release • 発表者が今使っているのは 0.1.18 なので、変わったところがあるかも ◦ scalar-jepsen のテストはそのうち対応予定 7
8. Architecture 8 Tested System Control Client worker Client worker ... DB DB DB Client issues requests to DB nodes Nemesis worker Nemesis injects failures Setup DB node by the control : Node (VM instance/machine) : Thread on the control node DB DB
9. Node • Control node (at most 1 node) ◦ Client (multiple threads) ◦ Nemesis (1 thread) • DB node (multiple nodes) 9 Tested System Control Client worker Client worker ... DB DB DB Nemesis worker DB DB
10. Control Node 1. Initialize a test 2. Setup DB nodes 3. Start the client workers ◦ Give what types of requests to be invoked 4. Start the nemesis worker ◦ Give what types of failure to be injected 5. Teardown all workers and DB nodes 6. Check the result 10 Tested System Control Client worker Client worker ... DB DB DB Nemesis worker DB DB
11. Client worker 1. Prepare a requester 2. Initialize DB tables if needed 3. Invoke requests to DB nodes ◦ Read, write, other operations 4. Close the requester 11 Tested System Control Client worker Client worker ... DB DB DB Nemesis worker DB DB
12. Nemesis worker • Invoke nemesis to inject failure to DB node ◦ Network partition, crash, clock drift etc. • Recover DB node from the nemesis 12 Tested System Control Client worker Client worker ... DB DB DB Nemesis worker DB DB
13. DB node • Setup DB process by the control node • DB process just runs ◦ The DB process doesn’t know failure injection 13 Tested System Control Client worker Client worker ... DB DB DB Nemesis worker DB DB
14. Check the result • Control node check the result with Checker ◦ Control node records all requests and the results ◦ Checker can be defined by developers ◦ Jepsen-io provides libraries ▪ Knossos: Linearizability checker ▪ Elle: Transactional consistency checker 14
15. Report • Checking result • Performance • Each request timing 15 https://jepsen.io/
16. Summary • Jepsen はテストフレームワーク、分散システムの独自テストを作れる • Jepsen テストでは、Control node と DB node が動く ◦ Client と Nemesis は Control node 上で動く • FYI: ◦ 汎用テストフレームワーク Kelpie 作りました ▪ https://github.com/scalar-labs/kelpie 16
17. End 17