How to Fast Verify Data Backups for ISO certification review @ TECHPULSE 2023

- Speaker: Winter Hung
- Event: http://techpulse.line.me/

備份就像保險一樣，有效備份可以減少停機時間與節省災難期間所帶來的隱藏成本。為達到快速恢復，必須有良好的配置文件，本次將介紹 LINE SHOPPING 如何透過 GORM 和 simple bash testing framework 來確保與驗證在數據備份的正確性。

Transcript

2. Agenda › LINE SHOPPING › Data Backup Analysis › Simple Bash Testing Framework › GORM & Jenkins Integration › Conclusion
3. LINE SHOPPING Popular e-commerce platform 3
4. › Aggregation Service › LINE Points Back › Search & Price Comparison › Lowest Price Product › LINE Official Account › ISO Certification LINE SHOPPING
5. LINE SHOPPING ORDER FLOW Homepage Partner Transform Order 5 1 1
6. Order User Journey Visual Representation Database CPS Report Database CMS Log Database Order Log Database ECID Log Partner Fee Postback Partner Order Postback 1 2 3 4 5 6
7. Data Backup Analysis Migrate Database
8. ISO Certification Review Information Security Management ISO 27001 Make Decisions 70% MySQLdump 100%
9. MySQL Data Backup Architecture MYSQL Master Real MySQL Slave Backup MySQL Slave InnoDB DBA
10. Check backup data Simple Bash Testing Framework
11. How to Data verification Simple Bash Testing Framework Data verification is a process in which different types of data are checked for accuracy and inconsistencies after data migration is done. So the LINE SHOPPING team uses a simple bash testing framework and writes tests that run bash commands, and reports on any failures. Each test is its own bash script
12. Simple Bash Testing Framework Minimal Test Script # Load the testing framework . ./testlib.sh TEST_SUTE_NAME=“Test something” #Number of tests TOTAL_TEST=1 # PUT your tests here assert “Sample test” ‘true’ # Print results 1 2 3 4 5
13. Simple Bash Testing Framework Minimal Test Script # Load the testing framework . ./testlib.sh TEST_SUTE_NAME=“Test something” #Number of tests TOTAL_TEST=1 # PUT your tests here assert “Sample test” ‘true’ # Print results 1 2 3 4 5
14. Simple Bash Testing Framework Minimal Test Script # Load the testing framework . ./testlib.sh TEST_SUTE_NAME=“Test something” # Number of tests TOTAL_TEST=1 # PUT your tests here assert “Sample test” ‘true’ # Print results 1 2 3 4 5
15. Simple Bash Testing Framework Minimal Test Script # Load the testing framework . ./testlib.sh TEST_SUTE_NAME=“Test something” # Number of tests TOTAL_TEST=1 # PUT your tests here assert “Sample test” ‘true’ # Print results 1 2 3 4 5
16. Simple Bash Testing Framework Minimal Test Script # Load the testing framework . ./testlib.sh TEST_SUTE_NAME=“Test something” # Number of tests TOTAL_TEST=1 # PUT your tests here assert “Sample test” ‘true’ # Print results 1 2 3 4 5
17. Simple Bash Testing Framework Contains Several Parts › Set the name of the test suite. This will be displayed in the report › Set the total number of tests, If this amount of tests aren’t run › A Line to load the testing library for TAP（Test Anything Protocol） › The tests themselves › A command to print the results at the end
18. Simple Bash Testing Framework Test Commands Assert “Test Name” “command” to test something Assert_not_ran_ok “Test Name” to check the return value of the last Assert_ran_ok “Test Name” to test a condition
19. Continuous Validation Data Testing GORM & Jenkins Integration
20. GORM and Jenkins Integration › Golang Object-Relational Mapping › Manipulate data from a database › Data defined in classes to SQL › CRUD functions already implemented › Popular choice for Go developers
21. OVERVIEW Gin + GORM + MySQL MySQL Slave Client Gin GORM Client Client Restful Web Service
22. MySQL Backup Validation Architecture Backup MySQL Slave Real MySQL Slave Jenkins Bash Test Framework Real GORM Result Backup GORM RESTful Web Service Client
23. MySQL Backup Validation Minimal Test Script # Load the testing framework . ./testlib.sh #REAL ORM URL real_orm_url=“http://real_mysql_slave.orm.url:9002” #BACKUP ORM URL backup_orm_url=“http://backup_mysql_slave.orm.url:9003” # assert data assert_not_ran_ok $real_data $backup_data # Print results
24. MySQL Backup Validation Print Result 1
25. Conclusion
26. Conclusion › Understanding customers journey › Bezos' 70% decision rule › Backups are the last line of defense in the event of a disaster › GORM operates at a simple test framework › Simple bash testing framework to quickly evaluate and verify profiles
27. Thank you