How to Fast Verify Data Backups for ISO certification review @ TECHPULSE 2023

- Speaker: Winter Hung
- Event: http://techpulse.line.me/

備份就像保險一樣，有效備份可以減少停機時間與節省災難期間所帶來的隱藏成本。為達到快速恢復，必須有良好的配置文件，本次將介紹 LINE SHOPPING 如何透過 GORM 和 simple bash testing framework 來確保與驗證在數據備份的正確性。

Transcript

1. View Slide

2. Agenda
   › LINE SHOPPING
   › Data Backup Analysis
   › Simple Bash Testing Framework
   › GORM & Jenkins Integration
   › Conclusion
   

   View Slide

3. LINE SHOPPING
   Popular e-commerce platform
   3
   

   View Slide

4. › Aggregation Service
   › LINE Points Back
   › Search & Price Comparison
   › Lowest Price Product
   › LINE Official Account
   › ISO Certification
   LINE SHOPPING
   

   View Slide

5. LINE SHOPPING
   ORDER FLOW
   Homepage
   Partner Transform Order
   5
   1
   1
   

   View Slide

6. Order User Journey
   Visual Representation
   Database
   CPS
   Report
   Database
   CMS
   Log
   Database
   Order
   Log
   Database
   ECID
   Log
   Partner
   Fee
   Postback
   Partner
   Order
   Postback
   1 2 3 4 5 6
   

   View Slide

7. Data Backup Analysis
   Migrate Database
   

   View Slide

8. ISO Certification Review
   Information Security Management
   ISO
   27001
   Make Decisions
   70%
   MySQLdump
   100%
   

   View Slide

9. MySQL Data Backup
   Architecture
   MYSQL Master
   Real MySQL Slave Backup MySQL
   Slave
   InnoDB
   DBA
   

   View Slide

10. Check backup data
    Simple Bash Testing
    Framework
    

    View Slide

11. How to Data verification
    Simple Bash Testing Framework
    Data verification is a process in which different types of data are checked for accuracy and
    inconsistencies after data migration is done. So the LINE SHOPPING team uses a simple
    bash testing framework and writes tests that run bash commands, and reports on any
    failures. Each test is its own bash script
    

    View Slide

12. Simple Bash Testing Framework
    Minimal Test Script
    # Load the testing framework
    . ./testlib.sh
    TEST_SUTE_NAME=“Test something”
    #Number of tests
    TOTAL_TEST=1
    # PUT your tests here
    assert “Sample test” ‘true’
    # Print results
    1
    2
    3
    4
    5
    

    View Slide

13. Simple Bash Testing Framework
    Minimal Test Script
    # Load the testing framework
    . ./testlib.sh
    TEST_SUTE_NAME=“Test something”
    #Number of tests
    TOTAL_TEST=1
    # PUT your tests here
    assert “Sample test” ‘true’
    # Print results
    1
    2
    3
    4
    5
    

    View Slide

14. Simple Bash Testing Framework
    Minimal Test Script
    # Load the testing framework
    . ./testlib.sh
    TEST_SUTE_NAME=“Test something”
    # Number of tests
    TOTAL_TEST=1
    # PUT your tests here
    assert “Sample test” ‘true’
    # Print results
    1
    2
    3
    4
    5
    

    View Slide

15. Simple Bash Testing Framework
    Minimal Test Script
    # Load the testing framework
    . ./testlib.sh
    TEST_SUTE_NAME=“Test something”
    # Number of tests
    TOTAL_TEST=1
    # PUT your tests here
    assert “Sample test” ‘true’
    # Print results
    1
    2
    3
    4
    5
    

    View Slide

16. Simple Bash Testing Framework
    Minimal Test Script
    # Load the testing framework
    . ./testlib.sh
    TEST_SUTE_NAME=“Test something”
    # Number of tests
    TOTAL_TEST=1
    # PUT your tests here
    assert “Sample test” ‘true’
    # Print results
    1
    2
    3
    4
    5
    

    View Slide

17. Simple Bash Testing Framework
    Contains Several Parts
    › Set the name of the test suite. This will be displayed in the report
    › Set the total number of tests, If this amount of tests aren’t run
    › A Line to load the testing library for TAP（Test Anything Protocol）
    › The tests themselves
    › A command to print the results at the end
    

    View Slide

18. Simple Bash Testing Framework
    Test Commands
    Assert “Test Name” “command” to test something
    Assert_not_ran_ok “Test Name” to check the return value of the last
    Assert_ran_ok “Test Name” to test a condition
    

    View Slide

19. Continuous Validation Data Testing
    GORM & Jenkins
    Integration
    

    View Slide

20. GORM and Jenkins Integration
    › Golang Object-Relational Mapping
    › Manipulate data from a database
    › Data defined in classes to SQL
    › CRUD functions already
    implemented
    › Popular choice for Go developers
    

    View Slide

21. OVERVIEW
    Gin + GORM + MySQL
    MySQL Slave
    Client Gin GORM
    Client
    Client
    Restful Web Service
    

    View Slide

22. MySQL Backup Validation
    Architecture
    Backup MySQL
    Slave
    Real MySQL Slave
    Jenkins
    Bash Test
    Framework Real GORM
    Result
    Backup GORM
    RESTful Web
    Service
    Client
    

    View Slide

23. MySQL Backup Validation
    Minimal Test Script
    # Load the testing framework
    . ./testlib.sh
    #REAL ORM URL
    real_orm_url=“http://real_mysql_slave.orm.url:9002”
    #BACKUP ORM URL
    backup_orm_url=“http://backup_mysql_slave.orm.url:9003”
    # assert data
    assert_not_ran_ok $real_data $backup_data
    # Print results
    

    View Slide

24. MySQL Backup Validation
    Print Result
    1
    

    View Slide

25. Conclusion
    

    View Slide

26. Conclusion
    › Understanding customers journey
    › Bezos' 70% decision rule
    › Backups are the last line of defense in the event of a disaster
    › GORM operates at a simple test framework
    › Simple bash testing framework to quickly evaluate and verify profiles
    

    View Slide

27. Thank you
    

    View Slide