Share
Embed
Testing and verifying your MySQL backup strategy
We all want to avoid the loss of valuable information and minimize downtime.
Your backup procedure is only as good as the ability to successfully restore your data.
This presentation will benefit DBAs responsible for maintaining MySQL systems whether you are well qualified with MySQL or Oracle.
This presentation will review the essential configuration options highlighting the common mistakes seen with many clients and providing a number of scenarios that can be reproduced to test your current and future strategies.
Ronald Bradford
November 09, 2016
Transcript
61 slides
-
Testing and Verifying your MySQL Backup Strategy EffectiveMySQL.com - Performance,
Scalability, Site Reliability Ronald Bradford http://ronaldbradford.com @RonaldBradford 2016.11 -
EffectiveMySQL.com - Performance, Scalability, Site Reliability About the Author 17
years with MySQL / 27 years with RDBMS Senior Consultant at MySQL Inc (06-08) Consultant for Oracle Corporation (96-99) Published author of 4 MySQL books Ronald BRADFORD http://ronaldbradford.com/presentations/ -
EffectiveMySQL.com - Performance, Scalability, Site Reliability "No one cares if
you can backup, only that you can restore." Adapted from W. Curtis Preston - Backup & Recovery (O'Reilly 2009) -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Problem To verify your
backup is to restore Test frequently Test for exceptions Restoring is a two part process Restore static backup Restore to point in time -
EffectiveMySQL.com - Performance, Scalability, Site Reliability "Testing is about breaking
your software. Testing is not about checking if it works." Ronald Bradford, 2006 -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Requirements? What is most
important? Your restore completes successfully? Your restore matches your backup files? Your restored system has all the data it should have? Your system is back online ASAP? All of these requirements are necessary for success -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Constraints? What part takes
the most time to restore? Copying your backup files across network? Uncompressing your backup? Restore the backup? Point in time recovery? -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Options mysqldump mydumper
XtraBackup MySQL Enterprise Backup LVM/SAN Snapshot Filesystem copy New options review -
EffectiveMySQL.com - Performance, Scalability, Site Reliability mysqldump Included with binary
distribution Open source Maintained by Oracle Only tool for schema information Poor defaults -
EffectiveMySQL.com - Performance, Scalability, Site Reliability mysqldump Locks schemas (by
default) Inconsistent for multi schema applications Does not obtain position needed for point in time recovery Does not backup all objects by default Single threaded Reads all data (into database buffers) Defaults -
EffectiveMySQL.com - Performance, Scalability, Site Reliability mysqldump ASCII copy of
data editable Cross platform/version compatible Can perform partial backups -
EffectiveMySQL.com - Performance, Scalability, Site Reliability mydumper Parallel implementation based
on mysqldump Open source Community developed and maintained Max Bubenick - Percona https://launchpad.net/mydumper -
EffectiveMySQL.com - Performance, Scalability, Site Reliability xtrabackup Open source Maintained
by 3rd party company Widely used for large InnoDB installations https://www.percona.com/doc/percona-xtrabackup/ -
EffectiveMySQL.com - Performance, Scalability, Site Reliability xtrabackup Non-blocking backup (of
transactional data) Consistent file copy -
EffectiveMySQL.com - Performance, Scalability, Site Reliability MEB Commercial license bundled
with subscription support Oracle provided enterprise version Features similar to Xtrabackup MySQL enterprise backup -
EffectiveMySQL.com - Performance, Scalability, Site Reliability snapshot LVM/SAN Snapshot DB
agnostic MyLVMBackup Open source Community Maintained http://www.lenzg.net/mylvmbackup/ -
EffectiveMySQL.com - Performance, Scalability, Site Reliability snapshot Read Consistent view
FLUSH TABLES WITH READ LOCK Can take a long time on highly concurrent systems -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Latest mysqlpump (5.7.8) Open
source Developed by Oracle rewrite of mysqldump parallel implementation of mysqldump https://dev.mysql.com/doc/refman/5.7/en/mysqlpump.html -
EffectiveMySQL.com - Performance, Scalability, Site Reliability point in time MySQL
Binary Logs Are they enabled? Binary log position Obtained consistently SaaS (per schema issues) -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Options cp rsync
Use a slave --log-slave-updates DRBD - Disk Replicated Block Device mysqlbinlog --read-from-remote-server (New in 5.6) -
EffectiveMySQL.com - Performance, Scalability, Site Reliability binary log file copy
Binary logs are sequential files cp/rsync mysqladmin flush-logs e.g. AWS 5 minutes strategy -
EffectiveMySQL.com - Performance, Scalability, Site Reliability mirrored binary log Remote
copy binary logs as created No automatic restart management Requires MySQL 5.6+ (can be on a slave) -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Design choices Time to
backup Time to restore Consistency Flexibility Partial Capabilities Cost -
EffectiveMySQL.com - Performance, Scalability, Site Reliability storage engines InnoDB only
MyISAM mysql schema Other storage engines Mixed data solutions -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Requirements Schema & Data
Schema Size/Tables etc Replication architecture Auditing Security Checksums Storage Engines Do these help to test and verify? -
EffectiveMySQL.com - Performance, Scalability, Site Reliability schema & objects Can
only be obtained via mysqldump Required for test systems & subsets Auditability of schema changes -
EffectiveMySQL.com - Performance, Scalability, Site Reliability meta info Schema size
Tables size Table objects -
EffectiveMySQL.com - Performance, Scalability, Site Reliability meta info SELECT FORMAT(SUM(data_length+index_length)/1024/1024,2)
AS total_mb, FORMAT(SUM(data_length)/1024/1024,2) AS data_mb, FORMAT(SUM(index_length)/1024/1024,2) AS index_mb, COUNT(DISTINCT table_schema) AS schema_cnt, COUNT(*) AS tables, CURDATE() AS today, VERSION() FROM information_schema.tables\G *************************** 1. row *************************** total_mb: 5344.63 data_mb: 4545.49 index_mb: 799.13 schema_cnt: 7 tables: 103 today: 2012-04-03 VERSION(): 5.1.61-0ubuntu0.11.10.1-log Your daily verification step should include this ALL Schemas Size https://github.com/ronaldbradford/mysql/blob/master/sql/database_size.sql -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Meta Info DROP TABLE
IF EXISTS db_size; CREATE TABLE db_size( table_schema VARCHAR(64) NOT NULL, table_name VARCHAR(64) NOT NULL, engine VARCHAR(64) NOT NULL, row_format VARCHAR(10) NULL, table_rows INT UNSIGNED NOT NULL, avg_row INT UNSIGNED NOT NULL, total_mb DECIMAL(7,2) NOT NULL, data_mb DECIMAL(7,2) NOT NULL, index_mb DECIMAL(7,2) NOT NULL, created_date DATETIME NOT NULL, INDEX (created_date,table_name) ) ENGINE=InnoDB; Table size Keep a copy of all tablesize info per backup https://github.com/ronaldbradford/mysql/blob/master/sql/db_size_table.sql -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Meta Info DELIMITER $$
DROP PROCEDURE IF EXISTS generate_db_size$$ CREATE PROCEDURE generate_db_size() BEGIN DECLARE l_created_date DATETIME; SET l_created_date := NOW(); INSERT INTO db_size(table_schema, table_name, engine, row_format, table_rows, avg_row, total_mb, data_mb, index_mb, created_date) SELECT table_schema, table_name, engine, row_format, table_rows, avg_row_length AS avg_row, ROUND((data_length+index_length)/1024/1024,2) AS total_mb, ROUND((data_length)/1024/1024,2) AS data_mb, ROUND((index_length)/1024/1024,2) AS index_mb, l_created_date FROM information_schema.tables WHERE table_schema=DATABASE() AND table_type='BASE TABLE' ORDER BY 6 DESC; SELECT l_created_date AS created_date, DATABASE() AS table_schema, COUNT(*) AS tables, FORMAT(SUM(total_mb),2) AS total_mb, FORMAT(SUM(data_mb),2) AS data_mb, FORMAT(SUM(index_mb),2) AS index_mb FROM db_size WHERE created_date = l_created_date; Table size https://github.com/ronaldbradford/mysql/blob/master/sql/generate_db_size.sql -
EffectiveMySQL.com - Performance, Scalability, Site Reliability architecture Do you backup
the master or the slave? Binary position option changes Data drift? Do you backup the entire DB? Is some data static? Can a recovered system start without need for all data? replication topology -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Options $ sudo
su - $ sync ; lvcreate -L1G -s -n dbsnapshot /dev/db/p0 $ mkdir -p /mnt/dbsnapshot $ mount -o ro /dev/db/dbsnapshot /mnt/dbsnapshot $ du -sh /mnt/dbsnapshot $ ls -al /mnt/dbsnapshot $ mkdir /mysql/backup/snapshot1 $ cp -r /mnt/dbsnapshot/* /mysql/backup/snapshot1 $ sudo su - $ mylvmbackup SNAPSHOT USAGE http://effectiveMySQL.com/article/configuring-a-new-hard-drive-for-lvm http://effectiveMySQL.com/article/using-mysql-with-lvm http://www.lenzg.net/mylvmbackup/ TIP: mylvmbackup does all the hard work -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Situations Incomplete backup file
Missing objects Locking Missing point in time position Restore time considerations Fake objects Trigger hell ^C does not kill mysqldump -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Options $ time
mysqldump --all-databases > /mysql/backup/dump1.sql real 1m31.631s user 1m12.533s sys 0m10.893s $ echo $? 0 $ ls -lh /mysql/backup/dump1.sql -rw-rw-r-- 1 uid gid 2.9G 2012-04-03 03:04 /mysql/ backup/dump1.sql mysqldump USAGE -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Options $ time
mysqldump --all-databases > /mysql/backup/dump1.sql mysqldump: Got error: 1142: SELECT,LOCK TABL command denied to user 'root'@'localhost' for table 'cond_instances' when using LOCK TABLES $ echo $? 2 VERIFICATION TIP: Error checking is essential and easy to implement -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Error examples mysqldump: Couldn't
execute 'SHOW FIELDS FROM `alarm`': View 'schema.alarm' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them (1356) mysqldump: Got error: 1044: Access denied for user 'rbradfor'@'localhost' to database 'schema' when using LOCK TABLES Invalid objects Lack of permissions -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Options $ time
mysqldump --all-databases \ --events --routines > /mysql/backup/dump1.sql mysqldump USAGE All objects are NOT included by default -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Options $ time
mysqldump --all-databases \ --events --routines \ --single-transaction > /mysql/backup/dump1.sql mysqldump USAGE Remove default schema locking -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Options $ time
mysqldump --all-databases \ --events --routines \ --single-transaction \ --master-data > /mysql/backup/dump1.sql mysqldump USAGE Point in Time recovery position $ time mysqldump --all-databases \ --events --routines \ --single-transaction \ --dump-slave > /mysql/backup/dump1.sql mysqldump: Error: Binlogging on server not active Fail on test system? Different for master or slave -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup improvment # Schema
$ mysqldump --all-databases --no-data \ --skip-triggers --add-drop-database \ > /mysql/backup/schema.sql # Objects $ mysqldump --all-databases --no-data --no-create-info \ --events --routines --triggers \ > /mysql/backup/objects.sql # Just the data $ mysqldump --all-databases --no-create-info \ --single-transaction --master-data --skip-triggers \ > /mysql/backup/data.sql Separate schema/objects TIP: Include daily dumps of database objects -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Restore Options $ time
mysqldump --all-databases > /mysql/backup/dump1.sql real 1m31.631s ... $ time mysql –u[user] -p -f < dump1.sql > dump1.out 2>&1 real 14m13.817s user 1m6.960s sys 0m1.516s $ echo $? 0 $ ls -l dump1.out -rw-rw-r-- 1 uid gid 0 2012-04-08 04:07 dump1.out mysqldump USAGE Continue on error or stop on error? -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Fake objects DROP TABLE
IF EXISTS `demo`; /*!50001 DROP VIEW IF EXISTS `demo`*/; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; /*!50001 CREATE TABLE `demo` ( `zip` tinyint NOT NULL, `lat` tinyint NOT NULL, `lon` tinyint NOT NULL ) ENGINE=MyISAM */; SET character_set_client = @saved_cs_client; /*!50001 DROP TABLE IF EXISTS `demo`*/; /*!50001 DROP VIEW IF EXISTS `demo`*/; /*!50001 CREATE ALGORITHM=UNDEFINED */ /*!50013 DEFINER=`root`@`localhost` SQL SECURITY DEFINER */ /*!50001 VIEW `demo` AS select `uszip`.`zip` AS `zip`,`uszip`.`lat` AS `lat`,`uszip`.`lon` AS `lon` from `uszip` */; mysqldump USAGE -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Fake objects -- --
Temporary view structure for view `demo` -- DROP TABLE IF EXISTS `demo`; /*!50001 DROP VIEW IF EXISTS `demo`*/; /*!50001 CREATE VIEW `demo` AS SELECT 1 AS `zip`, 1 AS `lat`, 1 AS `lon`*/; ... -- -- Final view structure for view `demo` -- /*!50001 DROP VIEW IF EXISTS `demo`*/; /*!50001 CREATE ALGORITHM=UNDEFINED */ /*!50013 DEFINER=`root`@`localhost` SQL SECURITY DEFINER */ /*!50001 VIEW `demo` AS select `uszip`.`zip` AS `zip`,`uszip`.`lat` AS `lat`,`uszip`.`lon` AS `lon` from `uszip` */; mysqldump USAGE 5.7 syntax -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Compression How long to
compress the data? Which compression do you use? How long to uncompress the data? 95% of recovery is from last backup Best strategy is to have an uncompressed copy on server. -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Compression Utility Comp (s)
Dec (s) Saving lzo (-3) 21 34 48% pigz (-1) 43 33 64% pigz [-6] 105 25 69% gzip [-6] 232 78 69% bzip2 540 175 74% lzo (-9) 20m 82 58% lzma 58m 180 78% xz 59m 160 78% Depends greatly on data types -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup needs Static Backup
Useless without Binary Log position mysql> SHOW MASTER STATUS\G File: mysql-bin.020616 Position: 63395562 Binlog_Do_DB: Binlog_Ignore_DB: WARNING: Can work on slave and provide the wrong information -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Backup Needs $ mysqldump
--master-data (or --dump-slave) CHANGE MASTER TO MASTER_LOG_FILE= 'mysql-bin.000146', MASTER_LOG_POS=810715371; $ cat xtrabackup_binlog_info mysql-bin.000001 37522 $ grep binlog meta/backup_variables.txt binlog_position=mysql-bin.000017:5555 MEB Xtrabackup $ cat export-20120407-230027/metadata Log: mysql-bin.000017 Pos: 8328 mydumper -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Situations Lost connection Transaction
too large ROW format statement SET max_allowed_packet Single threaded Binary Log -
EffectiveMySQL.com - Performance, Scalability, Site Reliability what about Failover Is
failover a better option? Do you have high availability (HA)implemented? Top Benefit Improved time to recovery Top Risk [Potential] loss of high availability -
EffectiveMySQL.com - Performance, Scalability, Site Reliability failover issues Failover does
not cater for all situations Data recovery to a prior point in time e.g. security problem building testing systems Building out replicas Entirely different set of problems to verify -
EffectiveMySQL.com - Performance, Scalability, Site Reliability What to consider Forget
the technology? What is most important for your business? B&R/DR decision should be based on loss/ cost to business RPO/RDO/SLA -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Terminology MTTD - Mean
Time To Detect MTTR - Mean Time to Recover RPO - Recovery Point Objective RDO - Recovery Data Objective SLA - Service Level Agreement Determining business priorities is important for any strategy -
EffectiveMySQL.com - Performance, Scalability, Site Reliability Conclusion B&R is a
complex process Business will want all of the data, all of the time Justify the business needs first Discounts [easier] options