mysql备份恢复方式,大致可分为3大类:
逻辑备份, mysqldump, mydumper等
物理备份,常用xtrabackup工具
快照备份,常见云实例,polardb等
上述几种备份方法及特点,有兴趣的可以自行搜索文档了解。
本文主要介绍一种新的备份方法,使用mysqlshell来备份及恢复数据,按照分类,mysqlshell备份属于逻辑备份。2020年7月,mysql官方在mysqlshell 8.0.21中,集成了实例和库级别的备份函数;2020年10月,8.0.22版本又发布了表级别备份函数。
mysqlshell备份工具非常强大,在云实例维护中,使用特别方便。云实例,服务器文件我们是无法访问的,因此,大数据量备份时,不能使用xtrabackup。逻辑备份常用的有:mysqldump,单线程,速度太慢,恢复也很慢;mydumper多线程并行备份和恢复,比较常用;现在我们又多了一种快速备份恢复工具——mysqlshell。
mysqlshell与其它备份方案对比,参考文末链接,备份恢复速度比mydumper要快,备份文件直接时压缩好的,压缩比还不错
mysqlshell简介
mysqlsh在日常使用的过程中,完全可替代mysql命令行客户端,且更强大,有python/js/sql三种交互方式。连接接方式如下
Usage examples:
$ mysqlsh root@localhost/schema
$ mysqlsh mysqlx://[email protected]:3307/world_x
$ mysqlsh --uri root@localhost --py -f sample.py sample param
$ mysqlsh root@targethost:33070 -s world_x -f sample.js
$ mysqlsh -- util check-for-server-upgrade root@localhost --output-format=JSON
$ mysqlsh mysqlx://user@host/db --import ~/products.json shop
mysql5.7以上版本,官方强烈建议使用mysqlshell 8.0。
mysql shell utilities
mysqlshell包含多种实用程序集(工具集),在 python/javascript模式下可用:
mysqlsh -- util --help
Error importing module 'requests', check if it's installed (Python 3.9.5)
The following operations are available at 'util':
check-for-server-upgrade
Performs series of tests on specified MySQL server to check if the
upgrade process will succeed.
dump-instance
Dumps the whole database to files in the output directory.
dump-schemas
Dumps the specified schemas to the files in the output directory.
dump-tables
Dumps the specified tables or views from the given schema to the files in
the target directory.
export-table
Exports the specified table to the data dump file.
import-json
Import JSON documents from file to collection or table in MySQL Server
using X Protocol session.
import-table
Import table dump stored in files to target table using LOAD DATA LOCAL
INFILE calls in parallel connections.
load-dump
Loads database dumps created by MySQL Shell.
check-for-server-upgrade 版本升级检查
dump-instance 备份实例
dump-schemas 备份指定schema
dump-tables 备份指定table
export-table 导出表至备份文件,类似select .... into outfile...
import-json 使用mysqlX将json文件导入表中
import-table 将备份文件数据导入表中,类似load data...
load-dump 恢复dump-tables/dump-schemas/dump-instance备份的数据,多线程load data...
mysqlshell备份数据
控制参数
dryRun: [ true | false ] 备份检查,是否实际执行备份。默认false,直接执行备份
threads:int默认4,备份mysql数据的线程数
maxRate: "string" 默认0,没有限制,每个线程的读吞吐,单位:k/M/G
defaultCharacterSet: "string" 默认utf8mb4
consistent: [ true | false ] 是否开启一执性快照备份,默认true。开启时,通过flush tables with read lock获取全局读锁,每个备份线程设置会话隔离级别RR,开启一致性快照事务。当所有线程开启事务后,获取实例级备份锁(lock intance for backup),释放全局读锁。
consistent参数,日常使用时需注意,true会有一个获取全局读锁的过程,业务繁忙实例,对业务影响比较大。
过滤参数
ddlOnly: [ true | false ] true只备份表结构,默认false
dataOnly: [ true | false ] true只备份数据,默认false
users: [ true | false ] dump-instance可用,默认true,备份用户,角色
excludeUsers: "userA@host, userB" dump-instance可用,不备份指定用户
includeUsers: "userA@host, userB" dump-instance可用,备份指定用户
excludeSchemas: "schemaA, schemaB" 不备份指定schema
includeSchemas: "schemaA, schemaB" 只备份指定schema
excludeTables: "dbA.tableA, dbaB.tableB" 不备份指定table
includeTables: "dbA.tableA, dbB.tableB" 只备份指定table
events: [ true | false ] 是否备份events
routines: [ true | false ] 默认true
all: [ true | false ] dump-tables可用,默认false。true时,设置表为空,备份所有视图和表
"hr", [], "emp", { "all": true }) util.dumpTables(
triggers: [ true | false ] 默认true
备份数据
dump-tables
# shell
mysqlsh root@127.0.0.1:3306 -- util dump-tables sbtest 'sbtest2,sbtest1' --output-url='/server/tmp/test' --threads=4 --consistent=false --dryRun=true
# python
[root@localhost ~ ]$ mysqlsh root@127.0.0.1:3306 --py
MySQL 127.0.0.1:3306 ssl Py > util.dump_tables('sbtest',['sbtest1','sbtest2'],'/server/tmp/test',{'threads':4,'consistent':'false','dryRun':'true'})
备份输出
Initializing - done
Gathering information - done
WARNING: The dumped value of gtid_executed is not guaranteed to be consistent
Writing global DDL files
Running data dump using 4 threads.
NOTE: Progress information uses estimated values and may not be accurate.
Writing schema metadata - done
Writing DDL - done
Writing table metadata - done
Starting data dump
(2.00M rows / ~1.97M rows), 1.28M rows/s, 227.01 MB/s uncompressed, 103.25 MB/s compressed
Dump duration: 00:00:01s
Total duration: 00:00:01s
Schemas dumped: 1
Tables dumped: 2
Uncompressed data size: 387.78 MB
Compressed data size: 176.31 MB
Compression ratio: 2.2
Rows written: 2000000
Bytes written: 176.31 MB
Average uncompressed throughput: 236.72 MB/s
Average compressed throughput: 107.63 MB/s
dump-schemas
# shell
mysqlsh root@127.0.0.1:3306 -- util dump-schemas 'sbtest,dba' --output-url='/server/tmp/test' --threads=4 --consistent=false --dryRun=true
# py
util.dump_schemas(['sbtest','dba'],'/server/tmp/test',{'threads':4,'consistent':'false','dryRun':'true'})
备份输出
Initializing - done
Gathering information - done
WARNING: The dumped value of gtid_executed is not guaranteed to be consistent
Writing global DDL files
Running data dump using 4 threads.
NOTE: Progress information uses estimated values and may not be accurate.
Writing schema metadata - done
Writing DDL - done
Writing table metadata - done
Starting data dump
(139.49M rows / ~138.29M rows), 641.85K rows/s, 158.22 MB/s uncompressed, 19.07 MB/s compressed
Dump duration: 00:01:31s
Total duration: 00:01:31s
Schemas dumped: 2
Tables dumped: 23
Uncompressed data size: 15.88 GB
Compressed data size: 4.02 GB
Compression ratio: 4.0
Rows written: 139489767
Bytes written: 4.02 GB
Average uncompressed throughput: 174.16 MB/s
Average compressed throughput: 44.08 MB/s
sbtest, dba库目录数据文件数据16G, 备份后数据3.8G,测试数据压缩比一般,生产数据10%左右。
dump-instance
# shell
mysqlsh root@127.0.0.1:3306 -- util dump-instance '/server/tmp/test' --threads=4 --consistent=false --dryRun=true
# py
util.dump_instance('/server/tmp/test',{'threads':4,'consistent':'false','dryRun':'true'})
mysqlshell load-dump恢复数据
控制参数
dryRun: 输出执行详细信息及步骤,true,不真正执行导入,默认false
waitDumpTimeout: 备份数据块恢复(导入)完成后,等新备份数据的超时时间,默认0,恢复完后,不再等待
schema: 指定要恢复到的schema,仅dump-tables产生的备份可用
threads: 并发数,恢复数据时,比较耗cpu:解压数据&导入数据
backgroundThreads: 后台读取文件内容的线程数,8.0.27版本引入,默认,从本机导入时,和threads相同,从远程服务器导入时,是threads的4倍
progressFile: 导入过程的状态文件,跟踪导入数据文件的导入进展,默认备份文件目录下的load-progress-server-di.json。设置为空时,无法断点续传。
resetProgress: [ true | false ] 忽略导入状态文件,重新从头导入
skipBinlog: [ true | false ] 不记录binlog
ignoreVersion: [ true | false ] 备份和恢复目标实例的主版本不同,true忽略版本不同错误,数据如果导入成功,则无版本兼容问题。
ignoreExistingObjects: [ true | false ] 忽略已存在的对象,慎用,无法判断与备份文件内容是否一致
characterSet: schema/instance dump默认备份产生的字符集,table dump默认utf8mb4
maxBytesPerTransaction:8.0.27,单次load data语句最大的导入字节数,单位k/M/G
数据参数
loadIndexes: [ true | false ] 默认true,创建二级索引
deferTableIndexes: [ off | fulltext | all ] 推迟二级索引的创建,可以节省恢复数据的时间。off导入时创建所有索引;fulltext仅推迟全文索引的创建;all推迟所有二级索引的创建,导入时只创建主键和(8.0.22起)包含自增列的唯一索引。8.0.21,如果有自增列的唯一索引,不要设置为all。
analyzeTables: [ off | on | histogram ] 是否执行analyze table,默认off,on所有表都执行,histogram只执行备份文件中包含histogram数据的表
showMetadata: [ true | false ] 以yaml格式打印源实例gtid/binlog信息。备份目录始终有gtid/binlog信息的文体
updateGtidSet: [ off | append | replace ] 在目标实例上追加/替换备份数据的gtid
过滤参数
loadDdl: [ true | false ] 默认true,导入ddl备份文件
loadData: [ true | false ] 默认true,导入数据文件
loadUsers: [ true | false ] 默认true,导入用户权限和角色
excludeUsers: 不导入指定用户和角色
includeUsers: 只导入指定用户和角色
excludeSchemas: 不导入指定schema。dump-instance备份时mysql/sys/information_schema/performance_schema这些库不备份
includeSchemas: 只导入指定的schema,可以和excludeSchemas同时使用
excludeTables: "dbA.tableA,dbB.tableB" 不导入指定表
includeTables: "dbA.tableA,dbB.tableB" 只导入指定表
events/triggers/routines 这里不做过多介绍,参考文末文档
恢复数据
恢复dump-schemas/dump-instace备份的数据
mysqlsh root@127.0.0.1:3306 -- util load-dump '/server/tmp/test' --analyzeTables=on --skipBinlog=true --updateGtidSet=off --dryRun=true
util.load_dump('/server/tmp/test',{'threads':4,'analyzeTables':'on','skipBinlog':'true','updateGtidSet':'off','dryRun':'false'})
输出
Loading DDL and Data from '/server/tmp/test' using 4 threads.
Opening dump...
Target is MySQL 8.0.25. Dump was produced from MySQL 8.0.25
Scanning metadata - done
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL - done
Executing view DDL - done
Starting data load
2 thds loading / 100% (15.88 GB / 15.88 GB), 29.03 MB/s, 23 / 23 tables done
Recreating indexes - done
Analyzing tables - done
Executing common postamble SQL
241 chunks (139.49M rows, 15.88 GB) for 23 tables in 2 schemas were loaded in 5 min 45 sec (avg throughput 46.00 MB/s)
0 warnings were reported during the load.
dump-tables备份sbtest库sbtest1/sbtest2的数据,恢复至dba库
mysqlsh root@127.0.0.1:3306 -- util load-dump '/server/tmp/test2' --analyzeTables=on --skipBinlog=true --updateGtidSet=off --schema=dba --dryRun=false
util.load_dump('/server/tmp/test2',{'threads':4,'analyzeTables':'on','skipBinlog':'true','updateGtidSet':'off','schema':'dba','dryRun':'true'})
输出
Loading DDL and Data from '/server/tmp/test2' using 4 threads.
Opening dump...
Target is MySQL 8.0.25. Dump was produced from MySQL 8.0.25
Scanning metadata - done
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL - done
Executing view DDL - done
Starting data load
2 thds loading / 100% (387.78 MB / 387.78 MB), 37.78 MB/s, 2 / 2 tables done
Executing common postamble SQL
Recreating indexes - done
Analyzing tables - done
8 chunks (2.00M rows, 387.78 MB) for 2 tables in 1 schemas were loaded in 9 sec (avg throughput 44.27 MB/s)
0 warnings were reported during the load.
参考
https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-utilities-dump-instance-schema.html#mysql-shell-utilities-dump-opt-control
https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-utilities-load-dump.html
https://www.percona.com/blog/backup-restore-performance-conclusion-mysqldump-vs-mysql-shell-utilities-vs-mydumper-vs-mysqlpump-vs-xtrabackup/
https://www.percona.com/blog/dump-performance-comparison-mysqldump-vs-mysql-shell-utilities-vs-mydumper/