vlambda博客
学习文章列表

mysql备份恢复神器----mysqlshell

mysql备份恢复方式,大致可分为3大类:

  • 逻辑备份, mysqldump, mydumper等

  • 物理备份,常用xtrabackup工具

  • 快照备份,常见云实例,polardb等

上述几种备份方法及特点,有兴趣的可以自行搜索文档了解。

本文主要介绍一种新的备份方法,使用mysqlshell来备份及恢复数据,按照分类,mysqlshell备份属于逻辑备份。2020年7月,mysql官方在mysqlshell 8.0.21中,集成了实例和库级别的备份函数;2020年10月,8.0.22版本又发布了表级别备份函数。


mysqlshell备份工具非常强大,在云实例维护中,使用特别方便。云实例,服务器文件我们是无法访问的,因此,大数据量备份时,不能使用xtrabackup。逻辑备份常用的有:mysqldump,单线程,速度太慢,恢复也很慢;mydumper多线程并行备份和恢复,比较常用;现在我们又多了一种快速备份恢复工具——mysqlshell。


mysqlshell与其它备份方案对比,参考文末链接,备份恢复速度比mydumper要快,备份文件直接时压缩好的,压缩比还不错

mysqlshell简介

mysqlsh在日常使用的过程中,完全可替代mysql命令行客户端,且更强大,有python/js/sql三种交互方式。连接接方式如下

Usage examples:$ mysqlsh root@localhost/schema$ mysqlsh mysqlx://[email protected]:3307/world_x$ mysqlsh --uri root@localhost --py -f sample.py sample param$ mysqlsh root@targethost:33070 -s world_x -f sample.js$ mysqlsh -- util check-for-server-upgrade root@localhost --output-format=JSON$ mysqlsh mysqlx://user@host/db --import ~/products.json shop

mysql5.7以上版本,官方强烈建议使用mysqlshell 8.0。

mysql shell utilities

mysqlshell包含多种实用程序集(工具集),在 python/javascript模式下可用:

mysqlsh -- util --help
Error importing module 'requests', check if it's installed (Python 3.9.5)The following operations are available at 'util':
check-for-server-upgrade Performs series of tests on specified MySQL server to check if the upgrade process will succeed.
dump-instance Dumps the whole database to files in the output directory.
dump-schemas Dumps the specified schemas to the files in the output directory.
dump-tables Dumps the specified tables or views from the given schema to the files in the target directory.
export-table Exports the specified table to the data dump file.
import-json Import JSON documents from file to collection or table in MySQL Server using X Protocol session.
import-table Import table dump stored in files to target table using LOAD DATA LOCAL INFILE calls in parallel connections.
load-dump Loads database dumps created by MySQL Shell.
  • check-for-server-upgrade 版本升级检查

  • dump-instance 备份实例

  • dump-schemas 备份指定schema

  • dump-tables 备份指定table

  • export-table 导出表至备份文件,类似select .... into outfile...

  • import-json 使用mysqlX将json文件导入表中

  • import-table 将备份文件数据导入表中,类似load data...

  • load-dump 恢复dump-tables/dump-schemas/dump-instance备份的数据,多线程load data...

mysqlshell备份数据

控制参数

  • dryRun: [ true | false ] 备份检查,是否实际执行备份。默认false,直接执行备份

  • threads:int默认4,备份mysql数据的线程数

  • maxRate: "string" 默认0,没有限制,每个线程的读吞吐,单位:k/M/G

  • defaultCharacterSet: "string" 默认utf8mb4

  • consistent: [ true | false ] 是否开启一执性快照备份,默认true。开启时,通过flush tables with read lock获取全局读锁,每个备份线程设置会话隔离级别RR,开启一致性快照事务。当所有线程开启事务后,获取实例级备份锁(lock intance for backup),释放全局读锁。

consistent参数,日常使用时需注意,true会有一个获取全局读锁的过程,业务繁忙实例,对业务影响比较大。

过滤参数

  • ddlOnly: [ true | false ] true只备份表结构,默认false

  • dataOnly: [ true | false ] true只备份数据,默认false

  • users: [ true | false ] dump-instance可用,默认true,备份用户,角色

  • excludeUsers: "userA@host, userB" dump-instance可用,不备份指定用户

  • includeUsers: "userA@host, userB" dump-instance可用,备份指定用户

  • excludeSchemas: "schemaA, schemaB" 不备份指定schema

  • includeSchemas: "schemaA, schemaB" 只备份指定schema

  • excludeTables: "dbA.tableA, dbaB.tableB" 不备份指定table

  • includeTables: "dbA.tableA, dbB.tableB" 只备份指定table

  • events: [ true | false ] 是否备份events

  • routines: [ true | false ] 默认true

  • all: [ true | false ] dump-tables可用,默认false。true时,设置表为空,备份所有视图和表

shell-js> util.dumpTables("hr", [], "emp", { "all": true })
  • triggers: [ true | false ] 默认true

备份数据

  • dump-tables

# shellmysqlsh root@127.0.0.1:3306 -- util dump-tables sbtest 'sbtest2,sbtest1' --output-url='/server/tmp/test' --threads=4 --consistent=false --dryRun=true
# python[root@localhost ~ ]$ mysqlsh root@127.0.0.1:3306 --py
MySQL 127.0.0.1:3306 ssl Py > util.dump_tables('sbtest',['sbtest1','sbtest2'],'/server/tmp/test',{'threads':4,'consistent':'false','dryRun':'true'})

备份输出

Initializing - doneGathering information - doneWARNING: The dumped value of gtid_executed is not guaranteed to be consistentWriting global DDL filesRunning data dump using 4 threads.NOTE: Progress information uses estimated values and may not be accurate.Writing schema metadata - doneWriting DDL - doneWriting table metadata - doneStarting data dump101% (2.00M rows / ~1.97M rows), 1.28M rows/s, 227.01 MB/s uncompressed, 103.25 MB/s compressedDump duration: 00:00:01sTotal duration: 00:00:01sSchemas dumped: 1Tables dumped: 2Uncompressed data size: 387.78 MBCompressed data size: 176.31 MBCompression ratio: 2.2Rows written: 2000000Bytes written: 176.31 MBAverage uncompressed throughput: 236.72 MB/sAverage compressed throughput: 107.63 MB/s
  • dump-schemas

# shellmysqlsh root@127.0.0.1:3306 -- util dump-schemas 'sbtest,dba' --output-url='/server/tmp/test' --threads=4 --consistent=false --dryRun=true
# pyutil.dump_schemas(['sbtest','dba'],'/server/tmp/test',{'threads':4,'consistent':'false','dryRun':'true'})

备份输出

Initializing - doneGathering information - doneWARNING: The dumped value of gtid_executed is not guaranteed to be consistentWriting global DDL filesRunning data dump using 4 threads.NOTE: Progress information uses estimated values and may not be accurate.Writing schema metadata - doneWriting DDL - doneWriting table metadata - doneStarting data dump100% (139.49M rows / ~138.29M rows), 641.85K rows/s, 158.22 MB/s uncompressed, 19.07 MB/s compressedDump duration: 00:01:31sTotal duration: 00:01:31sSchemas dumped: 2Tables dumped: 23Uncompressed data size: 15.88 GBCompressed data size: 4.02 GBCompression ratio: 4.0Rows written: 139489767Bytes written: 4.02 GBAverage uncompressed throughput: 174.16 MB/sAverage compressed throughput: 44.08 MB/s

sbtest, dba库目录数据文件数据16G, 备份后数据3.8G,测试数据压缩比一般,生产数据10%左右。

  • dump-instance

# shellmysqlsh root@127.0.0.1:3306 -- util dump-instance '/server/tmp/test' --threads=4 --consistent=false --dryRun=true
# pyutil.dump_instance('/server/tmp/test',{'threads':4,'consistent':'false','dryRun':'true'})

mysqlshell load-dump恢复数据

控制参数

  • dryRun: 输出执行详细信息及步骤,true,不真正执行导入,默认false

  • waitDumpTimeout: 备份数据块恢复(导入)完成后,等新备份数据的超时时间,默认0,恢复完后,不再等待

  • schema: 指定要恢复到的schema,仅dump-tables产生的备份可用

  • threads: 并发数,恢复数据时,比较耗cpu:解压数据&导入数据

  • backgroundThreads: 后台读取文件内容的线程数,8.0.27版本引入,默认,从本机导入时,和threads相同,从远程服务器导入时,是threads的4倍

  • progressFile: 导入过程的状态文件,跟踪导入数据文件的导入进展,默认备份文件目录下的load-progress-server-di.json。设置为空时,无法断点续传。

  • resetProgress: [ true | false ] 忽略导入状态文件,重新从头导入

  • skipBinlog: [ true | false ] 不记录binlog

  • ignoreVersion: [ true | false ] 备份和恢复目标实例的主版本不同,true忽略版本不同错误,数据如果导入成功,则无版本兼容问题。

  • ignoreExistingObjects: [ true | false ] 忽略已存在的对象,慎用,无法判断与备份文件内容是否一致

  • characterSet: schema/instance dump默认备份产生的字符集,table dump默认utf8mb4

  • maxBytesPerTransaction:8.0.27,单次load data语句最大的导入字节数,单位k/M/G

数据参数

  • loadIndexes: [ true | false ] 默认true,创建二级索引

  • deferTableIndexes: [ off | fulltext | all ] 推迟二级索引的创建,可以节省恢复数据的时间。off导入时创建所有索引;fulltext仅推迟全文索引的创建;all推迟所有二级索引的创建,导入时只创建主键和(8.0.22起)包含自增列的唯一索引。8.0.21,如果有自增列的唯一索引,不要设置为all。

  • analyzeTables: [ off | on | histogram ] 是否执行analyze table,默认off,on所有表都执行,histogram只执行备份文件中包含histogram数据的表

  • showMetadata: [ true | false ] 以yaml格式打印源实例gtid/binlog信息。备份目录始终有gtid/binlog信息的文体

  • updateGtidSet: [ off | append | replace ] 在目标实例上追加/替换备份数据的gtid

过滤参数

  • loadDdl: [ true | false ] 默认true,导入ddl备份文件

  • loadData: [ true | false ] 默认true,导入数据文件

  • loadUsers: [ true | false ] 默认true,导入用户权限和角色

  • excludeUsers: 不导入指定用户和角色

  • includeUsers: 只导入指定用户和角色

  • excludeSchemas: 不导入指定schema。dump-instance备份时mysql/sys/information_schema/performance_schema这些库不备份

  • includeSchemas: 只导入指定的schema,可以和excludeSchemas同时使用

  • excludeTables: "dbA.tableA,dbB.tableB" 不导入指定表

  • includeTables: "dbA.tableA,dbB.tableB" 只导入指定表

  • events/triggers/routines 这里不做过多介绍,参考文末文档

恢复数据

  • 恢复dump-schemas/dump-instace备份的数据

# shellmysqlsh root@127.0.0.1:3306 -- util load-dump '/server/tmp/test' --analyzeTables=on --skipBinlog=true --updateGtidSet=off --dryRun=true
# pyutil.load_dump('/server/tmp/test',{'threads':4,'analyzeTables':'on','skipBinlog':'true','updateGtidSet':'off','dryRun':'false'})

输出

Loading DDL and Data from '/server/tmp/test' using 4 threads.Opening dump...Target is MySQL 8.0.25. Dump was produced from MySQL 8.0.25Scanning metadata - doneChecking for pre-existing objects...Executing common preamble SQLExecuting DDL - doneExecuting view DDL - doneStarting data load2 thds loading / 100% (15.88 GB / 15.88 GB), 29.03 MB/s, 23 / 23 tables doneRecreating indexes - doneAnalyzing tables - doneExecuting common postamble SQL241 chunks (139.49M rows, 15.88 GB) for 23 tables in 2 schemas were loaded in 5 min 45 sec (avg throughput 46.00 MB/s)0 warnings were reported during the load.
  • dump-tables备份sbtest库sbtest1/sbtest2的数据,恢复至dba库

# shellmysqlsh root@127.0.0.1:3306 -- util load-dump '/server/tmp/test2' --analyzeTables=on --skipBinlog=true --updateGtidSet=off --schema=dba --dryRun=false
# pyutil.load_dump('/server/tmp/test2',{'threads':4,'analyzeTables':'on','skipBinlog':'true','updateGtidSet':'off','schema':'dba','dryRun':'true'})

输出

Loading DDL and Data from '/server/tmp/test2' using 4 threads.Opening dump...Target is MySQL 8.0.25. Dump was produced from MySQL 8.0.25Scanning metadata - doneChecking for pre-existing objects...Executing common preamble SQLExecuting DDL - doneExecuting view DDL - doneStarting data load2 thds loading / 100% (387.78 MB / 387.78 MB), 37.78 MB/s, 2 / 2 tables doneExecuting common postamble SQLRecreating indexes - doneAnalyzing tables - done8 chunks (2.00M rows, 387.78 MB) for 2 tables in 1 schemas were loaded in 9 sec (avg throughput 44.27 MB/s)0 warnings were reported during the load.


参考

https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-utilities-dump-instance-schema.html#mysql-shell-utilities-dump-opt-control


https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-utilities-load-dump.html


https://www.percona.com/blog/backup-restore-performance-conclusion-mysqldump-vs-mysql-shell-utilities-vs-mydumper-vs-mysqlpump-vs-xtrabackup/


https://www.percona.com/blog/dump-performance-comparison-mysqldump-vs-mysql-shell-utilities-vs-mydumper/