redis-migrate-tool数据迁移工具测试
邢良才
工具特性
- 快速
- 多线程
- 基于redis复制
- 实时迁移
- 迁移过程中，源集群不影响对外提供服务
- 异构迁移
- 支持Twemproxy集群，redis cluster集群，rdb文件 和 aof文件。
- 过滤功能
- 当目标集群是Twemproxy，数据会跳过Twemproxy直接导入到后端的redis
- 迁移状态显示
- 完善的数据抽样校验(-C redis_check)
redis-migrate-tool 是维品会开源的一款redis数据迁移工具，基于redis复制，快速，稳定。
是一个方便和有用的工具，用于在redis之间迁移数据。
github 地址：https://github.com/vipshop/redis-migrate-tool
安装&编译
# 安装依赖性
root@ubuntu-x64_01:~# apt-get install -y automake libtool autoconf bzip2
root@ubuntu-x64_01:~# apt-get install -y libtool libsysfs-dev
root@ubuntu-x64_01:~# apt-get install -y autoreconf
# 下载安装包
root@ubuntu-x64_01:/opt# wget -c "https://github.com/vipshop/redis-migrate-tool/archive/refs/heads/master.zip"
# 安装配置
root@ubuntu-x64_01:/opt/redis-migrate-tool-master# autoreconf -fvi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy --force
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'config'.
libtoolize: copying file 'config/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
libtoolize: 'AC_PROG_RANLIB' is rendered obsolete by 'LT_INIT'
autoreconf: running: /usr/bin/autoconf --force
autoreconf: running: /usr/bin/autoheader --force
autoreconf: running: automake --add-missing --copy --force-missing
configure.ac:29: installing 'config/compile'
configure.ac:16: installing 'config/missing'
src/Makefile.am:34: warning: whitespace following trailing backslash
src/Makefile.am: installing 'config/depcomp'
autoreconf: Leaving directory `.'
# 创建Makefile文件
root@ubuntu-x64_01:/opt/redis-migrate-tool-master# ./configure
########### ...... 省略部分输出...... ###########
configure: creating ./config.status
config.status: creating Makefile
config.status: creating dep/Makefile
config.status: creating src/Makefile
config.status: creating src/ae/Makefile
config.status: creating src/intset/Makefile
config.status: creating src/ziplist/Makefile
config.status: creating src/zipmap/Makefile
config.status: creating src/lzf/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
## 编译
root@ubuntu-x64_01:/opt/redis-migrate-tool-master# make
########### ...... 省略部分输出...... ###########
make[3]: Leaving directory /opt/redis-migrate-tool-master/src
make[2]: Leaving directory /opt/redis-migrate-tool-master/src
make[2]: Entering directory /opt/redis-migrate-tool-master
make[2]: Leaving directory /opt/redis-migrate-tool-master
make[1]: Leaving directory /opt/redis-migrate-tool-master
# 检查工具版本
root@ubuntu-x64_01:/opt/redis-migrate-tool-master/src# ./redis-migrate-tool -h
This is redis-migrate-tool-0.1.0
Usage: redis-migrate-tool [-?hVdIn] [-v verbosity level] [-o output file]
[-c conf file] [-C command]
[-f source address] [-t target address]
[-p pid file] [-m mbuf size] [-r target role]
[-T thread number] [-b buffer size]
Options:
-h, --help : this help
-V, --version : show version and exit
-d, --daemonize : run as a daemon
-I, --information : print some useful information
-n, --noreply : do not receive the target redis reply
-v, --verbosity=N : set logging level (default: 5, min: 0, max: 11)
-o, --output=S : set logging file (default: stderr)
-c, --conf-file=S : set configuration file (default: rmt.conf)
-p, --pid-file=S : set pid file (default: off)
-m, --mbuf-size=N : set mbuf size (default: 512)
-C, --command=S : set command to execute (default: redis_migrate)
-r, --source-role=S : set the source role (default: single, you can input: single, twemproxy or redis_cluster)
-R, --target-role=S : set the target role (default: single, you can input: single, twemproxy or redis_cluster)
-T, --thread=N : set how many threads to run the job(default: 2)
-b, --buffer=S : set buffer size to run the job (default: 1048576 byte, unit:G/M/K)
-f, --from=S : set source redis address (default: 127.0.0.1:6379)
-t, --to=S : set target redis group address (default: 127.0.0.1:6380)
-s, --step=N : set step (default: 1)
Commands:
redis_migrate : Migrate data from source group to target group.
redis_check : Compare data between source group and target group. Default compare 1000 keys. You can set a key count behind.
redis_testinsert : Just for test! Insert some string, list, set, zset and hash keys into the source redis group. Default 1000 keys. You can set key type and key count behind.
部分指令解析：
-h,–help：帮助
-V,–version：显示版本
-d,–daemonize：后台进程运行
-I,–information：打印一些有用的信息，包括可以解析的指令（126个），不支持的指令（14个）等等
-v,–verbosity=N：设置日志等级。(默认: 5, 最低: 0, 最高: 11)
-o,–output=S：设置输出的日志文件
-c,–conf-file=S：设置配置文件。(默认: rmt.conf)
-C,–command=S：设置运行的指令(默认: redis_migrate ，迁移)。 redis_check 比较源和目的，默认1000个样本key。 redis_testinsert测试插入Keys，默认所有类型总共1000个。
-T,–thread=N：设置多少个线程用来运行工具。(默认: 4)
警告
在运行工具之前，确保源redis所在的机器有足够的内存可以允许至少一个redis生成.rdb文件，
如果源机器有大量足够的内存允许所有的redis生成.rdb，可以在配置文件rmt.conf设置 source_safe:false。
生成快照
这里测试生成快照文件 ，然后再清空Redis，最后通过快照RDB文件来恢复~
192.168.88.11:6379> LASTSAVE
(integer) 1662704287
192.168.88.11:6379> BGSAVE
Background saving started
192.168.88.11:6379> LASTSAVE
(integer) 1662709634
#当前Redis键值，生产环境不用使用keys操作！！！
192.168.88.11:6379> keys *
1) "k5"
2) "k4"
3) "k1"
4) "k3"
5) "k6"
6) "k2"
#检查快照文件，并复制一份
root@ubuntu-x64_01:/data/redis/data# ls -tlh
total 21M
-rw-rw-r-- 1 redis redis 239 9月 9 15:47 dump.rdb
-rw-r--r-- 1 redis redis 21M 9月 9 14:14 appendonly.aof
root@ubuntu-x64_01:/data/redis/data# cp dump.rdb /opt/dump.rdb.save
# 清空Redis数据库，生产环境请谨慎操作！！ 操作前备份RDB文件
192.168.88.11:6379> flushdb
OK
192.168.88.11:6379>
192.168.88.11:6379>
192.168.88.11:6379> keys *
(empty list or set)
rmt配置文件
配置文件包含三部分：[source], [target] 和 [common]
迁移工具的来源（source）可以是：单独的redis实例，twemproxy集群，redis cluster，rdb文件，aof文件。迁移工具的目标（target）可以是：单独的redis实例，twemproxy集群，redis cluster，rdb文件。
Load data from rdb file to redis cluster.
root@ubuntu-x64_01:/opt/redis-migrate-tool-master# cat rmt.conf
[source]
type: rdb file
servers:
- /opt/dump.rdb.save
[target]
type: single
servers:
-192.168.88.11:6379
[common]
listen: 0.0.0.0:8888
运行迁移
注意： 以下是在 redis 3.2.10 版本测试 ~
-d 指定为后台运行，如果再次运行可能需要杀死占用当前端口的进程。
netstat -tnulp | grep xxxx 查看找到redis-migrate-tool的端口号， kill-9[端口号]杀死再运行。
指定输出日志文件为log，可通过 tail -f
# 运行迁移
root@ubuntu-x64_01:/opt/redis-migrate-tool-master# src/redis-migrate-tool -c rmt.conf -o log -d
# 查看日志
root@ubuntu-x64_01:/opt/redis-migrate-tool-master# cat log
[2022-09-09 16:56:53.412] rmt_core.c:525 Nodes count of source group : 1
[2022-09-09 16:56:53.412] rmt_core.c:526 Total threads count : 2
[2022-09-09 16:56:53.412] rmt_core.c:527 Read threads count assigned: 1
[2022-09-09 16:56:53.412] rmt_core.c:528 Write threads count assigned: 1
[2022-09-09 16:56:53.412] rmt_core.c:2444 Total threads count in fact: 1
[2022-09-09 16:56:53.412] rmt_core.c:2445 Read threads count in fact: 0
[2022-09-09 16:56:53.412] rmt_core.c:2446 Write threads count in fact: 1
[2022-09-09 16:56:53.412] rmt_core.c:2488 write thread(0):
[2022-09-09 16:56:53.412] rmt_core.c:2494 /opt/dump.rdb.save
[2022-09-09 16:56:53.412] rmt_core.c:2551 migrate job is running...
[2022-09-09 16:56:53.412] rmt_redis.c:6601 Rdb file for node[/opt/dump.rdb.save] parsed finished, use: 0 s.
备注
redis-migrate-tool-master 工具在 redis_version:5.0.5 测试不支持解析RDB文件格式，迁移时会报错：
不适用redis4.0.x及以上版本：
root@ubuntu-x64_01:/opt/redis-migrate-tool-master# cat log
[2022-09-09 16:06:57.506] rmt_core.c:525 Nodes count of source group : 1
[2022-09-09 16:06:57.506] rmt_core.c:526 Total threads count : 2
[2022-09-09 16:06:57.506] rmt_core.c:527 Read threads count assigned: 1
[2022-09-09 16:06:57.506] rmt_core.c:528 Write threads count assigned: 1
[2022-09-09 16:06:57.506] rmt_core.c:2444 Total threads count in fact: 1
[2022-09-09 16:06:57.506] rmt_core.c:2445 Read threads count in fact: 0
[2022-09-09 16:06:57.506] rmt_core.c:2446 Write threads count in fact: 1
[2022-09-09 16:06:57.506] rmt_core.c:2488 write thread(0):
[2022-09-09 16:06:57.506] rmt_core.c:2494 /opt/dump.rdb.save
[2022-09-09 16:06:57.506] rmt_core.c:2551 migrate job is running...
[2022-09-09 16:06:57.506] rmt_redis.c:6446 ERROR: Can't handle RDB format version 1396813928
[2022-09-09 16:06:57.506] rmt_redis.c:6715 ERROR: Rdb file for node[/opt/dump.rdb.save] parsed failed
redis 5.0 、6.0、7.0 迁移请使用 RedisShake 工具
其它rmt配置文件
其它的rmt配置文件 可以参考如下：
For example, the configuration file shown below is to migrate data from single to twemproxy.
[source]
type: single
servers:
- 127.0.0.1:6379
- 127.0.0.1:6380
- 127.0.0.1:6381
- 127.0.0.1:6382
[target]
type: twemproxy
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers:
- 127.0.0.1:6380:1 server1
- 127.0.0.1:6381:1 server2
- 127.0.0.1:6382:1 server3
- 127.0.0.1:6383:1 server4
[common]
listen: 0.0.0.0:8888
threads: 2
step: 1
mbuf_size: 1024
source_safe: true
Migrate data from twemproxy to redis cluster.
[source]
type: twemproxy
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers:
- 127.0.0.1:6379
- 127.0.0.1:6380
- 127.0.0.1:6381
- 127.0.0.1:6382
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 1
mbuf_size: 512
Migrate data from a redis cluster to another redis cluster with key filter(key’s prefix is “abc”).
[source]
type: redis cluster
servers:
- 127.0.0.1:8379
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
filter: abc*
Load data from rdb file to redis cluster.
[source]
type: rdb file
servers:
- /data/redis/dump1.rdb
- /data/redis/dump2.rdb
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 2
mbuf_size: 512
source_safe: false
Just save rdb file from redis cluster.
[source]
type: redis cluster
servers:
- 127.0.0.1:7379
[target]
type: rdb file
[common]
listen: 0.0.0.0:8888
source_safe: true
Load data from aof file to redis cluster.
[source]
type: aof file
servers:
- /data/redis/appendonly1.aof
- /data/redis/appendonly2.aof
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 2
类似资料：