ceph（luminous）-Bluestore，ceph-kvstore-tool工具说明

前言

此工具主要是作用是针对ceph中各种kv存储进行操作和展示的工具.
先看一下help列表

[root@test-1 ceph]# ceph-kvstore-tool --help
Usage: ceph-kvstore-tool <leveldb|rocksdb|bluestore-kv> <store path> command [args...]
Commands:
  list [prefix] //列出所有key，可以指定prefix
  list-crc [prefix] //列出crc
  exists <prefix> [key] //判断某个key是否存在
  get <prefix> <key> [out <file>] //获取具体key的value，保存到文件中
  crc <prefix> <key> //获取crc
  get-size [<prefix> <key>] //获取db的大小或者某个kv的大小
  set <prefix> <key> [ver <N>|in <file>] //设置kv
  rm <prefix> <key> //删除kv
  rm-prefix <prefix> //删除指定前缀的kv组
  store-copy <path> [num-keys-per-tx] //拷贝kv
  store-crc <path> //store的crc
  compact //合并kv
  compact-prefix <prefix>
  compact-range <prefix> <start> <end>

基本上所有的操作都需要把进程停止了以后再进行，因为rocksdb或者leveldb有一个文件锁，每次打开db之前都会给文件上锁，如果文件已经被锁上了（osd或者mon进程还在），就无法再次进行上锁（ceph-kvstore-tool进程还需要对db上锁）.

数据库主要分为3类，leveldb和rocksdb都是可配的，bluestore-kv是bluestore下专用的kv存储，其内部也是rocksdb。

1. list
   可以列出数据库的所有key值
   下面列出了bluestore-key

[root@test-1 crushmap]# ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-0/ list > key.list
2018-10-23 10:58:25.645999 7fcd99345ec0  1 bluestore(/var/lib/ceph/osd/ceph-0/) _mount path /var/lib/ceph/osd/ceph-0/
.
.
.
//列出的key如下
B       blocks  //B是prefix，blocks是key
B       blocks_per_key
B       bytes_per_block
B       size
C       10.0_head//  C是prefix，表示collection的意思.
C       10.10_head
C       10.11_head
C       10.12_head
C       10.1e_head
C       10.20_head
C       10.21_head
C       10.25_head
C       10.29_head
C       10.2a_head
C       10.2d_head
C       10.2e_head
C       10.2f_head
C       10.32_head
C       10.33_head
C       10.36_head
C       10.38_head
C       10.3c_head
C       10.3d_head
C       10.42_head

bluestore-kv中prefix的含义:
S* superblock block size、configure options，整个存储的核心数据
B* block allocation： freeblock、block that used
C* collection name -> cnode_t
O对象名存储到ONODE的映射关系
L wal 写前日志
M元数据信息
const string PREFIX_SUPER = “S”; // field -> value
const string PREFIX_STAT = “T”; // field -> value(int64 array)
const string PREFIX_COLL = “C”; // collection name -> cnode_t
const string PREFIX_OBJ = “O”; // object name -> onode_t
const string PREFIX_OMAP = “M”; // u64 + keyname -> value
const string PREFIX_DEFERRED = “L”; // id -> deferred_transaction_t
const string PREFIX_ALLOC = “B”; // u64 offset -> u64 length (freelist)
const string PREFIX_SHARED_BLOB = “X”; // u64 offset -> shared_blob_t
omap中保存了pglog，pglog中的***epoch_为小端存储，65 1c 00 00 = 7269

下面列出了mon-rocksdb的key

[root@test-1 crushmap]# ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-test-1/store.db/ list > mon.list
2018-10-23 11:03:48.824528 7f9fef12cec0  4 rocksdb: RocksDB version: 5.4.0

2018-10-23 11:03:48.824571 7f9fef12cec0  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
.
.
.

2. list-crc
   显示kv的crc
3. exists
   判断key是否存在

[root@test-1 ceph]# ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-test-1/store.db/ exists monmap
2018-10-23 11:08:09.189204 7f5611b27ec0  4 rocksdb: RocksDB version: 5.4.0

2018-10-23 11:08:09.189247 7f5611b27ec0  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2018-10-23 11:08:09.189250 7f5611b27ec0  4 rocksdb: Compile date Sep 20 2018
.
.
.
2018-10-23 11:08:09.201440 7f5611b27ec0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1540264089201431, "job": 1, "event": "recovery_finished"}
2018-10-23 11:08:09.201987 7f5611b27ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_files.cc:314] [JOB 2] Tried to delete a non-existing file /var/lib/ceph/mon/ceph-test-1/store.db///MANIFEST-024885 type=3 #24885 -- IO error: /var/lib/ceph/mon/ceph-test-1/store.db///MANIFEST-024885: No such file or directory

2018-10-23 11:08:09.202022 7f5611b27ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_files.cc:314] [JOB 2] Tried to delete a non-existing file /var/lib/ceph/mon/ceph-test-1/store.db//024886.log type=0 #24886 -- IO error: /var/lib/ceph/mon/ceph-test-1/store.db//024886.log: No such file or directory

2018-10-23 11:08:09.210464 7f5611b27ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7f5614128000
(monmap, ) exists  //存在prefix为monmap
2018-10-23 11:08:09.211004 7f5611b27ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2018-10-23 11:08:09.211307 7f5611b27ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:343] Shutdown complete

4. get
   从kv数据库中获取某个指定key的value，并输出为文件.可以用于导出osdmap，monmap时使用.

[root@test-1 crushmap]# ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-test-1/store.db/ get monmap 9 out monmap9 //把prefix为monmap key为9的monmap（epoch为9的monmap输出为文件 monmap9）
2018-10-23 11:27:56.087672 7fa96ec0eec0  4 rocksdb: RocksDB version: 5.4.0
.
.
.
2018-10-23 11:27:56.103530 7fa96ec0eec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_files.cc:314] [JOB 2] Tried to delete a non-existing file /var/lib/ceph/mon/ceph-test-1/store.db//024898.log type=0 #24898 -- IO error: /var/lib/ceph/mon/ceph-test-1/store.db//024898.log: No such file or directory

2018-10-23 11:27:56.111757 7fa96ec0eec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7fa970420000
(monmap, 9)
2018-10-23 11:27:56.112429 7fa96ec0eec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2018-10-23 11:27:56.112698 7fa96ec0eec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:343] Shutdown complete
[root@test-1 crushmap]# monmaptool --print monmap9 
monmaptool: monmap file monmap9
epoch 9
fsid acc6dc6a-79cd-45dc-bf1f-83a576eb8039
last_changed 2018-10-22 16:32:01.468845
created 2018-09-29 17:21:07.728600
0: 192.168.125.109:6789/0 mon.test-1
1: 192.168.125.110:6789/0 mon.test-2
2: 192.168.125.111:6789/0 mon.test-3

5. crc
   获取指定prefix下key的crc

6. get-size
   获取指定prefix下key的大小,或者获取整个db的大小.

7. set
   设置指定prefix下key的value，可以通过文件方式或者变量赋值.

[root@test-1 crushmap]# ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-test-1/store.db set monmap 9 in monmap9 //通过文件方式设置变量
2018-10-23 11:39:23.720162 7fd3b7176ec0  4 rocksdb: RocksDB version: 5.4.0

2018-10-23 11:39:23.720204 7fd3b7176ec0  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2018-10-23 11:39:23.720207 7fd3b7176ec0  4 rocksdb: Compile date Sep 20 2018
.
.
.
2018-10-23 11:39:23.723255 7fd3b7176ec0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1540265963723231, "job": 1, "event": "recovery_started", "log_files": [24919]}
2018-10-23 11:39:23.723279 7fd3b7176ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_open.cc:482] Recovering log #24919 mode 2
2018-10-23 11:39:23.728876 7fd3b7176ec0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1540265963728847, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 24920, "file_size": 1147, "table_properties": {"data_size": 199, "index_size": 28, "filter_size": 18, "raw_key_size": 16, "raw_average_key_size": 16, "raw_value_size": 402, "raw_average_value_size": 402, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2018-10-23 11:39:23.728944 7fd3b7176ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/version_set.cc:2395] Creating manifest 24921

2018-10-23 11:39:23.744083 7fd3b7176ec0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1540265963744074, "job": 1, "event": "recovery_finished"}
2018-10-23 11:39:23.753370 7fd3b7176ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7fd3b9210000
2018-10-23 11:39:23.757713 7fd3b7176ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2018-10-23 11:39:23.758026 7fd3b7176ec0  4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:343] Shutdown complete

8. rm
   删除指定prefix下的kv值.

9. rm-prefix
   删除指定prefix

10. store-copy
    拷贝kvstore，只能拷贝为同类型的kv,如果是bluestore-kv想导出为rocksdb可以使用ceph-bluestore-tool进行导出.

11. store-crc
    显示store的crc

12. compact
    合并kv，由于rocksdb和leveldb的特性都是一直追加写log文件，等到log文件到了一定大小再找出相同key值进行合并，此操作可以强制kvdb进行合并操作，一般合并后kvdb大小会减小.

13. compact-prefix
    合并指定prefix的kv

14. compact-range
    合并指定范围的kv