ceph osd down 的处理办法及新增加OSD
| 1: OSD 状态
有二个OSD down | [root@node-11 ~]# [root@node-11 ~]# [root@node-11 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -10 0 IDC rack01 -1 0.05598 root IDC-01 -3 0.01900 host node-10 0 hdd 0.01900 osd.0 up 1.00000 1.00000 -5 0.01900 host node-11 1 hdd 0.01900 osd.1 down 1.00000 1.00000 -7 0.01799 host node-12 2 hdd 0.01799 osd.2 down 0 1.00000 [root@node-11 ~]# |
| 2： Ceph集群显示状态异常 | [root@node-11 ~]# ceph -s cluster: id: e59a2590-37b2-4153-8d90-3a14dfdb8f43 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 1 osds down 1 host (1 osds) down 10 pool(s) backfillfull 3 pool(s) have non-power-of-two pg_num Reduced data availability: 249 pgs inactive Degraded data redundancy: 10078/15117 objects degraded (66.667%), 96 pgs degraded, 249 pgs undersized 3 daemons have recently crashed
services: mon: 3 daemons, quorum node-10,node-11,node-12 (age 14m) mgr: node-10(active, since 2h) mds: cephfs:1/1 {0=node-12=up:replay} 2 up:standby osd: 3 osds: 1 up (since 14m), 2 in (since 17m)
data: pools: 10 pools, 249 pgs objects: 5.04k objects, 18 GiB usage: 38 GiB used, 594 MiB / 39 GiB avail pgs: 100.000% pgs not active 10078/15117 objects degraded (66.667%) 153 undersized+peered 96 undersized+degraded+peered
[root@node-11 ~]# |
| 3: OSD 删除前的确认操作 | 删除前，运行前面提到的ok-to-stop和safe-to-destroy命令，根据返回的结果来决定是否能够执行删除OSD操作。
[root@node-11 ~]# ceph osd ok-to-stop osd.1 OSD(s) 1 are ok to stop without reducing availability or risking data, provided there are no other concurrent failures or interventions. 0 PGs are likely to be degraded (but remain available) as a result. [root@node-11 ~]# ceph osd ok-to-stop osd.2 OSD(s) 2 are ok to stop without reducing availability or risking data, provided there are no other concurrent failures or interventions. 0 PGs are likely to be degraded (but remain available) as a result. [root@node-11 ~]# [root@node-11 ~]#
[root@node-11 ~]# ceph osd safe-to-destroy osd.1 Error EAGAIN: OSD(s) 1 have no reported stats, and not all PGs are active+clean; we cannot draw any conclusions. [root@node-11 ~]# ceph osd safe-to-destroy osd.2 Error EAGAIN: OSD(s) 2 have no reported stats, and not all PGs are active+clean; we cannot draw any conclusions. [root@node-11 ~]# |
|
4: 确认osd-0的Crush信息 | [root@node-11 ~]# ceph osd crush ls osd.1 osd.1 [root@node-11 ~]# ceph osd crush ls osd.2 osd.2 [root@node-11 ~]# |
| 5:获取osd-0相关的cephx信息 | [root@node-11 ~]# ceph auth get osd.1 exported keyring for osd.1 [osd.1] key = AQAn0HleOu2nExAAyyyda7dOo/5z7wpr0Kx6cg== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [root@node-11 ~]# ceph auth get osd.2 exported keyring for osd.2 [osd.2] key = AQBU0Hle8gNoFxAAQglVj1Y7QrU01xeSf6vw4g== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [root@node-11 ~]# |
| 6:如果是替换现有OSD-0的磁盘，则执行destro y操作，你会发现只有OSD-0的keyring被删除。 | [root@node-11 ~]# ceph osd destroy osd.1 --yes-i-really-mean-it destroyed osd.1 [root@node-11 ~]# ceph osd destroy osd.2 --yes-i-really-mean-it destroyed osd.2 [root@node-11 ~]#
[root@node-11 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -10 0 IDC rack01 -1 0.05598 root IDC-01 -3 0.01900 host node-10 0 hdd 0.01900 osd.0 up 1.00000 1.00000 -5 0.01900 host node-11 1 hdd 0.01900 osd.1 destroyed 1.00000 1.00000 -7 0.01799 host node-12 2 hdd 0.01799 osd.2 destroyed 0 1.00000 [root@node-11 ~]#
[root@node-11 ~]# ceph osd crush ls osd.1 osd.1 [root@node-11 ~]# ceph osd crush ls osd.2 osd.2 [root@node-11 ~]# ceph auth get osd.1 Error ENOENT: failed to find osd.1 in keyring [root@node-11 ~]# ceph auth get osd.2 Error ENOENT: failed to find osd.2 in keyring [root@node-11 ~]# |
| 7:如果是彻底删除OSD-0，则执行purge操作，你会发现所有OSD-0关联的信息都被删除。 | [root@node-11 ~]# [root@node-11 ~]# systemctl stop ceph-osd@1 [root@node-11 ~]# systemctl stop ceph-osd@2 [root@node-11 ~]# ceph osd purge osd.1 --yes-i-really-mean-it purged osd.1 [root@node-11 ~]# ceph osd purge osd.2 --yes-i-really-mean-it purged osd.2 [root@node-11 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -10 0 IDC rack01 -1 0.01900 root IDC-01 -3 0.01900 host node-10 0 hdd 0.01900 osd.0 up 1.00000 1.00000 -5 0 host node-11 -7 0 host node-12 [root@node-11 ~]#
[root@node-11 ~]# ceph osd crush ls osd.1 Error ENOENT: node 'osd.1' does not exist [root@node-11 ~]# ceph osd crush ls osd.2 Error ENOENT: node 'osd.2' does not exist [root@node-11 ~]# ceph auth get osd.1 Error ENOENT: failed to find osd.1 in keyring [root@node-11 ~]# ceph auth get osd.2 Error ENOENT: failed to find osd.2 in keyring [root@node-11 ~]# |
| 8:最后如果你需要清理原来OSD-0对应数 据盘的数据，可以执行以下命令 | [root@node-11 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 50G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 48.9G 0 part └─centos-root 253:0 0 48.9G 0 lvm / sdb 8:16 0 19G 0 disk └─ceph--83bde6e7--155e--4f2c--ab98--b03b9538f0cb-osd--block--4bd18b82--5285--41ff--bed3--08f3d26c2f39 253:1 0 19G 0 lvm [root@node-11 ~]#
[root@node-11 ~]# [root@node-11 ~]# ceph-volume lvm zap --destroy /dev/sdb --> Zapping: /dev/sdb Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied stderr: , 0.0186076 s, 564 MB/s --> Zapping successful for: <Raw Device: /dev/sdb> [root@node-11 ~]# |
|
| [root@node-12 ~]# [root@node-12 ~]# [root@node-12 ~]# ceph-volume lvm zap --destroy /dev/sdb --> Zapping: /dev/sdb --> Zapping lvm member /dev/sdb. lv_path is /dev/ceph-e67c2622-46d5-4a96-af51-fa634d57b052/osd-block-163b7415-368c-40e9-b324-5f310789440e --> Unmounting /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2 stderr: umount: /var/lib/ceph/osd/ceph-2 (tmpfs) unmounted Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-e67c2622-46d5-4a96-af51-fa634d57b052/osd-block-163b7415-368c-40e9-b324-5f310789440e bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied stderr: , 0.0758462 s, 138 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-e67c2622-46d5-4a96-af51-fa634d57b052 Running command: /usr/sbin/vgremove -v -f ceph-e67c2622-46d5-4a96-af51-fa634d57b052 stderr: Removing ceph--e67c2622--46d5--4a96--af51--fa634d57b052-osd--block--163b7415--368c--40e9--b324--5f310789440e (253:1) stderr: Archiving volume group "ceph-e67c2622-46d5-4a96-af51-fa634d57b052" metadata (seqno 17). stderr: Releasing logical volume "osd-block-163b7415-368c-40e9-b324-5f310789440e" stderr: Creating volume group backup "/etc/lvm/backup/ceph-e67c2622-46d5-4a96-af51-fa634d57b052" (seqno 18). stdout: Logical volume "osd-block-163b7415-368c-40e9-b324-5f310789440e" successfully removed stderr: Removing physical volume "/dev/sdb" from volume group "ceph-e67c2622-46d5-4a96-af51-fa634d57b052" stdout: Volume group "ceph-e67c2622-46d5-4a96-af51-fa634d57b052" successfully removed Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied stderr: , 0.0153976 s, 681 MB/s --> Zapping successful for: <Raw Device: /dev/sdb> [root@node-12 ~]# [root@node-12 ~]# [root@node-12 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 50G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 48.9G 0 part └─centos-root 253:0 0 48.9G 0 lvm / sdb 8:16 0 18G 0 disk [root@node-12 ~]# |
| 9: 新增OSD | 格式化磁盘 [root@node-10 cluster]# ceph-deploy disk zap node-11 /dev/sdb [root@node-10 cluster]# ceph-deploy disk zap node-12 /dev/sdb
创建OSD，如在每个节点上我们都有一块磁盘，分别为sdb
[root@node-11 ~]# ceph-volume lvm create --data /dev/sdb [root@node-11 ~]# [root@node-11 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 50G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 48.9G 0 part └─centos-root 253:0 0 48.9G 0 lvm / sdb 8:16 0 19G 0 disk └─ceph--481d09c0--5bbd--491e--8f0e--bd5b136e2e2c-osd--block--2454cf36--8f92--435b--9177--200a0f261e03 253:1 0 19G 0 lvm [root@node-11 ~]# [root@node-11 ~]# ceph-volume lvm list
====== osd.1 =======
[block] /dev/ceph-481d09c0-5bbd-491e-8f0e-bd5b136e2e2c/osd-block-2454cf36-8f92-435b-9177-200a0f261e03
block device /dev/ceph-481d09c0-5bbd-491e-8f0e-bd5b136e2e2c/osd-block-2454cf36-8f92-435b-9177-200a0f261e03 block uuid A1Rwg0-xID5-Y6Py-2F3d-UNzS-S7Zs-i1AK6r cephx lockbox secret cluster fsid e59a2590-37b2-4153-8d90-3a14dfdb8f43 cluster name ceph crush device class None encrypted 0 osd fsid 2454cf36-8f92-435b-9177-200a0f261e03 osd id 1 type block vdo 0 devices /dev/sdb [root@node-11 ~]# |
|
| [root@node-12 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 50G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 48.9G 0 part └─centos-root 253:0 0 48.9G 0 lvm / sdb 8:16 0 18G 0 disk [root@node-12 ~]# ceph-volume lvm create --data /dev/sdb Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ae9f5273-0bbe-4818-84a7-291fe17fde6b Running command: /usr/sbin/vgcreate --force --yes ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d /dev/sdb stdout: Physical volume "/dev/sdb" successfully created. stdout: Volume group "ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d" successfully created Running command: /usr/sbin/lvcreate --yes -l 100%FREE -n osd-block-ae9f5273-0bbe-4818-84a7-291fe17fde6b ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d stdout: Logical volume "osd-block-ae9f5273-0bbe-4818-84a7-291fe17fde6b" created. Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d/osd-block-ae9f5273-0bbe-4818-84a7-291fe17fde6b Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/ln -s /dev/ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d/osd-block-ae9f5273-0bbe-4818-84a7-291fe17fde6b /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap stderr: 2020-05-19 16:52:45.735 7fb0a78c5700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2020-05-19 16:52:45.735 7fb0a78c5700 -1 AuthRegistry(0x7fb0a0064fe8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx stderr: got monmap epoch 1 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name osd.2 --add-key AQBcnsNeEwATGRAA/L/9X86ilMkZVN2dM8qbIA== stdout: creating /var/lib/ceph/osd/ceph-2/keyring added entity osd.2 auth(key=AQBcnsNeEwATGRAA/L/9X86ilMkZVN2dM8qbIA==) Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid ae9f5273-0bbe-4818-84a7-291fe17fde6b --setuser ceph --setgroup ceph --> ceph-volume lvm prepare successful for: /dev/sdb Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d/osd-block-ae9f5273-0bbe-4818-84a7-291fe17fde6b --path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d/osd-block-ae9f5273-0bbe-4818-84a7-291fe17fde6b /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-ae9f5273-0bbe-4818-84a7-291fe17fde6b stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-2-ae9f5273-0bbe-4818-84a7-291fe17fde6b.service to /usr/lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2 --> ceph-volume lvm create successful for: /dev/sdb [root@node-12 ~]# [root@node-12 ~]# [root@node-12 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 50G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 48.9G 0 part └─centos-root 253:0 0 48.9G 0 lvm / sdb 8:16 0 18G 0 disk └─ceph--6fac2b4c--6c25--4ac4--a844--0568fc77022d-osd--block--ae9f5273--0bbe--4818--84a7--291fe17fde6b 253:1 0 18G 0 lvm [root@node-12 ~]#
[root@node-12 ~]# ceph-volume lvm list
====== osd.2 =======
[block] /dev/ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d/osd-block-ae9f5273-0bbe-4818-84a7-291fe17fde6b
block device /dev/ceph-6fac2b4c-6c25-4ac4-a844-0568fc77022d/osd-block-ae9f5273-0bbe-4818-84a7-291fe17fde6b block uuid 99rJJe-jiiL-FVfK-6d0T-fYUx-SLAa-wU8k0S cephx lockbox secret cluster fsid e59a2590-37b2-4153-8d90-3a14dfdb8f43 cluster name ceph crush device class None encrypted 0 osd fsid ae9f5273-0bbe-4818-84a7-291fe17fde6b osd id 2 type block vdo 0 devices /dev/sdb [root@node-12 ~]#
[root@node-12 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -10 0 IDC rack01 -1 0.05518 root IDC-01 -3 0.01900 host node-10 0 hdd 0.01900 osd.0 up 1.00000 1.00000 -5 0.01859 host node-11 1 hdd 0.01859 osd.1 up 1.00000 1.00000 -7 0.01759 host node-12 2 hdd 0.01759 osd.2 up 1.00000 1.00000 [root@node-12 ~]# |
| Ceph Pg重组，数据回拷 | [root@node-10 cluster]# ceph -s cluster: id: e59a2590-37b2-4153-8d90-3a14dfdb8f43 health: HEALTH_WARN 1 backfillfull osd(s) 10 pool(s) backfillfull 3 pool(s) have non-power-of-two pg_num Reduced data availability: 15 pgs inactive Degraded data redundancy: 2473/15117 objects degraded (16.359%), 48 pgs degraded, 34 pgs undersized 16 daemons have recently crashed
services: mon: 3 daemons, quorum node-10,node-11,node-12 (age 44m) mgr: node-10(active, since 2h) mds: cephfs:1 {0=node-12=up:active} 2 up:standby osd: 3 osds: 3 up (since 4m), 3 in (since 4m); 33 remapped pgs rgw: 1 daemon active (node-10)
data: pools: 10 pools, 249 pgs objects: 5.04k objects, 18 GiB usage: 50 GiB used, 7.2 GiB / 57 GiB avail pgs: 6.024% pgs not active 2473/15117 objects degraded (16.359%) 194 active+clean 19 active+recovery_wait+undersized+degraded+remapped 14 active+recovery_wait+degraded 14 undersized+degraded+remapped+backfill_wait+peered 7 active+recovery_wait 1 undersized+degraded+remapped+backfilling+peered
io: client: 383 B/s rd, 0 B/s wr, 17 op/s rd, 0 op/s wr recovery: 68 MiB/s, 18 objects/s
[root@node-10 cluster]# |